Skip to content
LumenaryAcademy
Frontier 80 min

Infrastructure for AI in the cloud

Renting accelerators by the thousand — the instances, the networks that bind them, schedulers that start a job all at once or not at all, serving that scales on tokens, and what a token costs.

Code tested · 5AI-draftedAwaiting review by a cloud engineer

Your guides: Chloe · The Computer Scientist, Margaret · The Experimenter, Gideon · The Verifier

MotivationH

A cluster you rent by the hour

A research group in Seoul wants to adapt an open-weight language model to Korean legal documents. A start-up in Nairobi wants to serve a speech model to farmers' phones across East Africa. A university in São Paulo has a month of training on 256 GPUs. All of them will rent, not build — and soon discover that renting accelerators is not like renting the web servers of the earlier lessons.

Four things are different. The hardware is scarce and expensive: a single 8-GPU server can cost more per hour than dozens of ordinary virtual machines, and the newest generation is often unavailable without a reservation. The machines must work as one: a training job on 256 GPUs is a single program that stops if any one of them stops, so it needs all of them at once, close together on a fast network. Serving is measured in tokens, not requests, and its load and its costs follow tokens. And power and cooling set the limits of where and how fast capacity can grow.

The chips themselves, the ways of splitting a model across them, and the inner workings of a serving engine are taught in Lumenary CS: accelerators and distributed training and serving models. How a data centre is supplied with tens or hundreds of megawatts is in Lumenary EE's powering data centres. This lesson is about the layer between: what you rent, how it is wired, how jobs are scheduled onto it, how serving scales, and what it all costs. It builds on Kubernetes, observability and autoscaling and cost and FinOps.

DefinitionsH

What an accelerator instance is

Definition— Accelerator instance

A virtual machine or bare-metal server rented with one or more accelerators attached — GPUs or purpose-built chips for matrix arithmetic — together with the host CPUs, memory, local NVMe storage and network adapters needed to keep them fed. The largest instances expose a whole server of 8 accelerators; rack-scale offerings expose dozens of accelerators joined by one high-bandwidth fabric as a single unit of rental.

Look inside a typical training-class instance of 2024–26 and you find eight GPUs, each with 80 to almost 300 GB of high-bandwidth memory; a switch chip or mesh joining them at hundreds of gigabytes per second each (NVLink and NVSwitch in NVIDIA's designs, Infinity Fabric in AMD's); two host CPUs with a terabyte or two of RAM; several terabytes of local NVMe; and — the part most easily overlooked — around eight network adapters of 200 or 400 Gb/s, roughly one per GPU, for talking to other servers. Ordinary instances have one network adapter. The network is a large part of what you are paying for.

What is on offer, as of late 2026, falls into a few families, each described by its provider's own documentation (see the references):

  • NVIDIA GPUs (the Hopper and Blackwell generations, with Rubin announced for 2026) are offered by every large provider — Amazon Web Services, Microsoft Azure, Google Cloud, Oracle, Alibaba Cloud and others — and by a wave of specialised GPU clouds. Rack-scale systems put 72 GPUs in one NVLink domain.
  • AMD Instinct GPUs (the MI300 and MI350 series, with a rack-scale MI400 series announced for 2026) are offered by several providers, including Azure and Oracle, and by specialised clouds.
  • Providers' own chips. Google rents TPUs in slices of a pod, from a few chips to thousands, joined by a dedicated inter-chip interconnect; Ironwood, the seventh generation, was announced in 2025. Amazon offers Trainium for training and inference and the older Inferentia for inference. Microsoft has its Maia accelerators, used mainly for its own services. In China, where export controls restrict the most advanced foreign chips, clouds increasingly offer domestic accelerators such as Huawei's Ascend series and chips designed by Alibaba's T-Head and by Baidu's Kunlunxin.

Which to choose is an engineering question, not a loyalty one: what your software stack supports (CUDA is the most widely supported; ROCm, the JAX/XLA stack for TPUs and AWS's Neuron SDK each have their own coverage), what is actually available in the region and quantity you need, and the measured price per unit of your workload. Vendor peak-FLOP figures say little about that. Audited benchmarks such as MLPerf are a better start, and a day of benchmarking your own model is better still.

Note

GPUs can also be shared. A GPU can be split into hardware-isolated slices (NVIDIA's multi-instance GPU), time-sliced between containers, or rented fractionally. That suits small models and notebooks; large training and serving use whole devices, usually whole servers.

IntuitionH

The networks that make thousands of chips one machine

A training cluster has two networks inside it, and a third beside it.

  • The scale-up network joins the accelerators of one server or rack: NVLink domains of 8 or 72 GPUs, a TPU slice's inter-chip links, Trainium's NeuronLink. Bandwidths are hundreds of gigabytes per second per chip, so this is where the most communication-hungry parallelism (tensor and expert parallelism) lives. UALink, an open consortium specification first published in 2025, aims to offer the same kind of fabric across vendors.
  • The scale-out (or back-end) network joins servers: InfiniBand, Ethernet with RDMA (RoCE), or a provider's own transport, such as AWS's Elastic Fabric Adapter with its SRD protocol. Roughly 50 GB/s per GPU in 2024–26 designs — an order of magnitude below scale-up.
  • The front-end network, beside them, carries everything else: datasets and checkpoints from storage, logs, the internet. Keeping it separate stops a checkpoint write from slowing the gradients.
spinespinerail 0 leafrail 1 leafrail 2 leafrail 3 leafserver Aserver BGPU 0GPU 1GPU 2GPU 3GPU 0GPU 1GPU 2GPU 3scale-up switch (NVLink / NVSwitch)scale-up switch (NVLink / NVSwitch)
A rail-optimised training fabric (four of each server's eight GPUs drawn). Inside a server, GPUs share a scale-up switch. Each GPU also has its own network adapter, wired to the leaf switch for its 'rail', so GPU 2 of every server is one hop from GPU 2 of every other. Leaf switches connect through spine switches; if the leaves have less uplink capacity than downlink, the spine is oversubscribed.

Large fabrics are built as Clos or fat-tree networks of leaf and spine switches, like the networks of ordinary cloud data centres but for a very different traffic pattern. Two design choices matter most to you as a tenant.

Rails. In a rail-optimised design, GPU ii of every server plugs into leaf switch ii. The libraries that implement collectives, such as NCCL, exploit this: they first combine data among the eight GPUs of a server over the fast scale-up switch, so that each GPU is left responsible for one-eighth of it, then run eight independent rings or trees across servers, one per rail, each staying within its own leaf switch where possible.

Oversubscription. A leaf switch with 32 ports down to GPUs and 16 up to the spine is 2:1 oversubscribed: if every GPU sends across the spine at once, each gets half its bandwidth. Training traffic does exactly that — every GPU communicating at the same moment, in lockstep — so AI back-end networks are built non-blocking (1:1) wherever possible, and a job placed across several such blocks pays for any oversubscription between them. That is why providers offer placement groups, compact placement policies or topology blocks, and why schedulers increasingly read topology labels.

The block below estimates what this means for one data-parallel training step: the gradients of an 8-billion-parameter model, all-reduced across 32 servers. It uses the alpha-beta cost model: each message costs a fixed latency α\alpha plus its size divided by the bandwidth. The ring algorithm itself is derived in accelerators and distributed training.

python
# How much does the network cost a training run? A data-parallel step: compute, then all-reduce the gradients.
# Alpha-beta model: each step of a ring costs a latency (alpha) plus bytes / bandwidth. Figures are illustrative.
GB = 1e9
params = 8e9                         # an 8-billion-parameter model
grad_bytes = 2 * params              # BF16 gradients: 16 GB
nodes, g = 32, 8                     # 32 servers of 8 GPUs = 256 GPUs
nvlink = 450 * GB                    # per GPU, each direction, inside a server
nic = 50 * GB                        # per GPU: one 400 Gb/s network adapter per GPU

def ring(n, size, bw, alpha):
    """Ring all-reduce of `size` bytes over n members: 2(n-1) steps, each moving size/n."""
    return 0.0 if n == 1 else 2 * (n - 1) * (alpha + size / n / bw)

def hierarchical(size, inter_bw, inter_alpha):
    """Reduce inside each server over NVLink, then one ring per rail across servers, then share inside."""
    return ring(g, size, nvlink, 5e-6) + ring(nodes, size / g, inter_bw, inter_alpha)

# Compute per step: 6 x params x tokens FLOPs, at 45% of an H100-class GPU's dense BF16 peak.
tokens_per_step = 4e6
compute = 6 * params * tokens_per_step / (nodes * g * 989e12 * 0.45)

placements = {
    "one non-blocking fabric":      hierarchical(grad_bytes, nic, 15e-6),
    "spine oversubscribed 2:1":     hierarchical(grad_bytes, nic / 2, 15e-6),
    "spine oversubscribed 4:1":     hierarchical(grad_bytes, nic / 4, 15e-6),
    "one ring through all 256":     ring(nodes * g, grad_bytes, nic, 15e-6),
    "two regions 1,000 km apart":   ring(g, grad_bytes, nvlink, 5e-6)
                                    + ring(nodes, grad_bytes / g, 100 * GB / 16, 5e-3),
}

price = 3.00                         # USD per GPU-hour: an example 2026 on-demand price, not a quote
base_cost = nodes * g * 30 * 24 * price
print(f"compute per step: {compute:.2f} s   (256 GPUs, {tokens_per_step / 1e6:.0f}M tokens per step)\n")
print(f"{'placement':28s} {'all-reduce':>10s} {'slowdown':>9s} {'extra cost of a 30-day run':>27s}")
for name, comm in placements.items():
    slow = comm / compute                     # if none of the communication overlaps computation
    print(f"{name:28s} {comm * 1000:7.0f} ms {slow:9.1%}       USD {base_cost * slow:12,.0f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Read the rows as upper bounds — real frameworks overlap much of the gradient exchange with the backward pass — but the ranking is robust. On one non-blocking fabric the exchange takes 141 ms against 1.69 s of compute. A 4:1 oversubscribed spine makes it 373 ms; on a month-long run that difference is worth more than USD 75,000 of idle accelerators at the example price. A single ring threaded through all 256 GPUs is slow for a less obvious reason: it crosses each server boundary only once, so it uses one of the server's eight network adapters and leaves seven idle — which is why collective libraries build one ring or tree per rail. And spreading one job across two regions, with 800 Gb/s between them and 5 ms of one-way latency, makes the exchange take more than half as long as the computation itself.

Training across sites is not impossible — Google's Gemini report describes training across multiple data centres, and methods such as DiLoCo synchronise rarely so that slow links matter less — but it is a deliberate design, not something a scheduler should stumble into. Production systems such as ByteDance's MegaScale and the clusters in Meta's Llama 3 report give a candid picture of what running such fabrics involves. On the standards side, the Ultra Ethernet Consortium published its 1.0 specification in June 2025, an attempt to give Ethernet the transport features that AI and HPC clusters need.

ComputationC

Scheduling a job that needs everything at once

A web service with 40 replicas is happy to start with 30 and get the rest later. A training job on 64 GPUs is not. Its processes meet at a barrier when they start, set up their collective communication, and only then begin; until the last one arrives, the others sit holding their GPUs and doing nothing. Kubernetes' default scheduler places pods one at a time (Kubernetes), which is exactly wrong for such a job.

Definition— Gang scheduling

A scheduling policy in which all the tasks of a job (its gang) are started together or not at all: resources are committed only when every task can be placed. The idea, also called coscheduling, comes from parallel computers of the early 1980s (John Ousterhout's 1982 work on scheduling concurrent systems) and has been standard in supercomputer batch systems ever since.

Without it, two jobs can each grab half of a cluster and wait forever for the other half. Gang scheduling removes that deadlock but creates a new inefficiency: if the job at the head of the queue needs 48 GPUs and only 40 are free, strict first-in-first-out leaves 40 GPUs idle until enough others finish.

Definition— Backfilling (EASY)

A rule that lets a later, smaller job jump ahead of a blocked job without delaying it. When the job at the head of the queue cannot start, the scheduler computes its shadow time — the earliest moment enough resources will be free, from the running jobs' declared time limits — and reserves resources for it then. A later job may start now if it fits in the free resources and either finishes before the shadow time or uses only resources the head job will not need at the shadow time. This version, from the EASY scheduler at Argonne National Laboratory in the mid-1990s, guarantees the head job is never delayed.

The simulation runs a small cluster of 8 servers (64 GPUs) and 24 jobs, each needing between one and eight whole servers, under three policies.

python
import numpy as np

NODES = 8                                  # a small cluster: 8 servers of 8 GPUs; one pod = one whole server
rng = np.random.default_rng(3)
jobs = []                                  # (name, arrival hour, servers needed, hours to run)
for j in range(24):
    jobs.append((f"J{j:02d}", int(rng.integers(0, 30)), int(rng.choice([1, 2, 2, 4, 4, 6, 8])), int(rng.integers(2, 10))))
jobs.sort(key=lambda job: job[1])

def simulate(policy, horizon=400):
    held = {}                              # job -> servers bound to it (running, or waiting at its barrier)
    started, finished = {}, {}
    for t in range(horizon):
        for name, arr, need, dur in jobs:  # jobs that finish now release their servers
            if name in started and started[name] + dur == t:
                finished[name] = t
                held.pop(name)
        free = NODES - sum(held.values())
        waiting = [jb for jb in jobs if jb[1] <= t and jb[0] not in started]
        if policy == "pod-by-pod":
            # Each pod is placed on its own; pending jobs take turns, one pod each, while servers last.
            progress = True
            while free > 0 and progress:
                progress = False
                for name, arr, need, dur in waiting:
                    if free > 0 and held.get(name, 0) < need:
                        held[name] = held.get(name, 0) + 1
                        free -= 1
                        progress = True
            for name, arr, need, dur in waiting:
                if held.get(name, 0) == need:
                    started[name] = t      # all pods present: the job passes its start-up barrier
        else:
            # Gang scheduling: a job gets all its servers at once or none.
            for i, (name, arr, need, dur) in enumerate(waiting):
                if need <= free:
                    held[name], started[name] = need, t
                    free -= need
                elif policy == "gang, FIFO":
                    break                  # strict order: nobody may overtake the job at the head
                else:
                    # EASY backfill: when can the head job surely start (its shadow time)?
                    ends = sorted((started[n] + d, k) for n, a, k, d in jobs if n in started and n not in finished)
                    avail, shadow = free, t
                    for end, servers in ends:
                        avail += servers
                        if avail >= need:
                            shadow = end
                            break
                    spare = avail - need   # servers still free at the shadow time once the head starts
                    for name2, arr2, need2, dur2 in waiting[i + 1:]:
                        if name2 in started or need2 > free:
                            continue
                        if t + dur2 <= shadow or need2 <= spare:
                            held[name2], started[name2] = need2, t
                            free -= need2
                            if t + dur2 > shadow:
                                spare -= need2
                    break
        busy = sum(held[n] for n in held if n in started)
        if len(finished) == len(jobs):
            break
        if busy == 0 and free == 0:
            return f"DEADLOCK at hour {t}: all {NODES * 8} GPUs held by {len(held)} half-started jobs, none computing"
    makespan = max(finished.values())
    waits = [started[n] - a for n, a, need, d in jobs]
    used = sum(need * d for n, a, need, d in jobs)
    return (f"finished at hour {makespan:3d}   utilisation {used / (NODES * makespan):5.1%}   "
            f"mean wait {np.mean(waits):5.1f} h   longest wait {max(waits):3d} h")

print("first jobs:", ", ".join(f"{n} ({need}x8 GPUs, {d} h)" for n, a, need, d in jobs[:4]), "...")
for policy in ["pod-by-pod", "gang, FIFO", "gang + backfill"]:
    print(f"{policy:16s} {simulate(policy)}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Placing pods independently deadlocks at hour 7: an eight-server job has seven of its servers, a two-server job has the eighth, each waits at its barrier for a server the other holds, and no GPU computes again until a human or a timeout intervenes. Gang scheduling with strict FIFO finishes everything, at 78.8% utilisation; adding EASY backfilling raises that to 90.7%, finishes 13 hours sooner and cuts the mean wait by almost eleven hours, without delaying any job at the head of the queue. On a cluster that costs thousands of dollars an hour, those twelve points of utilisation are the scheduler's whole reason to exist.

The tools, side by side

  • Slurm, the most widely used batch scheduler on supercomputers, allocates whole gangs by design, with backfilling, fair-share and topology-aware placement; the large providers each offer tooling to deploy it on their instances.
  • Kubernetes add-ons. Kueue queues jobs and admits a whole job only when its team's quota covers every pod, optionally within one topology block; Volcano replaces the scheduler with one that does gang scheduling; JobSet describes multi-part jobs such as a leader with workers.
  • Upstream Kubernetes added a Workload API and gang scheduling as alpha features, off by default, in version 1.35 (December 2025) and revised them in 1.36 (2026); dynamic resource allocation, a richer way to request devices than a GPU count, became stable in 1.34.
  • Ray reserves groups of resources together through placement groups.

Here is how a team asks for a 4-server gang under Slurm, and under Kubernetes with Kueue (not run here; the topology label is a placeholder for the one your provider sets on its nodes):

bash
#!/bin/bash
# Not run here: a Slurm batch script. Slurm starts it only when all 4 servers are allocated.
#SBATCH --job-name=finetune-ko-legal
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --time=12:00:00          # the time limit is what backfilling relies on
srun python train.py --config ko-legal.yaml
yaml
# Not run here: a Kubernetes Job admitted by Kueue as a whole, from the team's queue.
apiVersion: batch/v1
kind: Job
metadata:
  generateName: finetune-ko-legal-
  namespace: research
  labels:
    kueue.x-k8s.io/queue-name: research-gpus
spec:
  parallelism: 4
  completions: 4
  completionMode: Indexed
  template:
    metadata:
      annotations:
        kueue.x-k8s.io/podset-required-topology: "example.com/topology-block"
    spec:
      restartPolicy: Never
      containers:
      - name: trainer
        image: registry.example.com/trainer:2026.11
        command: ["python", "train.py", "--config", "ko-legal.yaml"]
        resources:
          limits:
            nvidia.com/gpu: 8
ComputationC

Serving at scale: autoscaling on tokens

Inside one replica, a serving engine batches requests, manages the KV cache and streams tokens (serving models shows how). The cloud problem is everything around the replicas: how many to run, where, and how to route requests among them.

Routing. Round-robin works badly for language models: requests differ in cost by orders of magnitude, and a replica that already holds a conversation's prefix in its KV cache answers it far more cheaply. Model-aware gateways route on queue length, KV-cache occupancy and prefix affinity; examples include the Kubernetes Gateway API Inference Extension, and serving stacks such as KServe, llm-d, NVIDIA Dynamo and Ray Serve, several of which also run disaggregated pools for the prefill and decode phases (DistServe, Splitwise).

Scaling signals. The CPU utilisation that drives an ordinary autoscaler (observability and autoscaling) says nothing here. What tracks load is tokens: tokens per second generated, requests and tokens waiting in the queue, the fraction of KV-cache memory in use, and the latency targets themselves — time to first token and time per output token.

Cold starts. A new replica must get a machine, start a container image that is often over 10 GB, load the weights — 140 GB for a 70-billion-parameter model in BF16 — and warm up the engine. From object storage at 1 GB/s the weights alone take over two minutes; from a cold start with no cached image, ten minutes is not unusual. Teams shorten this with weights cached on local NVMe, parallel streaming from object storage, pre-pulled images, warm standby replicas and, increasingly, snapshots of an initialised process.

The simulation serves one synthetic day of traffic for a chat model, minute by minute, including a 90-minute spike in the afternoon. Each replica is an 8-GPU server that sustains 6,000 output tokens per second within its latency target (an illustrative figure for a mid-sized model). The autoscaler asks for enough replicas to run at a target fraction of capacity; new replicas serve only after a cold start, and idle ones are released only after 15 quiet minutes.

python
import math
import numpy as np

# One synthetic day of demand for a chat model, in output tokens per second, minute by minute:
# a daily cycle, noise, and a spike at 14:00 when a news story sends users to the app.
minutes = np.arange(24 * 60)
rng = np.random.default_rng(7)
demand = 40_000 + 30_000 * np.sin((minutes / 60 - 9) / 24 * 2 * np.pi)       # peak mid-afternoon
demand *= rng.normal(1.0, 0.04, minutes.size)
demand[14 * 60:14 * 60 + 90] *= 1.6                                           # a 90-minute spike

CAPACITY = 6_000         # output tokens/s one replica (an 8-GPU server) sustains within its latency target
PRICE = 8 * 3.00         # USD per replica-hour: 8 GPUs at an example USD 3.00 per GPU-hour

def run(target, cold_start, scale_down_after=15):
    """Each minute, ask for enough replicas to run at `target` of capacity. New replicas serve only after
    `cold_start` minutes; surplus ones are removed after a quiet spell."""
    ready, booting, quiet = 4, [], 0          # replicas serving; start times of replicas booting
    replica_minutes, short_minutes, unserved = 0, 0, 0.0
    for m in minutes:
        ready += sum(1 for s in booting if s + cold_start == m)
        booting = [s for s in booting if s + cold_start > m]
        want = math.ceil(demand[m] / (CAPACITY * target))
        if want > ready + len(booting):
            booting += [m] * (want - ready - len(booting))
            quiet = 0
        elif want < ready:
            quiet += 1
            if quiet >= scale_down_after:     # stabilisation window: don't remove capacity on a blip
                ready, quiet = max(want, 1), 0
        replica_minutes += ready + len(booting)   # booting replicas are billed too
        served = min(demand[m], ready * CAPACITY)
        if served < demand[m]:
            short_minutes += 1
            unserved += (demand[m] - served) * 60
    tokens = demand.sum() * 60 - unserved
    cost = replica_minutes / 60 * PRICE
    return replica_minutes / 60, short_minutes, unserved / (demand.sum() * 60), cost / tokens * 1e6

peak = math.ceil(demand.max() / CAPACITY)
static_cost = peak * 24 * PRICE / (demand.sum() * 60) * 1e6
print(f"tokens in the day: {demand.sum() * 60 / 1e9:.2f} billion; the peak needs {peak} replicas\n")
print(f"{'policy':40s} {'replica-h':>9s} {'short min':>9s} {'unserved':>9s} {'USD / M tokens':>15s}")
print(f"{'static, sized for the peak':40s} {peak * 24:9d} {0:9d} {0:9.2%} {static_cost:15.3f}")
for target, cold in [(0.7, 10), (0.9, 10), (0.7, 2), (0.9, 2)]:
    hours, short, lost, usd = run(target, cold)
    print(f"{f'autoscale: target {target:.0%}, cold start {cold:2d} min':40s} {hours:9.0f} {short:9d} {lost:9.2%} {usd:15.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Provisioning for the peak all day costs USD 3.29 per million output tokens at these example prices; autoscaling at a 70% target cuts that to about USD 1.78, and at 90% to about USD 1.41, because the fleet shrinks in the night trough. The price is paid in the spike: with a ten-minute cold start the fleet is short of capacity for ten minutes, and at the 90% target 0.41% of the day's tokens wait beyond the latency target (in a real system they queue, and users see slow responses). Cutting the cold start to two minutes removes most of that harm at almost no cost — usually a better investment than a lower target. A busy service keeps a floor of warm replicas; a rarely used model may scale to zero and accept a slow first request.

The night trough has a second use. Many model APIs sell batch processing — results within hours rather than seconds — at a large discount, which lets the provider fill the troughs of its daily curve with work that can wait.

ExplanationH

Capacity: scarcity, reservations and spot

Since 2023 the newest accelerators have been, at times, simply unavailable on demand in the quantities teams want. Prices for older generations fell steeply in 2024–25 as supply caught up, but each new generation has repeated the pattern, and a request for hundreds of GPUs in one region, on one fabric, at short notice still often fails. Providers ration with quotas and sell certainty in several forms:

OptionWhat you getFits
On demandPay by the second or hour, if capacity exists; quotas applyExperiments, small serving fleets
Spot or preemptibleDeep discounts; the provider can reclaim the machine with seconds to minutes of warningCheckpointed training, batch inference, evaluation
Short-term reservationA cluster for days or weeks, booked ahead or queued until capacity frees (e.g. AWS Capacity Blocks for ML, Google Cloud's Dynamic Workload Scheduler, Azure capacity reservations)A training run with a deadline
Commitment of one to three yearsThe lowest hourly price, paid whether used or notThe steady base of a serving fleet; labs with continuous training

The arithmetic of commitment is the break-even rule from cost and FinOps: a commitment at a fraction cc of the on-demand price pays off for capacity that would be busy more than a fraction cc of the time. The serving simulation above is a typical shape: a base of replicas busy around the clock (commit), a daily swell (on demand), and spikes (on demand, plus whatever the cold start allows).

Example— Spot for a fine-tuning run

A fine-tuning run needs 400 GPU-hours. On demand, at an example USD 3.00 per GPU-hour, that is USD 1,200. On spot at USD 1.20, suppose the run is interrupted on average once every 10 hours and each interruption loses 20 minutes of work since the last checkpoint plus 10 minutes to restart: about 5% extra time, so 420 GPU-hours, USD 504. The saving is real only because the job checkpoints often and restarts automatically. Without checkpoints, a run longer than the typical time between interruptions might never finish at all.

Capacity is ultimately limited by power. New AI data-centre campuses are measured in hundreds of megawatts, racks of rack-scale GPU systems draw over 100 kW each and need liquid cooling, and in several regions the wait for a grid connection now sets the pace of construction. Lumenary EE's powering data centres explains why, and the sustainable cloud lesson counts the energy and carbon.

Worked examplesH

What a token costs

Everything in this lesson ends in one number. For a deployment that rents GG accelerators at price pp per accelerator-hour and generates rr tokens per second when busy, at average utilisation uu (the fraction of paid time the capacity is doing useful work),

cost per million tokens=G p3600 r u×106.\text{cost per million tokens} = \frac{G \, p}{3600 \, r \, u} \times 10^6 .

Example— One server, three ways

An 8-GPU server at an example USD 3.00 per GPU-hour costs USD 24 an hour. At 6,000 tokens per second it could make 21.6 million tokens an hour, so at full utilisation the hardware costs about USD 1.11 per million tokens. At 70% average utilisation it is USD 1.59; at the 34% of the simulation's fleet sized for its peak, USD 3.29. A one-year commitment at 60% of the on-demand price brings the full-utilisation figure to USD 0.67 — but only for capacity that really is used.

The formula hides the levers that move rr: batch size, quantisation, speculative decoding and prefix caching, all from serving models, and the routing and disaggregation above. Real prices also cover input tokens (cheaper, because prefill is compute-efficient), networking, storage, redundancy across zones, staff and margin. The discipline is the same as for any cloud cost: measure tokens served per accelerator-hour as a first-class metric, alongside latency.

ExplanationH

The professional view

Teams that run AI infrastructure well tend to share a few habits.

  • Benchmark before committing. Measure tokens per dollar, or time per training step, for your own model on each candidate instance type, including accelerators from different vendors; peak figures rarely predict it.
  • Treat topology as a resource. Ask for placement groups or topology blocks, and check with a short communication benchmark that the job got the fabric it asked for.
  • Make jobs interruptible. Frequent asynchronous checkpoints and automatic restarts turn spot capacity and hardware failures from disasters into delays.
  • Queue, don't hoard. A shared queue with quotas and backfilling gets far more out of a fixed reservation than teams holding machines "just in case".
  • Plan capacity like a supply chain: reservations for the base, on demand and spot for the rest, and more than one region — often more than one accelerator family — so that one shortage does not stop the work.
Common misconceptionsH

Common misconceptions

Misconception

"A GPU instance is just a VM with a GPU." For training, the network adapters and the fabric they connect to matter as much as the GPUs; the same eight GPUs on an oversubscribed or cross-zone network can be a fifth slower or worse.

Misconception

"Kubernetes will schedule my distributed training job." The default scheduler places pods independently, which can leave half-started jobs holding GPUs indefinitely. Distributed jobs need gang semantics, from Slurm, Kueue, Volcano or upstream Kubernetes' newer gang-scheduling features.

Misconception

"Higher GPU utilisation always means a cheaper service." Pushing the autoscaling target from 70% to 90% saved about a fifth of the cost in the simulation, but more than doubled the tokens that missed their latency target in the spike. Cost per token and latency must be read together.

Misconception

"The fastest chip gives the cheapest tokens." What matters is tokens per dollar on your model with your software, and availability where you need it. A previous-generation accelerator that is plentiful and cheap often wins.

Exercises

Exercises

1. Infrastructure for AI in the cloud

CoreHuman

A 70-billion-parameter model is trained with data parallelism on 16 servers of 8 GPUs. Its BF16 gradients (2 bytes per parameter, 140 GB) are all-reduced hierarchically: first inside each server, so each GPU is left responsible for one-eighth of the gradients, then by one ring per rail across the 16 servers, at 50 GB/s per GPU. Ignoring latency and the work inside servers, how many milliseconds does the across-server ring take? Use the ring cost 2 n−1n SB2\,\frac{n-1}{n}\,\frac{S}{B} for nn members, SS bytes and bandwidth BB.

2. Infrastructure for AI in the cloud

CoreComputational

Implement the EASY backfilling test. Write can_backfill(now, free, head_need, running, cand_need, cand_hours) returning True if a waiting candidate job may start now without delaying the job at the head of the queue.

  • free: servers free now; the head job needs head_need > free, so it cannot start now.
  • running: a list of (end_time, servers) for running jobs; their servers become free at end_time.
  • The shadow time is the earliest end time at which free plus the servers released so far reaches head_need; the spare servers are those still free at that moment once the head job has taken what it needs.
  • The candidate may start if it fits in free now and either finishes by the shadow time (now + cand_hours <= shadow) or needs no more than the spare servers.
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

3. Infrastructure for AI in the cloud

Warm-upComputational

Write cost_per_million(gpus, price_per_gpu_hour, tokens_per_second, utilisation) returning the hardware cost in US dollars of one million generated tokens, for a deployment of gpus accelerators that generates tokens_per_second when busy and is busy a fraction utilisation of the paid time. Raise ValueError if utilisation is not in (0,1](0, 1].

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

4. Infrastructure for AI in the cloud

CoreHuman

You run a language-model serving deployment on Kubernetes and want to autoscale its replicas. Which signal is the best basis for scaling?

Choose one

5. Infrastructure for AI in the cloud

Warm-upHuman

A cluster has 8 free servers. Two distributed training jobs, each needing 6 servers (one pod per server), are submitted at the same moment to a scheduler that places pods one at a time, alternating between the jobs. Each job's pods wait at a start-up barrier until all its pods are running. What happens?

Choose one

6. Infrastructure for AI in the cloud

CoreHuman

An accelerator costs USD 3.00 per hour on demand, or an effective USD 1.95 per hour with a one-year commitment that is paid for every hour, used or not. What is the minimum percentage of hours the accelerator must be doing useful work for the commitment to be the cheaper choice? (Answer as a whole-number percentage.)

7. Infrastructure for AI in the cloud

StretchHuman

A fine-tuning run needs 100 hours of work on one 8-GPU server. On spot capacity at USD 1.20 per GPU-hour, it is interrupted 10 times in total; each interruption loses on average 30 minutes of work since the last checkpoint and 10 minutes to restart. What is the expected total cost in US dollars on spot? (On demand at USD 3.00 per GPU-hour it would be USD 2,400.)

Further reading and resources