Deployment

GPU cluster SLA: uptime vs training goodput

What uptime means for a GPU cluster and why goodput matters more: published failure rates from large training runs, what fails inside a node, interruptions by cluster size, checkpoint and spare-node math, and how to write an SLA for training and for inference.

All notesPublished 12 min read
In this note
  1. Uptime, availability and goodput
  2. Published GPU failure rates
  3. What fails inside a GPU node
  4. Interruptions by cluster size
  5. Why per-node uptime overstates cluster availability
  6. Checkpoints: what each interruption costs
  7. Training goodput for a 32 to 128-node cluster
  8. Spares, hot standby and replacement time
  9. How to write a GPU cluster SLA
  10. Questions to ask a GPU provider
  11. What Lightyard's SLA covers

A GPU cluster can meet a 99.9% uptime SLA on every node and still lose a tenth of its training time. A synchronous training job needs every node it was launched on, and each failure costs the whole cluster a restart plus the work done since the last checkpoint. Sixty-four nodes that are each up 99.9% of the time are all up together only about 94% of the time.

The number that matters for training is goodput: the share of the hours you pay for that turns into training progress you keep. Large published runs report 85% to 97%. On our estimates, a 64-node HGX cluster lands between about 85% and 98%, depending mostly on two things an SLA rarely mentions: how fast a failed node is replaced by a healthy one, and how long a checkpoint blocks the job.

The failure rates are published research. The goodput model is our estimate, with the assumptions in tables.

Uptime, availability and goodput

TermWhat it measuresWho controls it
Node availabilityShare of time one node is healthy and usableThe provider
Cluster availabilityShare of time at least the nodes your job needs are healthy togetherThe provider, and spare capacity
Effective training time ratio (ETTR)Share of wall-clock time spent on training steps that are keptProvider and your training stack
GoodputProductive training time over reserved time; used here the same way as ETTRBoth

Google's TPU fleet paper splits goodput into scheduling goodput (everything the job needs is available at once), runtime goodput (counted only up to the last saved checkpoint) and program goodput (how close the code runs to ideal) (Google (opens in a new tab)). A provider's SLA can only speak to the first. Your checkpoint and restart tooling decide most of the second.

Published GPU failure rates

None of the public data covers Blackwell yet.

SourceFleetWhat it reports
Meta, Llama 3 405B pre-training (opens in a new tab)16,384 H100, 54 days419 unexpected and 47 planned interruptions; about 78% of unexpected ones hardware; over 90% effective training time
Meta FAIR research clusters (opens in a new tab)16k and 8k A100, 11 months6.50 and 2.34 failures per 1,000 node-days; mean time to failure 47.7 days for 8-GPU jobs, 7.9 hours for 1,024-GPU jobs; ETTR 0.85 to 0.9 on the largest jobs
NCSA Delta, UIUC study (opens in a new tab)448 A100, 608 H100 (GH200)Node availability about 99.4% (A100) and 99.3% (H100); H100 memory errors 3.2 times as frequent per GPU as A100
ByteDance (opens in a new tab)9,600-GPU jobs97% ETTR on a three-month job, with warm standby machines
Google, Gemini training (opens in a new tab)TPU podsGoodput on the largest job raised from 85% to 97%; silent data corruption expected "every week or two"

The Llama 3 run is the best benchmark for a dedicated training cluster: one job, one cluster, mature tooling. It works out to about 3.8 unexpected interruptions per 1,000 node-days, or about 1.4 per 8-GPU node a year, 1.1 of them hardware, the rate our cost model uses. We use Meta's research-cluster rate of 6.50 as the pessimistic case. Its 7.9-hour figure for 1,024-GPU jobs is worse than hardware rates predict, because real jobs also fail on filesystems, drivers and their own code.

What fails inside a GPU node

The counts are Meta's, from the Llama 3 run; the fix column is our summary.

ComponentLlama 3 interruptions (of 419)Usual fix
Faulty GPU148Reset; if it repeats, swap the GPU or the node
GPU HBM3 memory72Row remapping or reset; swap the GPU if it keeps failing
Software bug and dependency56Your stack or a driver
Network switch or cable35Reseat or replace optic or cable; replace switch
Unplanned host maintenance32Drain and fix the host
GPU SRAM, system processor, thermal42Firmware or reset; swap if it repeats
NIC and NCCL watchdog timeouts14Replace NIC; find the straggler or bad link
Silent data corruption6Replay the step to find the GPU; swap it
SSD, power supply, chassis, CPU, memory, IO board14Field-replaceable part

Faulty GPUs and HBM alone were 220 of 419. Memory needs watching on newer parts: the Delta study found the 512 spare rows per GPU for remapping didn't grow with H100's larger memory, and only about 59% of remapping attempts succeeded (Delta study (opens in a new tab)).

Meta's research-cluster paper lists backend network links, filesystem mounts, GPU memory and PCIe errors as the leading attributed causes, and saw 50% to 75% bandwidth loss from link problems during bring-up (Meta FAIR (opens in a new tab)). A 64-node cluster with eight 800 Gb/s NICs per node has 512 compute links with an optic or cable at each end, and a flapping link usually shows up as a slow job, not a failed one.

The same paper found 40 "lemon" nodes, 1.2% and 1.7% of the two clusters, that kept failing jobs. Removing them cut the failure rate of jobs over 512 GPUs from 14% to 4%. A node that fails twice should be replaced, not repaired again, and the SLA should say so.

Blackwell adds a dedicated RAS (reliability, availability and serviceability) engine for early fault detection (NVIDIA (opens in a new tab)). Nobody has published fleet data on how much it helps.

Interruptions by cluster size

Mean time between interruptions (MTBI) is one over the job's node count times the per-node rate. Our projections from the two published rates:

ClusterGPUsMTBI at Llama 3 ratePer monthMTBI at Meta research ratePer month
16 HGX nodes12816.5 days1.89.6 days3.2
32 HGX nodes2568.2 days3.74.8 days6.3
4 NVL72 racks2887.3 days4.14.3 days7.1
64 HGX nodes5124.1 days7.42.4 days12.7
8 NVL72 racks5763.7 days8.32.1 days14.2
128 HGX nodes1,0242.1 days14.81.2 days25.3
Llama 3 scale16,3843.1 hours (observed)2361.8 hours (Meta's projection)405

A month is 730 hours. The H100 and A100 rates are applied to Blackwell unadjusted, for lack of public data. At 32 to 128 nodes that's between one interruption a week and one a day: manageable with automation, expensive without it.

Why per-node uptime overstates cluster availability

If each node is independently up a fraction a of the time, all N are up together a to the power N of the time. Spare nodes change the requirement from "all N" to "any N of N plus k".

Per-node availabilityNodes the job needsNo spare1 spare2 spares4 spares
99.9%6493.8%99.8%100.0%100.0%
99.9%12888.0%99.2%100.0%100.0%
99.5%6472.6%95.8%99.5%100.0%
99.3% (Delta H100)6463.8%92.4%98.9%100.0%
99.3% (Delta H100)12840.7%77.2%93.6%99.8%

Failures aren't fully independent, and a switch failure takes out several nodes at once, so treat this as the best case. A per-node percentage tells you little about a synchronous job. What decides cluster availability is how many healthy nodes stand by and how fast one replaces a failed node.

Checkpoints: what each interruption costs

Every interruption costs detection, restart, and the work since the last checkpoint, on average half the interval. Saving more often cuts rework but adds blocking time. The Young-Daly rule balances the two: the best interval is the square root of 2 times the blocking save time times the MTBI.

Checkpoints are large. Mixed-precision training with Adam keeps BF16 weights, FP32 master weights and two FP32 optimizer moments, about 14 bytes per parameter (our arithmetic; a 16-byte figure adds gradients, which aren't saved), so a 70B dense model's checkpoint is about 1 TB. NVIDIA's DGX B300 reference architecture recommends 40 GB/s of aggregate write per 72-node scalable unit at the standard level and notes that a synchronous checkpoint stops training while it writes (NVIDIA (opens in a new tab)). At that rate 1 TB takes about 25 seconds if storage is the only limit, and a large MoE takes minutes.

Asynchronous checkpointing copies the state to host memory and writes it in the background. ByteDance measured 0.01 to 0.04 seconds of blocking per save that way, against 6.8 to 13 seconds for a synchronous Megatron save (ByteDance (opens in a new tab)). Meta's research-cluster paper assumes 5 minutes per checkpoint and 5 to 20 minutes to restart (Meta FAIR (opens in a new tab)).

Training goodput for a 32 to 128-node cluster

Three cases, our estimates. A and B assume a healthy spare node replaces a failed one at restart, so the job always resumes at full size. C has no spare: when a node has to be pulled for repair, the job waits for it.

AssumptionA: async checkpoints, hot spareB: sync checkpoints, hot spareC: sync checkpoints, no spare
Interruption rateLlama 3 (Meta research rate in brackets)SameLlama 3
Blocking time per checkpoint30 seconds5 minutes5 minutes
Detection plus restart5 + 15 minutes10 + 30 minutes10 + 30 minutes
Checkpoint intervalYoung-Daly optimumYoung-Daly optimumYoung-Daly optimum
Interruptions that need a node pulledCovered by the spareCovered by the spare40%
Wait for the pulled nodeNoneNone8 hours (part on site) or 24 hours
ClusterABC, 8-hour repairC, 24-hour repair
32 nodes, 256 GPUs98.9% (98.5%)96.8% (95.6%)95.1%91.9%
64 nodes, 512 GPUs98.4% (97.7%)95.2% (93.5%)92.0%85.5%
128 nodes, 1,024 GPUs97.5% (96.4%)92.8% (90.1%)86.4%73.5%

Case B uses the checkpoint and restart times Meta's research-cluster paper assumes. Case A is close to the 97% ByteDance and Google report after years of tooling work. Case C with a multi-day RMA is why nobody runs a large job that way: teams hold spares or re-shard onto fewer nodes.

These figures leave out planned maintenance. Meta reports at least one interruption a day from firmware and kernel upgrades at 16,384 GPUs, so ask for rolling maintenance that drains a few nodes at a time.

Our B300 and GB300 NVL72 ownership models take 1% to 3% of GPU-hours off for failures, counting the days a node is out and a restart per event. A single synchronous job also loses the rework since its last checkpoint, which is why cases B and C fall further.

Each point of goodput on 512 GPUs is about 3,700 GPU-hours a month: US$3,700 for every US$1 of rate per GPU-hour, or about US$18,000 at US$4.84, the all-in cost per productive GPU-hour our model gives for owning B300s. The gap between cases A and B on 64 nodes is roughly US$58,000 a month at that rate, and it's recovered with checkpoint and restart tooling, not hardware.

Spares, hot standby and replacement time

A hot spare is a node that's powered, imaged, burned in and ready to join the job. ByteDance sizes its standby pool at the 99th percentile of simultaneous failures and brings standbys straight into the job instead of requeuing it, which it reports cut recovery time almost elevenfold (ByteDance (opens in a new tab)). The same sizing on Llama 3 rates, assuming 40% of interruptions pull a node (our estimate):

ClusterNodes pulled per monthSpares at P99, repair in 8 hoursRepair in 24 hoursRepair in 72 hours
32 nodes1.5111
64 nodes3.0112
128 nodes5.9123

The lease checklist takes the conservative view that every hardware interruption takes a node out. That roughly doubles the pulls, to about 3 a month at 32 nodes, and adds one spare to each row of the 72-hour column and to 64 nodes at 24 hours.

SLAs often blur two numbers. Time to restore is how long until a healthy node is back in your job: minutes if a hot spare is already running, hours if one has to be brought up. Time to repair is how long the broken node takes to come back: hours with parts on site, days with an RMA. Ask for both.

Then ask whose capacity the spare is. If it's one of your reserved nodes, a 64-node reservation trains on 63, about 1.6% of capacity, though the spare can run evaluations between failures. If the provider holds spares outside your reservation, you train on all 64.

NVL72 racks change the unit. NVIDIA's partition guide says a failed compute tray marks its four GPUs as having no NVLink while the partition stays healthy, so work can continue on the remaining GPUs. A failed NVLink switch tray marks every GPU connected to it the same way, and the partition can't run NVLink workloads until it's fixed (NVIDIA (opens in a new tab)). One option is to size jobs to 16 of the 18 compute trays, leaving two in-rack spares. Ask which NVLink software release the racks run, how it handles a failed switch tray, and how fast switch trays are replaced, since that sets the worst-case outage.

How to write a GPU cluster SLA

A general-purpose cloud SLA counts whether an instance responds. A GPU cluster SLA has to count whether the hardware is fast enough to train or serve on.

ClauseTraining clusterInference deployment
Unit measuredHealthy nodes in your reserved set, per node and as N of the reserved totalThe replica set behind each endpoint, and the capacity it serves
What healthy meansAll GPUs visible, diagnostics pass, NCCL bandwidth and fabric links within an agreed margin of the acceptance baselineServes the agreed load within the agreed latency
DegradedA slow node or flapping link counts as downCapacity below the agreed replica count counts as degraded
Time to respondMinutes, around the clockMinutes, around the clock
Time to restoreA healthy node back in the job, from a spareReplica back in service; with N+1 sizing, users shouldn't notice
Time to repairFailed node fixed or replaced, separately statedSame
Credit basisNode-hours below the commitment, at the GPU-hour rateHours below the availability or capacity commitment
Repeat failuresA node that fails twice is replaced, not repaired againSame
ReportingIncident report per event; monthly availability with the raw health dataSame, plus latency percentiles

For inference, design does most of the work. Size replicas for the busiest hour plus one spare node per model, and a node failure costs headroom, not availability.

Exclusions to read carefully

  • Scheduled maintenance: notice, hours a month, rolling or cluster-wide
  • Your software, images and drivers, and changes you request
  • Failures outside the provider's boundary, such as your own links into the cluster
  • Force majeure, and whether utility power events count as one
  • Credits as the sole remedy, and the cap on them

Questions to ask a GPU provider

  1. What failure rate do you see per node per year on this platform, and can we see the data behind it?
  2. Is availability measured per node or per cluster, what counts as healthy, and does a slow node count as down?
  3. What are the time to respond, time to restore and time to repair, and are they in the contract?
  4. How many spare nodes are on site for our cluster, and are they inside or outside our reservation?
  5. Which parts are held on site: GPUs or whole nodes, NICs, optics, power supplies, switch trays?
  6. How are switch and fabric failures handled, since they affect many nodes at once?
  7. What does burn-in cover before acceptance, and can we watch and re-run the tests?
  8. How many nodes can planned maintenance drain at once?
  9. What happens to a node that keeps failing, and what monitoring data do we get?

The GPU lease agreement checklist covers the rest of the contract.

What Lightyard's SLA covers

Service levels are written into each contract, so they're set in the order form you sign rather than published here. Every reservation, bare metal or managed, includes monitoring, replacement of failed parts and running the cluster to an SLA. Nodes are single-tenant; on bare metal you hold root on every node, and firmware and the BMC stay with us. See how delivery works, contract terms, or talk to us about the service levels your workload needs.

Questions this note answers

Sources

  1. Meta: The Llama 3 Herd of Models, section 3.3.4 reliability (2024) (opens in a new tab)
  2. Meta FAIR: Revisiting Reliability in Large-Scale ML Research Clusters (2024) (opens in a new tab)
  3. UIUC: Story of Two GPUs, resilience of H100 and A100 on NCSA Delta (SC '25) (opens in a new tab)
  4. ByteDance: Robust LLM Training Infrastructure at ByteDance (SOSP '25) (opens in a new tab)
  5. Google DeepMind: Gemini technical report, training infrastructure (opens in a new tab)
  6. Google: Machine Learning Fleet Efficiency with ML Productivity Goodput (2025) (opens in a new tab)
  7. NVIDIA DGX SuperPOD reference architecture, DGX B300: storage architecture (opens in a new tab)
  8. NVIDIA Blackwell architecture: RAS Engine (opens in a new tab)
  9. NVIDIA GB200 NVL Partition User Guide v1.2: partition fault handling (opens in a new tab)
Get started

Start with one rack.

Tell us the size and timing you have in mind. We usually reply within a few hours and come back with a written proposal.

sales@lightyard.ai