In this note
- Step 1: the memory floor
- Step 2: the compute a run needs
- Step 3: what a GPU actually sustains
- Step 4: from GPU-hours to a GPU count
- Worked examples on HGX B300 and GB300 NVL72
- Why real clusters land at 256 to 1,024 GPUs
- Failures, checkpoints and goodput
- A sizing worksheet
- Building a training cluster with us
The number of GPUs you need to train an LLM is set by your deadline, not by the model's size. Memory sets a floor, and for most models it's low: one 8-GPU HGX B300 node holds the full training state of a model up to about 100B parameters. Compute sets the total work. Pretraining a 70B dense model on 15T tokens is about 6.3 × 10²⁴ FLOPs, or roughly 1.9M B300 GPU-hours at BF16 on our assumptions. Divide that by the weeks you're willing to wait and you have the GPU count: about 1,000 GPUs for a three-month run, while 64 GPUs would take nearly four years.
That division is why a dedicated training cluster for anything short of a frontier model usually lands between 256 and 1,024 Blackwell GPUs, or 32 to 128 HGX nodes, or 4 to 14 GB300 NVL72 racks. Below that, pretraining anything bigger than a small model takes most of a year. Above it, failures, communication and the size of the commitment grow faster than the weeks you save.
This note walks through the method in four steps, then works it for an 8B dense model, a 70B dense model and a 671B-total mixture-of-experts (MoE) model with 37B active parameters. For models under about 10B parameters, see small language model training cost.
Step 1: the memory floor
Training with mixed-precision Adam keeps 16 bytes per parameter on the GPUs: 2 for the BF16 weights, 2 for the gradients and 12 for the FP32 master weights and the two Adam moments. That's the 2Ψ + 2Ψ + 12Ψ accounting from the ZeRO paper (opens in a new tab), and FP8 recipes keep the same master weights and optimizer states, so it holds for both precisions. Sharded across the cluster with ZeRO-3 or FSDP, those model states should take no more than about 70% of aggregate GPU memory, which leaves the rest for activations, communication buffers and fragmentation.
An HGX B300 node has 8 × 288 GB = 2,304 GB of HBM3e, so 70% of it holds the states of about 100B parameters. A GB300 NVL72 rack has 72 × 288 GB, about 20.7 TB, which holds about 900B. A MoE model is sized on total parameters, because every expert stays resident even though each token only uses a few.
| Model | Total parameters | Model states at 16 bytes | Minimum HGX B300 nodes | Minimum GB300 NVL72 racks |
|---|---|---|---|---|
| 8B dense | 8B | 128 GB | 1 | 1 |
| 70B dense | 70B | 1.1 TB | 1 | 1 |
| 405B dense | 405B | 6.5 TB | 5 | 1 |
| MoE, 37B active | 671B | 10.7 TB | 7 | 1 |
| MoE, about 30B active | 1T | 16 TB | 10 | 2 |
These are floors, and nobody trains at them. They matter in two places: for large MoE models, where the expert weights push the minimum to several nodes before compute enters the picture, and for long-context training, where activations grow with sequence length and eat into the 30% headroom. For dense models up to 70B the memory floor is one node, and the real question is time.
Step 2: the compute a run needs
A forward and backward pass costs about 6 FLOPs per parameter per token: 2 forward, 4 backward. Training compute is therefore C ≈ 6 × N × D, where N is the parameter count and D the number of training tokens. For a MoE model N is the active count, because only the routed experts run for each token. The approximation leaves out attention's own FLOPs, which matter at long sequence lengths, so treat it as a floor.
The token count is the input most people underestimate. The Chinchilla paper found that compute-optimal training scales parameters and tokens together, at about 20 tokens per parameter: 1.4T tokens for a 70B model (Hoffmann et al. (opens in a new tab)). Teams that will serve the model for years train far past that, because a smaller model trained longer is cheaper to run. Llama 3.1 was pretrained on about 15T tokens at every size (Meta (opens in a new tab)), which is about 1,900 tokens per parameter for the 8B. DeepSeek-V3 used 14.8T (DeepSeek (opens in a new tab)) and Qwen3 36T (Qwen (opens in a new tab)). For a from-scratch pretraining run in 2026, plan on 15T tokens or more.
| Run | N (active) | D | 6 × N × D |
|---|---|---|---|
| 8B dense, pretraining | 8B | 15T | 7.2 × 10²³ |
| 70B dense, continued pretraining | 70B | 1T | 4.2 × 10²³ |
| 70B dense, pretraining | 70B | 15T | 6.3 × 10²⁴ |
| 671B MoE, pretraining | 37B | 15T | 3.3 × 10²⁴ |
| 405B dense, pretraining | 405B | 15.6T | 3.8 × 10²⁵ |
Meta reports 3.8 × 10²⁵ FLOPs for Llama 3 405B (Meta (opens in a new tab)), which matches the last row. Note where the MoE lands: 671B total parameters, but half the compute of the 70B dense model, because each token only touches 37B.
Step 3: what a GPU actually sustains
Divide the FLOPs by what each GPU sustains, which is its dense peak times model FLOPs utilization (MFU), the share of the peak that turns into model FLOPs.
NVIDIA publishes Tensor Core peaks with sparsity and says dense is half the figure. HGX B300 lists 36 PFLOPS BF16 and 72 PFLOPS FP8 for eight GPUs (NVIDIA (opens in a new tab)), so 2,250 TFLOPS BF16 and 4,500 FP8 dense per GPU. GB300 NVL72 lists 360 and 720 PFLOPS for 72 GPUs (NVIDIA (opens in a new tab)), so 2,500 and 5,000.
We plan at 40% MFU for dense models at BF16 and 30% for MoE, where routing and all-to-all traffic cost more. At FP8 we use 30% and 20%, against an FP8 peak counted at no more than twice BF16, because attention, normalization, communication and the optimizer stay in higher precision. That makes FP8 about 1.5 times as fast as BF16 for a dense model and 1.33 times for MoE.
| Sustained TFLOPS per GPU | Dense, BF16 (40%) | Dense, FP8 (30%) | MoE, BF16 (30%) | MoE, FP8 (20%) |
|---|---|---|---|---|
| HGX B300 | 900 | 1,350 | 675 | 900 |
| GB300 NVL72 | 1,000 | 1,500 | 750 | 1,000 |
Checking the assumptions against published runs
Meta measured 38% to 43% BF16 MFU on Llama 3 405B across 8,192 to 16,384 H100s (Meta (opens in a new tab)), in line with the 40% dense assumption. Over the whole run the average is lower. 3.8 × 10²⁵ FLOPs over the 30.84M GPU-hours on the model card (Meta (opens in a new tab)) is about 340 TFLOPS per GPU, 35% of the H100's 989 TFLOPS dense BF16 peak (NVIDIA (opens in a new tab)), or 33% if you count 15T tokens instead of Meta's FLOP figure. Our 40% with the 10% goodput allowance in step 4 comes to 36%, close to that.
The 70B is a warning. 6.3 × 10²⁴ FLOPs over its 7.0M reported GPU-hours is about 250 TFLOPS per GPU, or 25% of peak. Model-card hours can include more than the final run, but if your stack lands there instead of at 40%, every time below is 1.6 times longer.
For MoE, DeepSeek-V3's pretraining took 2.664M H800 GPU-hours for 14.8T tokens with 37B active parameters (DeepSeek (opens in a new tab)). By 6 × N × D that's about 340 TFLOPS per GPU, around 17% of the H100's 1,979 TFLOPS dense FP8 peak; the H800 differs from the H100 in NVLink speed and FP64, not in Tensor Core throughput (OEM product guide (opens in a new tab)). That's a little under the 20% MoE FP8 assumption.
NVIDIA's tuned benchmarks for its NeMo 26.02 containers give the ceiling on Blackwell (NVIDIA (opens in a new tab)). They count attention FLOPs, which 6 × N × D leaves out, and they're short steady-state runs on reference systems, so production runs land below them.
| NVIDIA benchmark, TFLOPS per GPU | GPUs | DGX B300 | DGX GB300 | Share of FP8 dense peak |
|---|---|---|---|---|
| Llama 3 70B, FP8 | 64 | Not published | 2,353 | 47% on GB300 |
| DeepSeek-V3, MXFP8 | 256 | 806 | 1,219 | 18% on B300, 24% on GB300 |
| Qwen3 235B-A22B, MXFP8 | 256 | 391 | 974 | 9% on B300, 19% on GB300 |
| Qwen3 30B-A3B, MXFP8 | 8 | 678 | 700 | 15% on B300, 14% on GB300 |
For dense models our assumptions sit well under NVIDIA's ceiling, which leaves room for evaluation passes, data loading and longer sequences. For large MoE on HGX B300 they don't: the measured 806 is below the 900 we assume. If you plan a large fine-grained MoE on HGX nodes, use the lower figure, about 12% more time, or look at NVL72 racks.
Step 4: from GPU-hours to a GPU count
GPU-hours are the FLOPs divided by sustained FLOPS per GPU and by 3,600. Wall-clock time is GPU-hours divided by the GPU count. Turned around for a deadline:
GPUs needed = 6 × N × D ÷ (peak × MFU × 3,600) ÷ (weeks × 168 × goodput)
Goodput is the share of wall-clock time spent making forward progress, after failures, restarts and checkpoint writes. We use 90%, which is what Meta reports for Llama 3 ("higher than 90% effective training time", Meta (opens in a new tab)). The section on failures below shows where that number comes from.
| GPUs for the deadline, HGX B300, 90% goodput | GPU-hours | 8 weeks | 12 weeks | 16 weeks |
|---|---|---|---|---|
| 8B dense, 15T tokens, BF16 | 222,000 | 184 (23 nodes) | 122 (16 nodes) | 92 (12 nodes) |
| 70B dense, 1T tokens continued, BF16 | 130,000 | 107 (14 nodes) | 71 (9 nodes) | 54 (7 nodes) |
| 70B dense, 15T tokens, BF16 | 1.94M | 1,607 (201 nodes) | 1,072 (134 nodes) | 804 (101 nodes) |
| 70B dense, 15T tokens, FP8 | 1.30M | 1,072 (134 nodes) | 714 (90 nodes) | 536 (67 nodes) |
| 671B MoE, 37B active, 15T tokens, FP8 | 1.03M | 850 (107 nodes) | 566 (71 nodes) | 425 (54 nodes) |
| 405B dense, 15.6T tokens, BF16 | 11.7M | 9,673 | 6,448 | 4,836 |
The last row is why frontier dense models train on tens of thousands of GPUs: Meta ran Llama 3 405B on up to 16,384 H100s. Everything above it fits in 1,024 GPUs within about three months.
Worked examples on HGX B300 and GB300 NVL72
Planned weeks for each run, at the MFU assumptions above and 90% goodput. FP8 runs assume your recipe is validated at FP8; DeepSeek-V3 was trained that way.
| Cluster | GPUs | 8B, 15T, BF16 | 70B, 1T continued, BF16 | 70B, 15T, BF16 | 70B, 15T, FP8 | 671B MoE, 15T, FP8 |
|---|---|---|---|---|---|---|
| 8 HGX B300 nodes | 64 | 23 | 13.4 | 201 | 134 | 106 |
| 32 HGX B300 nodes | 256 | 5.7 | 3.3 | 50 | 33 | 27 |
| 64 HGX B300 nodes | 512 | 2.9 | 1.7 | 25 | 16.7 | 13.3 |
| 128 HGX B300 nodes | 1,024 | 1.4 | 0.8 | 12.6 | 8.4 | 6.6 |
| 4 GB300 NVL72 racks | 288 | 4.6 | 2.7 | 40 | 27 | 21 |
| 8 GB300 NVL72 racks | 576 | 2.3 | 1.3 | 20 | 13.4 | 10.6 |
| 14 GB300 NVL72 racks | 1,008 | 1.3 | 0.8 | 11.5 | 7.7 | 6.1 |
8B dense from scratch
222,000 B300 GPU-hours at BF16. 32 nodes finish in under six weeks, and 64 nodes in under three. An 8B model doesn't need a big cluster to train once; it needs one to train many times, across data mixes, tokenizers and ablations. Meta's model card puts Llama 3.1 8B at 1.46M H100-hours, which by the same arithmetic is about 14% of peak. Small models spread over very large clusters lose efficiency: per-GPU work shrinks and the communication doesn't. For 8B, 64 to 256 GPUs is the efficient range, and the SLM training cost note works through the rest.
70B dense
Continued pretraining on 1T tokens of domain data is a three-week job on 32 nodes. Pretraining from scratch on 15T tokens is a different size of problem: 1.94M GPU-hours at BF16. That's about a year on 32 nodes, six months on 64 and 12.6 weeks on 128, or 8.4 weeks at FP8. It's the clearest case for the top of the range. The full model states, 1.1 TB, fit in one node; the parallelism layout is about speed. A typical layout is tensor parallelism of 8 inside each node's NVLink domain, pipeline parallelism across a handful of nodes, and data parallelism over the rest, as in Meta's 405B configurations (TP 8, PP 16, DP 64 to 128).
671B-total MoE with 37B active
At FP8, 1.03M B300 GPU-hours: 6.6 weeks on 128 nodes at our assumption, or about 7.4 weeks at NVIDIA's measured B300 throughput for DeepSeek-V3. On 14 GB300 racks it's 6.1 weeks, and NVIDIA's benchmarks suggest the rack's advantage here is larger than the 11% the peaks show. The 10.7 TB of states set a floor of 7 B300 nodes, but the experts are what make the fabric matter: every token is routed to experts that may sit on other GPUs, and that all-to-all traffic stays on NVLink inside an NVL72 rack but crosses the network on HGX.
Why real clusters land at 256 to 1,024 GPUs
The tables explain the lower bound. Runs that are worth a multi-year commitment in 2026 (pretraining at 8B to 70B dense, MoE at 20B to 40B active, continued pretraining of large open-weights models on hundreds of billions to trillions of tokens) need 0.1M to 2M B300 GPU-hours. For a main run of 8 to 16 weeks, the deadline table above puts that at roughly 50 to 1,600 GPUs. Below 256 GPUs, anything beyond an 8B pretraining run or a continued-pretraining job takes the better part of a year, and the model is out of date before it ships.
The upper bound has four parts:
- Failures scale with GPU count. Meta measured a mean time to failure of 7.9 hours for 1,024-GPU jobs on its research clusters, against 47.7 days for 8-GPU jobs, and projects 1.8 hours at 16,384 GPUs (Meta (opens in a new tab)). Above a few thousand GPUs, restart engineering becomes a team of its own.
- Efficiency falls. Past some point you can't add data parallelism without growing the global batch beyond what trains well, and communication takes a larger share of each step.
- The cluster runs all year. A 1,024-GPU cluster finishes the 70B FP8 run about six times a year. Most teams can't feed it more than one flagship run, the ablations before it and the fine-tunes after.
- The commitment. Doubling the cluster doubles the GPU-hours you commit to and saves half the weeks. At around 1,000 GPUs the next halving is rarely worth the next doubling.
So a typical program is one cluster sized for its main run, with the same capacity spending the rest of the year on ablations, evaluation, post-training and then serving the result.
Failures, checkpoints and goodput
The 90% goodput allowance in step 4 is a planning number. A synchronous training job loses time in three ways: blocking while it writes checkpoints, redoing the work since the last checkpoint after an interruption, and the restart itself. The Young-Daly rule puts the best checkpoint interval at the square root of 2 × the blocking save time × the mean time between interruptions.
Interruptions grow with node count. Meta's Llama 3 run had 419 unexpected interruptions in 54 days on 16,384 H100s (Meta (opens in a new tab)), about 3.8 per 1,000 node-days. At that rate a 128-node cluster sees one about every two days, and at the 6.50 per 1,000 node-days of Meta's research clusters about one a day. Counted per job, which adds filesystem, driver and code failures, Meta measured a mean time to failure of 7.9 hours for 1,024-GPU jobs (Meta (opens in a new tab)). All of this is A100 and H100 data; nobody has published fleet figures for Blackwell yet.
Checkpoints are large. A checkpoint holds the BF16 weights, the FP32 master weights and the Adam moments but not the gradients, so it's about 14 bytes per parameter rather than the 16 held in GPU memory: about 1 TB for a 70B model and 9.4 TB for the 671B MoE. NVIDIA's reference architecture recommends 40 GB/s of aggregate write per 72-node B300 scalable unit at its standard level (NVIDIA (opens in a new tab)). At one unit's rate a synchronous save blocks training for about 25 seconds on the 70B and about 4 minutes on the MoE. Asynchronous checkpointing, which copies the state to host memory and writes it in the background, cuts the block to seconds.
| 128 HGX nodes, 1,024 GPUs, Llama 3 interruption rate | Goodput (our estimate) |
|---|---|
| Asynchronous checkpoints, hot spare node | 97.5% |
| Synchronous checkpoints blocking 5 minutes, hot spare node | 92.8% |
| Synchronous checkpoints blocking 5 minutes, no spare, 24-hour repair | 73.5% |
The cases and their assumptions are worked through in GPU cluster SLA and goodput. Meta reported more than 90% effective training time on Llama 3, and 0.85 to 0.9 for the largest jobs on its research clusters. So 90% is a cautious middle: a cluster with fast checkpoints and a healthy spare ready does better, and one that waits days for a repair does much worse. At 1,024 GPUs the five points between the first two rows are about four days of the 12.6-week 70B run. How fast a failed node is replaced is an operations question as much as a hardware one, and it belongs in the contract.
A sizing worksheet
- Total parameters × 16 bytes, divided by 70% of the memory per node or rack, gives the minimum unit count. Check it, then mostly ignore it.
- Pick the token budget: 15T or more for pretraining, hundreds of billions to a few trillion for continued pretraining.
- 6 × active parameters × tokens gives the FLOPs.
- Divide by dense peak × MFU (start at 40% dense BF16, 30% dense FP8, 30% MoE BF16, 20% MoE FP8) and by 3,600 for GPU-hours.
- Divide by the weeks you'll accept × 168 × 0.9 for the GPU count, then round up to whole nodes or racks.
- If it's over about 1,000 GPUs, check whether FP8, a longer deadline or a smaller model with more tokens gets you there first.
- Size the year, not the run: count the ablations, evaluations, fine-tunes and serving that will share the cluster.
Building a training cluster with us
We design, source, build and operate NVIDIA GPU clusters, from 8 HGX nodes (64 GPUs) or one NVL72 rack (72 GPUs). Most clusters are 32 to 128 nodes; larger ones are scoped case by case. The fabric is RoCE v2 Ethernet, spine-leaf, designed from the start for the node count you plan to reach, and large clusters are handed over in monthly waves of 32 nodes, so training can start on the first wave. You can own the cluster and have us build it or run it, or reserve one we hold. See HGX B300 and GB300 NVL72, or send us your model, token budget and deadline.
Questions this note answers
Memory sets a low floor: one 8-GPU HGX B300 node holds the full training state of a model up to about 100B parameters. The deadline sets the real number. Pretraining a 70B dense model on 15T tokens takes about 1.9M B300 GPU-hours at BF16, so finishing in about three months takes around 1,000 GPUs. Below the frontier labs, dedicated training clusters usually land between 256 and 1,024 Blackwell GPUs.
On our assumptions, pretraining a 70B dense model on 15T tokens needs about 1.94M HGX B300 GPU-hours at BF16 or 1.30M at FP8. That is about 12.6 weeks on 1,024 GPUs (128 nodes) at BF16, or 8.4 weeks at FP8, allowing 10% for failures and checkpoints. Continued pretraining of a 70B model on 1T tokens needs about 130,000 GPU-hours, roughly three weeks on 256 GPUs.
About 16 bytes per parameter with mixed-precision Adam: 2 for BF16 weights, 2 for gradients, 4 for FP32 master weights and 8 for the two Adam moments. A 70B model carries about 1.1 TB of model states before activations. Sharded with ZeRO-3 or FSDP, keep the states to about 70% of aggregate GPU memory so activations and buffers fit.
It needs fewer GPU-hours per token, because compute follows the active parameters, but more memory, because every expert is resident. A 671B-total MoE with 37B active trains on 15T tokens in about 1.03M B300 GPU-hours at FP8, less than a 70B dense model, but its 10.7 TB of model states need at least 7 HGX B300 nodes. MoE also runs at lower utilization because of all-to-all traffic between experts.
For dense models up to about 70B the difference is small: GB300 has about 11% more dense compute per GPU and the same 288 GB of memory. For large MoE models it can be large. NVIDIA's own benchmarks show GB300 at 1.5 times B300's per-GPU throughput on DeepSeek-V3 and about 2.5 times on Qwen3 235B, mostly because expert-parallel traffic stays inside the 72-GPU NVLink domain.
Sources
- Meta: The Llama 3 Herd of Models (MFU table, interruptions, effective training time) (opens in a new tab)
- Meta: Llama 3.1 70B model card (training GPU-hours) (opens in a new tab)
- DeepSeek-V3 technical report (parameters, tokens, H800 GPU-hours) (opens in a new tab)
- Qwen3 technical report (36T pretraining tokens, 235B-A22B) (opens in a new tab)
- Hoffmann et al.: Training Compute-Optimal Large Language Models (Chinchilla) (opens in a new tab)
- Rajbhandari et al.: ZeRO, memory optimizations for training (16 bytes per parameter) (opens in a new tab)
- Meta: Revisiting Reliability in Large-Scale ML Research Clusters (MTTF, ETTR) (opens in a new tab)
- NVIDIA HGX platform: HGX B300 specifications (opens in a new tab)
- NVIDIA GB300 NVL72 specifications (opens in a new tab)
- NVIDIA H100 specifications (opens in a new tab)
- OEM product guide: NVIDIA H800 (H800 vs H100 differences) (opens in a new tab)
- NVIDIA Megatron Bridge: performance summary (NeMo 26.02 containers) (opens in a new tab)
- NVIDIA DGX SuperPOD B300 reference architecture: storage guidelines (opens in a new tab)