In this note
- Three ways to get a domain model
- How many tokens a small model needs
- From FLOPs to GPU-hours
- What real runs report
- Wall-clock and cluster size
- The final run is a fraction of the compute
- Distillation and pruning
- What a year of SLM work needs
- Five years of GPU-hours
- SLM vs LLM: why go small
- SLM training cost for a memo
- A cluster for an SLM program
Training a 7B model from scratch on 20 trillion tokens takes about 8.4 × 10²³ FLOPs, or about 259,000 B300 GPU-hours at 40% model FLOPs utilization (MFU). That's 42 days on a 32-node HGX B300 cluster. Small models sustain less than that once downtime is counted, so we plan at an effective 25%. The ablations, restarts and evaluations around a final run add more than half again, which brings a realistic plan for one 7B model to about 650,000 GPU-hours. Continued pretraining of an existing 7B on 1 trillion domain tokens is about 13,000 GPU-hours, and a full fine-tune on 20 billion tokens about 260.
Those numbers span three orders of magnitude, and most of the decision about building your own small language model (SLM) is which one you need. We count cost in GPU-hours and cluster time; multiply by whatever a GPU-hour costs you, owned or reserved. Figures we computed are estimates, with assumptions next to them.
Three ways to get a domain model
| Approach | What it changes | Typical tokens | 7B, B300 GPU-hours at 40% MFU |
|---|---|---|---|
| Pretraining from scratch | Everything: data, tokenizer, architecture, license | 1T to 36T | 259,000 at 20T |
| Continued pretraining | Adds domain knowledge to an open model's weights | 10B to 1T | 13,000 at 1T |
| Full fine-tune | Behavior, format and task skill | 1B to 20B | 260 at 20B |
| LoRA or QLoRA adapter | Behavior on a small slice of the weights | Under 1B to a few B | One node, hours |
A full fine-tune costs about 0.1% of a from-scratch run, and continued pretraining about 5%.
We'd start most domain projects with continued pretraining and fine-tuning on a strong open model. Pretraining from scratch earns its cost in four cases:
- You need to know and document every training token, for licensing, copyright or a regulator.
- Your domain needs its own tokenizer: genomic or chemical sequences, source code in an unusual language, or a script open tokenizers split badly.
- You want an architecture tuned to your serving target, such as a latency budget, a context length or edge hardware.
- You'll ship many versions, so the data pipeline and training recipe are a product you're building anyway.
How many tokens a small model needs
The Chinchilla paper (opens in a new tab) found that for a fixed training budget, parameters and tokens should grow together, which works out to about 20 tokens per parameter. Chinchilla itself was 70B parameters on 1.4 trillion tokens. Small models today are trained far past that:
| Model | Parameters | Training tokens | Tokens per parameter |
|---|---|---|---|
| Chinchilla (2022) | 70B | 1.4T | 20 |
| Gemma 3 12B | 12B | 12T | 1,000 |
| Gemma 3 4B | 4B | 4T | 1,000 |
| Gemma 3 1B | 1B | 2T | 2,000 |
| Llama 3.1 8B | 8B | about 15T | about 1,900 |
| SmolLM3 | 3B | 11.2T | about 3,700 |
| Llama 3.2 1B | 1B | up to 9T | up to 9,000 |
| Qwen3 family (0.6B to 235B) | 36T corpus | up to 4,500 at 8B |
Sources: Gemma 3 (opens in a new tab), Llama 3.1 (opens in a new tab), SmolLM3 (opens in a new tab), Llama 3.2 (opens in a new tab), Qwen3 (opens in a new tab). Ratios use nominal sizes.
Chinchilla minimizes training compute for a given quality. It ignores inference. Sardana et al. (opens in a new tab) redid the analysis with inference included and found that a team expecting around a billion inference requests should train smaller and longer than Chinchilla-optimal, with quality still improving at up to 10,000 tokens per parameter. You pay for the extra tokens once and get a cheaper model on every call afterwards. That's the reason to build an SLM in the first place.
The catch is data. Few domains have trillions of tokens of their own. BloombergGPT (opens in a new tab), a 50B finance model, mixed a 363 billion token financial dataset with 345 billion general tokens. A from-scratch domain SLM is mostly general web, code and synthetic data with the domain mixed in, and the mix is what the ablations are for. Repetition helps a little: Muennighoff et al. (opens in a new tab) found up to four epochs of repeated data barely changes the loss compared with unique data, but the value of more repetition decays toward zero after that.
From FLOPs to GPU-hours
Training compute is about 6 × N × D FLOPs, where N is the parameter count (active parameters for a mixture-of-experts model) and D the number of tokens. Divide by sustained throughput per GPU, which is the dense peak times MFU, to get GPU-hours. We use NVIDIA's dense peaks: 2.25 PFLOPS at BF16 and 4.5 PFLOPS at FP8 for both B300 and B200. At 40% MFU for a dense model at BF16 and 30% at FP8, one B300 GPU-hour delivers about 3.2 × 10¹⁸ model FLOPs at BF16 and 4.9 × 10¹⁸ at FP8.
| Model | Chinchilla, 20 tokens per param | Pretrain, 20T | Continued, 1T | Full fine-tune, 20B | Pretrain 20T wall-clock, 32 nodes |
|---|---|---|---|---|---|
| 1B | 37 | 37,000 | 1,850 | 37 | 6 days |
| 3B | 330 | 111,000 | 5,560 | 111 | 18 days |
| 7B | 1,810 | 259,000 | 13,000 | 259 | 42 days |
| 14B | 7,260 | 519,000 | 25,900 | 519 | 84 days |
B300 GPU-hours, BF16, 40% MFU, dense models. FP8 at 30% MFU cuts every figure by a third. Our estimate.
Pretraining on 20T tokens costs 70 times (14B) to 1,000 times (1B) more than training the same model to Chinchilla, so the token budget matters more than the model size. Memory isn't the constraint. Mixed-precision Adam holds about 16 bytes per parameter, so a 14B model carries 224 GB of weights, gradients and optimizer states, which fits comfortably in one 8-GPU node of either platform. On NVIDIA's dense figures B300 and B200 do the same training work per GPU-hour. B300's 288 GB per GPU matters more for long context and for serving.
What real runs report
40% is what a well-tuned large run reaches. Meta (opens in a new tab) reported 38% to 43% BF16 MFU for Llama 3 405B on 8,192 to 16,384 H100s. Small models are harder to keep busy, because their matrix multiplies are small and communication takes a larger share of each step. Dividing published GPU-hours by 6 × N × D against the H100's 989 TFLOPS dense BF16 peak gives an effective utilization that also includes downtime, restarts and whatever else the team counted:
| Run | Parameters | Tokens | Reported H100 GPU-hours | Effective utilization (our arithmetic) |
|---|---|---|---|---|
| Llama 3.1 405B | 405B | about 15T | 30.84M | about 33% |
| SmolLM3, main run | 3B | 11.2T | 276,480 | about 20% |
| Llama 3.1 8B | 8B | about 15T | 1.46M | about 14% |
| Llama 3.2 1B | 1B | up to 9T | 370,000 | about 5% |
The SmolLM3 team says it tuned its setup for about 30% MFU on 384 GPUs and that state-of-the-art training reaches 20% to 41% end to end (Smol Training Playbook (opens in a new tab)). Llama 3.2 1B is low for a different reason. Meta pruned it from Llama 3.1 8B and fed logits from the 8B and 70B models into pretraining, so the GPU-hours include running the teachers (model card (opens in a new tab)).
SmolLM3 held about 30% MFU while running and 20% over its whole allocation, which is where we put a tuned small-model stack: an effective 25%. For planning, take the 40% GPU-hours and multiply by 1.6, the step from 40% to 25%, and plan higher only once your own stack has shown it can hold more. A first run on a new stack can land nearer Llama 3.1 8B's 14%. Our cluster sizing note uses 40% MFU and 90% goodput, about 36% effective, because it sizes larger dense runs that keep GPUs busier.
Wall-clock and cluster size
Most teams pick a cluster size for wall-clock rather than memory: how long they're willing to wait for a run, and how many runs a year they want.
| Cluster | GPUs | 7B on 20T, at 40% | 7B on 20T, at 25% | 14B on 20T, at 40% | 14B on 20T, at 25% |
|---|---|---|---|---|---|
| 8 HGX B300 nodes | 64 | 169 days | 270 days | 338 days | 540 days |
| 32 nodes | 256 | 42 days | 68 days | 84 days | 135 days |
| 64 nodes | 512 | 21 days | 34 days | 42 days | 68 days |
| 128 nodes | 1,024 | 11 days | 17 days | 21 days | 34 days |
Our estimate, from the GPU-hours above.
The table assumes speed scales linearly with GPUs, and for small models it eventually stops. Every GPU needs a share of the global batch, and past the batch size that still trains well, adding GPUs mostly adds communication. SmolLM3 trained a 3B model on 384 GPUs. We'd treat a few hundred GPUs as a sensible ceiling for one 1B to 3B run and use the rest of a large cluster for other runs at the same time.
The final run is a fraction of the compute
The GPU-hours in a model card or paper are usually for the final run only. DeepSeek says plainly that its V3 training cost (opens in a new tab) of 2.788M H800 GPU-hours covers only the official training, not the earlier research and ablations on architecture, algorithms and data. Hugging Face published the full ledger for SmolLM3:
| SmolLM3 phase | GPUs | Days | H100 GPU-hours | Share |
|---|---|---|---|---|
| Main pretraining run | 384 | 30 | 276,480 | 63% |
| Pretraining ablations | 192 | 15 | 69,120 | 16% |
| Mid-training ablations | 192 | 10 | 46,080 | 11% |
| Training reset and debugging | 384 / 192 | 3 / 4 | 46,080 | 11% |
| Total | 437,760 |
From the Smol Training Playbook (opens in a new tab). It ran over 100 ablations, many of them on the full 3B architecture for 50 to 100 billion tokens, and puts evaluation at just under 10,000 more GPU-hours. Ablations and debugging came to 58% of the main run.
At the frontier the ratio is much steeper. Epoch AI (opens in a new tab) estimates that final training runs took 9.6% of OpenAI's R&D compute in 2024, and 22.6% and 12.3% at MiniMax and Z.ai going by their IPO filings. Labs that follow a known recipe can skip much of the exploration, and a domain SLM team mostly does. We'd plan on these multipliers:
| Planning factor | Multiplier on the 40% MFU GPU-hours | Basis |
|---|---|---|
| Effective utilization (40% to 25%) | 1.6× | Public small-model runs, table above |
| Ablations, resets, debugging | 1.4× to 1.8× | SmolLM3: 1.58× |
| Evaluation during training | about 1.03× | SmolLM3: under 10,000 of 438,000 GPU-hours |
| One from-scratch model, end to end | about 2.5× (2× to 3×) | Our estimate |
A team doing real architecture research, or its first run on a new stack, should plan nearer Epoch's range, where the final run is a quarter of the compute or less.
Distillation and pruning
The cheapest way to a good small model is often a bigger one. NVIDIA's Minitron (opens in a new tab) work pruned a pretrained 15B model to 8B and 4B and retrained them with distillation, using up to 40 times fewer training tokens per model than training from scratch and 1.8 times less compute for the whole family. All Gemma 3 (opens in a new tab) models are trained with distillation. For post-training, the Qwen3 (opens in a new tab) report measured 1,800 GPU-hours for on-policy distillation against 17,920 for reinforcement learning, with better results from distillation.
Distillation has its own compute bill, which the 6 × N × D formula doesn't show. Each token needs a forward pass through the teacher, about 2 FLOPs per teacher parameter. Logit distillation from a 70B teacher over a 7B student's whole corpus adds about 3.3 times the student's own training compute (2 × 70 against 6 × 7). The usual ways down are to distill on part of the corpus, to store the teacher's top logits once and reuse them across runs, or to use a teacher only a few times the student's size.
What a year of SLM work needs
A dedicated cluster runs every hour of the year. Each node gives 70,080 GPU-hours a year, so 8 nodes give about 561,000, 32 nodes 2.24M and 64 nodes 4.49M. These three programs use the 2.5× factor for every training run:
| Program, per year | Training GPU-hours, with 2.5× | Cluster | Share of the cluster's year |
|---|---|---|---|
| Adapt open models: 4 continued-pretraining runs of a 7B on 1T, 50 full fine-tunes on 20B | about 162,000 | 8 nodes | 29% |
| Build a domain SLM: 3B and 7B from scratch on 20T, twice a year, plus the adaptation work | about 2.01M | 32 nodes | 90% |
| Ship a family: 1B, 3B, 7B and 14B on 20T once a year, plus the adaptation work | about 2.48M | 64 nodes | 55% |
Our estimate, B300 or B200 at BF16.
The first program doesn't need a training cluster at all; the cluster is sized by inference, and training fills the nights. The second is about right for 32 nodes with little slack. The third could fit on 32 nodes by GPU-hours, but its 14B run would take 135 days there at 25% effective utilization. At 64 nodes it takes 68 days, and the other 2M GPU-hours a year serve the models and fund the next round of ablations.
Five years of GPU-hours
A multi-year cluster, owned or reserved, is easiest to compare in GPU-hours:
| Cluster | GPUs | GPU-hours a year | GPU-hours over 60 months |
|---|---|---|---|
| 8 nodes | 64 | 561,000 | 2.80M |
| 32 nodes | 256 | 2.24M | 11.2M |
| 64 nodes | 512 | 4.49M | 22.4M |
| 128 nodes | 1,024 | 8.97M | 44.9M |
On a dedicated cluster the run itself has no separate price; it uses a share of hours you're paying for either way. The 650,000 GPU-hours of a from-scratch 7B are about 6% of a 32-node cluster's five years. Idle hours cost the same as busy ones, which is why the programs above put serving and evaluation on the same capacity.
SLM vs LLM: why go small
Training cost is a one-off. The reason to go small is the cost of every call after it.
- Serving cost. NVIDIA researchers put serving a 7B model at 10 to 30 times cheaper than a 70B to 175B model in latency, energy and FLOPs (Belcak et al. (opens in a new tab)).
- Latency. Generation at small batch is bound by memory bandwidth, because every new token reads the full weights. A 7B model at BF16 is 14 GB, so on one B300 at 8 TB/s the ceiling is about 570 tokens a second per sequence. A 70B at 140 GB tops out near 57. These are upper bounds that ignore KV cache reads and kernel overhead, but the tenfold gap holds.
- Iteration. A full 7B fine-tune on 20B tokens takes about a day and a half on one node, and continued pretraining on 1T tokens about two days on 32 nodes, so you can retrain when your data or policy changes rather than once a year.
- Data control. You hold the weights, and on a single-tenant cluster the training data stays in your environment.
What you give up is breadth. A 1B to 15B model trained for a narrow job can match a much larger one on that job and fall well short on open-ended reasoning, long-horizon agents and anything outside its training mix. Most teams end up with a few small models for the high-volume paths and a larger model for the rest.
SLM training cost for a memo
- From scratch on 20T tokens, B300 or B200 at 40% MFU: about 37,000 GPU-hours for 1B, 111,000 for 3B, 259,000 for 7B and 519,000 for 14B. Continued pretraining on 1T is about 5% of that, a full fine-tune on 20B about 0.1%.
- Plan 2× to 3× the final run for a from-scratch model once utilization, ablations, resets and evaluation are counted.
- Token budgets: Chinchilla is 20 tokens per parameter; current small models train on 1,000 to several thousand.
- Wall-clock for a 7B on 20T at an effective 25%: 68 days on 32 nodes, 34 days on 64.
- Five-year scale: 32 nodes are 11.2M GPU-hours, 64 nodes 22.4M.
A cluster for an SLM program
We design, source, build and operate single-tenant NVIDIA GPU clusters, from 8 HGX nodes (64 GPUs) or one NVL72 rack (72 GPUs), in the region you choose. Most clusters are 32 to 128 nodes; larger ones are scoped case by case. Large clusters are handed over in monthly waves of 32 nodes, and the first runs can start on the first wave. You can own the cluster and have us build it or run it, or reserve one we hold. See HGX B300 and HGX B200, or send us your model and token budget.
Questions this note answers
It depends mostly on the token budget. At 6 × parameters × tokens FLOPs and 40% MFU on B300, a 1B model on 20 trillion tokens is about 37,000 GPU-hours, a 3B about 111,000, a 7B about 259,000 and a 14B about 519,000. Real runs and the ablations around them take two to three times the final run's estimate. Multiply the result by what a GPU-hour costs you.
From scratch on 20 trillion tokens, about 259,000 B300 GPU-hours at 40% MFU, or about 650,000 once realistic utilization, ablations and restarts are counted. Continued pretraining on 1 trillion tokens is about 13,000 GPU-hours, and a full fine-tune on 20 billion tokens about 260.
Far more than the Chinchilla ratio of about 20 tokens per parameter. Recent small models are trained on 1,000 to several thousand tokens per parameter: Llama 3.1 8B on about 15 trillion tokens, SmolLM3 3B on 11 trillion and Gemma 3 4B on 4 trillion. The extra training costs once and makes every inference call cheaper for a given quality.
Much cheaper to adapt. For a 7B model, continued pretraining on 1 trillion domain tokens takes about 5% of the compute of pretraining on 20 trillion, and a full fine-tune about 0.1%. Training from scratch makes sense when you need control of every training token, your own tokenizer or architecture, or you will ship many versions.
Size, and what follows from it. A small language model, roughly 1B to 15B parameters, fits in one GPU at BF16, reads a tenth of the weights per generated token that a 70B model does, and can be retrained in days to weeks on a mid-sized cluster. It gives up breadth and hard reasoning, so it suits narrow, high-volume tasks.
On 20 trillion tokens at 40% MFU, about 42 days on 32 HGX B300 nodes (256 GPUs) and 21 days on 64 nodes. At an effective 25%, closer to what small-model runs sustain, plan on about 68 and 34 days. Continued pretraining on 1 trillion tokens takes about two days on 32 nodes.
Sources
- Hoffmann et al.: Training Compute-Optimal Large Language Models (Chinchilla, 2022) (opens in a new tab)
- Sardana et al.: Beyond Chinchilla-Optimal, accounting for inference in scaling laws (opens in a new tab)
- Muennighoff et al.: Scaling Data-Constrained Language Models (opens in a new tab)
- Meta: Llama 3.1 8B model card (training GPU-hours and tokens) (opens in a new tab)
- Meta: Llama 3.2 1B model card (GPU-hours, pruning and distillation) (opens in a new tab)
- Meta: The Llama 3 Herd of Models (BF16 MFU, Table 4) (opens in a new tab)
- Hugging Face: SmolLM3, smol, multilingual, long-context reasoner (opens in a new tab)
- Hugging Face: The Smol Training Playbook (SmolLM3 compute breakdown, MFU) (opens in a new tab)
- Qwen3 Technical Report (36T tokens, distillation vs RL GPU-hours) (opens in a new tab)
- Gemma 3 Technical Report (token budgets, distillation) (opens in a new tab)
- DeepSeek-V3 Technical Report (training cost, official run only) (opens in a new tab)
- Epoch AI: R&D compute vs final training runs (March 2026) (opens in a new tab)
- NVIDIA: Compact Language Models via Pruning and Knowledge Distillation (Minitron) (opens in a new tab)
- NVIDIA Research: Small Language Models are the Future of Agentic AI (opens in a new tab)
- Wu et al.: BloombergGPT, a large language model for finance (opens in a new tab)