Machine learning infrastructure · Advanced · 35 min read
How to Estimate LLM Training Cost and Set a Spending Cap

A practical, reproducible way to turn a token target into GPU-hours, runtime, cost, and a budget guardrail that survives uncertain hardware utilization.
Training cost is rarely determined by the advertised hourly price alone. The difficult part is estimating how much useful work a cluster will complete per second when communication, evaluation, checkpointing, data loading, and interruptions are included. A calculation that assumes peak hardware performance can be precise to several decimal places and still be operationally unsafe.
This guide builds a calculator from first principles. It estimates compute from the number of training tokens and model parameters, converts compute into time using model FLOPs utilization (MFU), adds explicit non-training overhead, and applies a price model. It then turns the result into a spending cap for a single job.
What the estimate must answer
Before launching a run, you should be able to answer five separate questions:
- How many training tokens will the job process?
- How much training compute does the model and objective require?
- What effective throughput can the selected cluster sustain?
- How many GPU-hours and wall-clock hours does that imply?
- What hard limit should stop the job if reality becomes more expensive than planned?
The last question is the one most estimates omit. A forecast is an expectation. A spending cap is an operational control. You need both.
The cost model
For a dense decoder-only language model, a useful first-order estimate is:
training_flops = 6 × parameters × training_tokens
The factor 6 is a conventional approximation for forward and backward computation in a dense transformer training step. It is a model, not a measurement. The true value depends on architecture, sequence length, attention implementation, vocabulary operations, activation recomputation, sparsity, auxiliary losses, and other details.
Let:
Pbe the number of model parameters;Tbe the number of training tokens;Gbe the number of GPUs;Fbe the advertised or documented peak FLOPs per GPU for the selected precision;mbe MFU as a fraction, such as0.35for 35%;rbe the fraction of wall-clock time spent doing useful training compute;pbe the price per GPU-hour.
Then the effective cluster throughput is:
effective_flops_per_second = G × F × m × r
Estimated training time is:
training_seconds = training_flops / effective_flops_per_second
And the direct GPU cost is:
gpu_hours = training_seconds × G / 3600
gpu_cost = gpu_hours × p
There are two equivalent ways to represent non-training overhead. You can reduce useful-time utilization with r, or calculate ideal compute time and multiply it by an overhead multiplier. This article uses r because it makes the operational assumption visible:
wall_clock_seconds = training_flops / (G × F × m × r)
If you also pay for attached CPU instances, storage, networking, licenses, or reserved capacity, add those as separate line items. Do not hide them inside the GPU price unless your billing model genuinely bundles them.
A concrete planning case
Consider a hypothetical pre-launch plan:
| Input | Illustrative value | Why it matters |
|---|---|---|
| Model size | 13 billion parameters | Sets the dominant compute term |
| Training data | 250 billion tokens | Sets the number of optimization tokens |
| GPU count | 64 | Determines parallel throughput and GPU-hours |
| Peak per-GPU throughput | 312 teraFLOPs/s | Must match the selected precision and hardware |
| MFU scenarios | 25%, 35%, 45% | Captures uncertainty in sustained training efficiency |
| Useful-time ratio | 90% | Allows for evaluation, checkpointing, and stalls |
| GPU price | $2.00 per GPU-hour | Placeholder input, not a verified price |
The purpose of this case is not to predict a real bill. It is to show how the same job can produce materially different runtimes and costs when MFU changes. The calculator below lets you replace every input.
A reproducible Python calculator
Save the following as training_cost.py. It uses only the Python standard library and prints a sensitivity table. The script intentionally does not fetch provider prices or infer hardware specifications. You supply those inputs from your own quote, contract, benchmark, or measured run.
from dataclasses import dataclass
from typing import Iterable
@dataclass
class Scenario:
mfu: float
useful_time_ratio: float
@dataclass
class Estimate:
mfu_percent: float
compute_flops: float
wall_clock_hours: float
gpu_hours: float
gpu_cost: float
cost_with_reserve: float
def estimate_training_cost(
parameters: float,
training_tokens: float,
gpu_count: int,
peak_flops_per_gpu: float,
gpu_price_per_hour: float,
scenario: Scenario,
reserve_fraction: float = 0.0,
) -> Estimate:
if parameters <= 0:
raise ValueError("parameters must be positive")
if training_tokens <= 0:
raise ValueError("training_tokens must be positive")
if gpu_count <= 0:
raise ValueError("gpu_count must be positive")
if peak_flops_per_gpu <= 0:
raise ValueError("peak_flops_per_gpu must be positive")
if gpu_price_per_hour < 0:
raise ValueError("gpu_price_per_hour cannot be negative")
if not 0 < scenario.mfu <= 1:
raise ValueError("mfu must be in the interval (0, 1]")
if not 0 < scenario.useful_time_ratio <= 1:
raise ValueError("useful_time_ratio must be in the interval (0, 1]")
if reserve_fraction < 0:
raise ValueError("reserve_fraction cannot be negative")
# Conventional dense-transformer approximation.
compute_flops = 6.0 * parameters * training_tokens
effective_flops_per_second = (
gpu_count
* peak_flops_per_gpu
* scenario.mfu
* scenario.useful_time_ratio
)
wall_clock_seconds = compute_flops / effective_flops_per_second
wall_clock_hours = wall_clock_seconds / 3600.0
gpu_hours = wall_clock_hours * gpu_count
gpu_cost = gpu_hours * gpu_price_per_hour
cost_with_reserve = gpu_cost * (1.0 + reserve_fraction)
return Estimate(
mfu_percent=scenario.mfu * 100.0,
compute_flops=compute_flops,
wall_clock_hours=wall_clock_hours,
gpu_hours=gpu_hours,
gpu_cost=gpu_cost,
cost_with_reserve=cost_with_reserve,
)
def print_sensitivity(
parameters: float,
training_tokens: float,
gpu_count: int,
peak_flops_per_gpu: float,
gpu_price_per_hour: float,
scenarios: Iterable[Scenario],
reserve_fraction: float,
) -> None:
print("MFU wall_hours GPU_hours GPU_cost cost_with_reserve")
print("-" * 62)
for scenario in scenarios:
result = estimate_training_cost(
parameters=parameters,
training_tokens=training_tokens,
gpu_count=gpu_count,
peak_flops_per_gpu=peak_flops_per_gpu,
gpu_price_per_hour=gpu_price_per_hour,
scenario=scenario,
reserve_fraction=reserve_fraction,
)
print(
f"{result.mfu_percent:4.0f}% "
f"{result.wall_clock_hours:11.2f} "
f"{result.gpu_hours:10.2f} "
f"${result.gpu_cost:9.2f} "
f"${result.cost_with_reserve:17.2f}"
)
if __name__ == "__main__":
# Illustrative inputs only. Replace with documented or measured values.
parameters = 13e9
training_tokens = 250e9
gpu_count = 64
peak_flops_per_gpu = 312e12
gpu_price_per_hour = 2.00
scenarios = [
Scenario(mfu=0.25, useful_time_ratio=0.90),
Scenario(mfu=0.35, useful_time_ratio=0.90),
Scenario(mfu=0.45, useful_time_ratio=0.90),
]
print_sensitivity(
parameters=parameters,
training_tokens=training_tokens,
gpu_count=gpu_count,
peak_flops_per_gpu=peak_flops_per_gpu,
gpu_price_per_hour=gpu_price_per_hour,
scenarios=scenarios,
reserve_fraction=0.20,
)
Run it with:
python training_cost.py
Do not copy the example output into a budget document without recording the exact inputs. The output is only as reproducible as the parameter count, token count, hardware throughput, price, and utilization assumptions supplied to it.
Reading the MFU sensitivity analysis
MFU is the ratio between achieved model FLOPs per second and the hardware’s theoretical peak FLOPs per second. It is a useful planning variable because it compresses many implementation details into one measurable efficiency number. It is not the same as device utilization reported by a monitoring tool. A GPU can show high activity while the training step still achieves relatively low model FLOPs.
With all other inputs fixed, runtime and direct GPU cost are approximately inversely proportional to MFU:
cost ∝ 1 / MFU
runtime ∝ 1 / MFU
That means moving from 25% to 35% MFU does not make the run 10% cheaper. The relative cost is:
cost_at_35 / cost_at_25 = 0.25 / 0.35
Likewise, a fall from 35% to 25% increases the compute portion of the bill by the reciprocal ratio. The practical lesson is simple: choose a conservative MFU for the cap, and treat better MFU as upside rather than money already saved.
A useful sensitivity table has at least three rows:
- Floor: a strong, measured result from a comparable workload;
- Expected: the value you use for the planning forecast;
- Guardrail: a lower value used to size the maximum exposure.
If you have no measurement, do not describe an assumed MFU as a benchmark. Label it as an assumption and widen the range. A short pilot on the target model, sequence length, parallelism, and precision is usually more informative than a generic number from another stack.
From an estimate to a spending cap
A spending cap should be derived from the worst acceptable scenario, not the average scenario. One practical formulation is:
cap =
worst_case_gpu_cost
+ non_gpu_cost
+ restart_allowance
+ operational_reserve
For the GPU portion, use:
worst_case_gpu_cost =
6 × parameters × tokens
--------------------------------------------- × GPU_price
GPU_count × peak_FLOPs × guardrail_MFU × useful_time_ratio
× GPU_count / 3600
The GPU count cancels algebraically in the idealized formula. That does not mean cluster size is irrelevant. Larger jobs may have lower MFU, more communication overhead, more failure exposure, and higher minimum billing increments. Keep GPU_count in the calculator so that the wall-clock estimate remains visible and so that you can add cluster-level costs later.
Use an explicit reserve instead of a vague “round number.” For example:
| Component | Purpose | How to set it |
|---|---|---|
| Guardrail MFU | Protects against inefficient training | Use a conservative measured or declared assumption |
| Useful-time ratio | Accounts for non-training time | Include evaluation, saves, input stalls, and synchronization |
| Restart allowance | Accounts for failed or preempted attempts | Estimate from the failure policy and checkpoint interval |
| Operational reserve | Absorbs estimation error | Choose a documented percentage or fixed amount |
Example policy, stated as an example rather than a universal rule:
Stop the job when projected final cost exceeds the approved cap. The cap is calculated using the lower MFU scenario, the measured useful-time ratio, one planned restart, and a separately documented reserve. Any cap increase requires a new estimate and approval.
Turning the cap into a runtime control
A number in a spreadsheet does not stop a cluster. The cap needs to be connected to the job’s control plane.
At minimum, the launcher should know:
- the maximum wall-clock duration;
- the maximum number of training steps or tokens;
- the target price per GPU-hour;
- the number of GPUs actually allocated;
- the current accumulated GPU-hours;
- the approved cost ceiling.
The safest stop condition is based on projected final cost, not only elapsed time. Let C_now be the cost accumulated so far, t_now the elapsed training time, and t_total the planned total time. A simple projection is:
projected_cost = C_now × (t_total / t_now)
This assumes the recent rate is representative. Recalculate after warm-up and after major changes such as a new data phase, evaluation schedule, or cluster shape. If the projection exceeds the cap, finish the current safe checkpoint and stop.
Pseudocode for a training loop:
while tokens_seen < target_tokens:
train_one_step()
tokens_seen += tokens_in_step
if should_checkpoint():
save_checkpoint()
if elapsed_seconds() >= max_wall_clock_seconds:
save_checkpoint()
stop("wall-clock cap reached")
current_cost = gpu_count * elapsed_seconds() / 3600 * gpu_price
projected_cost = current_cost * target_tokens / max(tokens_seen, 1)
if projected_cost > spending_cap:
save_checkpoint()
stop("projected spending cap exceeded")
This is deliberately incomplete infrastructure pseudocode. Your launcher must account for billing granularity, queued time, startup charges, retries, and whether the provider bills allocated or actively used resources. Verify those rules against the provider’s current contract or billing records before relying on the control.
What to measure during a pilot
A pilot is not merely a smaller training run. It is an opportunity to replace assumptions with measurements that match the final configuration.
Record, at minimum:
- Model configuration and parameter count.
- Sequence length, batch size, and gradient accumulation.
- Precision and activation checkpointing settings.
- Parallelism dimensions and number of GPUs.
- Tokens processed per optimizer step.
- Steady-state step time after warm-up.
- Time spent in evaluation, checkpointing, and validation.
- Data-loader wait time and failed steps.
- Peak memory and any out-of-memory recovery.
- Measured billing rate and billed duration, if available.
Estimate steady-state throughput from a stable interval rather than from the first few steps:
tokens_per_second =
tokens_processed_in_interval / interval_seconds
Then compare observed throughput with the theoretical compute model. If the model has P parameters and processes q tokens per second, an approximate achieved model FLOPs rate is:
achieved_flops_per_second ≈ 6 × P × q
Divide that by the cluster’s theoretical peak FLOPs to obtain an approximate MFU. Treat the result as a diagnostic estimate. It inherits the assumptions in the six-FLOP approximation and should be compared only across runs using the same definition.
Common failure cases
Using peak FLOPs as delivered throughput
Peak FLOPs are a hardware ceiling under a particular precision and operation mix. They do not include communication, memory movement, kernel launch gaps, synchronization, or input stalls. Use peak FLOPs as an input to the model, then discount it with measured or conservative MFU.
Confusing GPU utilization with MFU
A monitoring dashboard may show a high utilization percentage while the workload performs less useful model computation than expected. Validate the metric definition and calculate MFU from the same model, precision, and token accounting used by the training job.
Counting tokens incorrectly
“Dataset size” can mean raw characters, documents, tokenized examples, or tokens presented to the optimizer. The calculator needs optimizer-visible training tokens. Confirm whether padding, packing, repeated samples, validation tokens, and discarded batches are included.
Ignoring evaluation and checkpoint time
Periodic evaluation and checkpoint saves can be substantial, particularly when evaluation runs on the same allocation or checkpoints are written over a constrained network. Include them in the useful-time ratio or model them as separate wall-clock intervals.
Assuming linear scaling across GPU counts
Doubling GPUs may reduce wall-clock time, but it does not guarantee doubled throughput. Communication and synchronization can reduce MFU. Measure the intended cluster shape or use a lower guardrail for larger allocations.
Budgeting only the successful run
Failed launches, preemptions, corrupted checkpoints, and debugging runs consume resources. Make restart allowance explicit. If failure probability is unknown, do not bury it in an unexplained percentage; record the policy and the reason for the allowance.
Using stale prices
Provider pricing, discounts, regions, commitment terms, minimum billing units, and taxes can change. The calculator deliberately accepts a price as an input. Attach the source and retrieval date to your internal estimate, but do not treat the example price in this article as a current quote.
Verification checklist before launch
Use this checklist as a release gate for the training job:
- Token count comes from the actual tokenization and packing pipeline.
- Parameter count matches the model configuration that will launch.
- Peak FLOPs match the GPU model, precision, and documented definition.
- MFU scenarios are labeled as measured, estimated, or hypothetical.
- Useful-time ratio includes evaluation, checkpointing, loading, and synchronization.
- GPU price and billing unit are recorded from the relevant account or contract.
- Non-GPU charges are itemized separately.
- Restart and preemption assumptions are documented.
- Maximum wall-clock time is configured in the scheduler or launcher.
- Checkpointing can resume the job without restarting from zero.
- Projected cost is recalculated from observed throughput during the run.
- Exceeding the cap triggers a safe stop and preserves the latest checkpoint.
Limitations of the calculator
This method is intentionally transparent rather than exhaustive. It is appropriate for a first-order budget and for comparing scenarios, but it does not replace a benchmark on the target stack.
Important limitations include:
- The six-FLOP rule is an approximation for dense transformer training.
- Mixture-of-experts and other sparse architectures need a more specific compute model.
- Sequence-length effects can change attention and memory costs.
- MFU can vary across warm-up, curriculum phases, sequence lengths, and batch shapes.
- GPU-hour cost may not equal total cloud cost.
- Provider billing may include idle, queued, startup, storage, or network charges.
- Failure risk cannot be inferred from hardware specifications alone.
- Spot, reserved, and on-demand capacity have different interruption and pricing behavior.
When the decision is financially material, run a pilot that uses the production model, precision, sequence length, parallelism, data path, and checkpoint policy. Use the pilot to calibrate MFU and the useful-time ratio, then rerun the calculator with those measured values.
A compact operating procedure
- Tokenize or otherwise determine the exact optimizer-visible token target.
- Enter parameter count, GPU count, peak FLOPs, and the verified price into the calculator.
- Run at least three MFU scenarios and one useful-time scenario that includes operational overhead.
- Run a target-stack pilot and replace assumptions with measured throughput where possible.
- Choose a guardrail scenario for the spending cap.
- Add itemized restart, non-GPU, and operational reserves.
- Configure wall-clock and projected-cost stop conditions.
- Log inputs, assumptions, code version, and approval before launch.
- Recalculate during training using observed tokens per second and accumulated cost.
- Stop safely when the cap is projected to be exceeded.
Final takeaway
The most dependable answer to “how much will this LLM training run cost?” is not a single number. It is a reproducible model with explicit inputs, a sensitivity range, a measured pilot, and a stop condition.
Start with the compute approximation, make MFU uncertainty visible, account for non-training time, and use the conservative scenario to set the cap. Then connect that cap to the scheduler and checkpoint system. This turns a fragile estimate into a controlled experiment: the job can improve on the forecast, but it cannot silently spend beyond the approved exposure.
Continue with the training and infrastructure guides or look up related terms in the Agent Lab Journal glossary.