Systems · Inference performance
Estimating LLM Decode Speed from Memory Bandwidth

A useful first estimate for generation speed is often not the number of tensor cores. During autoregressive decode, a large language model (LLM) repeatedly streams its weights and reads the KV cache. If you can account for those bytes, you can estimate a traffic budget, an idealized tokens-per-second ceiling, the batch at which the cache becomes as expensive as the weights, and the context length where that transition occurs.
The short answer
For one generated token, approximate memory traffic as:
traffic/token ≈ weight_bytes + batch × context × kv_bytes/token
Then divide usable memory bandwidth by that traffic:
decode_tokens/s ≈ usable_bandwidth / traffic/token
This is a bandwidth-bound estimate, not a benchmark result. It assumes that the relevant weights and cache are streamed efficiently, that transfers overlap with computation, and that arithmetic, kernel launch overhead, communication, and scheduling do not become the limiting factors.
The estimate becomes especially useful when comparing data types. Lower-precision weights reduce the fixed cost paid by every generated token. Lower-precision KV cache reduces the cost that grows with batch size and context length.
What decode is actually moving
In a typical decoder-only transformer, one decode step consumes one new token for every active sequence. The model must apply every layer to the current hidden state. The weight matrices are reused across the batch, while attention reads the keys and values stored for previous tokens.
During prefill, many tokens are processed together and matrix multiplication can have high reuse. During decode, the new-token dimension is small. The workload often has low arithmetic intensity: relatively few operations are performed for each byte fetched from memory.
The model can therefore approach a roofline-model limit set by memory bandwidth before it reaches the accelerator's advertised compute throughput. This does not make compute irrelevant. It means that compute should be checked as a second bound after the traffic estimate.
Notation
| Symbol | Meaning |
|---|---|
W |
Bytes occupied by model weights that must be read per decode step |
B |
Active batch size: sequences producing a token in this step |
C |
Average cached context length per active sequence |
K |
KV-cache bytes read per sequence and cached token |
BW |
Usable device memory bandwidth in bytes per second |
T |
Estimated aggregate decode rate in tokens per second |
Derive the KV-cache term
Let a model have L transformer layers, Hkv
key/value heads, head dimension D, and
Skv bytes per cache scalar. Each cached token
stores both a key and a value:
K = 2 × L × Hkv × D × Skv
The factor of two accounts for keys and values. For grouped-query attention or multi-query attention, use the number of KV heads, not the number of query heads. Confusing those two values can overestimate cache traffic by a large factor.
For a decode step, a simple attention-read model is:
KV traffic/token ≈ B × C × K
This treats the full cached prefix as read for every active sequence. Implementations can differ in layout, padding, page size, quantization metadata, and cache reuse, so this term is an approximation rather than a byte-for-byte trace.
Worked case: an 80-layer GQA model
The following is a reproducible numerical example, not a claim about a measured product or a particular vendor's implementation. Assume:
- 80 transformer layers
- 8 KV heads
- 128 values per head
- 70 billion parameters
- 16-bit weights, so
W = 70 × 109 × 2 = 140 GBbefore format-specific overhead - 1.6 TB/s usable memory bandwidth
- batch
B = 8 - cached context
C = 4096tokens
With a 16-bit KV cache:
K = 2 × 80 × 8 × 128 × 2 = 327,680 bytes/token
The cache traffic for the batch is:
8 × 4096 × 327,680 = 10,737,418,240 bytes ≈ 10.74 GB
The total simplified traffic budget is therefore about
140 + 10.74 = 150.74 GB per aggregate decode step. The
bandwidth estimate is:
T ≈ 1,600 GB/s ÷ 150.74 GB ≈ 10.6 aggregate tokens/s
If all eight sequences are equally active, that corresponds to about
1.3 tokens/s per sequence. These numbers are arithmetic from
the assumptions above. They are not measured throughput, and they omit
quantization scales, allocator padding, cache metadata, kernel effects,
and any multi-device traffic.
Compare KV-cache dtypes
| Weights | KV cache | Weight traffic | KV traffic at B=8, C=4096 | Estimated aggregate rate |
|---|---|---|---|---|
| 16-bit | 16-bit | 140.00 GB | 10.74 GB | 10.6 tokens/s |
| 16-bit | 8-bit | 140.00 GB | 5.37 GB | 11.0 tokens/s |
| 8-bit | 16-bit | 70.00 GB | 10.74 GB | 19.8 tokens/s |
| 8-bit | 8-bit | 70.00 GB | 5.37 GB | 21.2 tokens/s |
The table assumes exactly one byte per scalar for the 8-bit cases and ignores scales and other metadata. Real quantized formats may have block scales, zero points, alignment requirements, or dequantization behavior that changes both capacity and traffic.
Ridge batch: when cache traffic catches the weights
The ridge batch is the batch size at which the estimated KV-cache traffic equals the weight traffic:
Bridge = W ÷ (C × K)
In the worked case with 16-bit weights, a 16-bit KV cache, and
C = 4096:
Bridge = 140,000,000,000 ÷ (4096 × 327,680) ≈ 104
Below a batch of roughly 104, the simplified weight-read term is larger than the cache-read term. Above it, context-dependent cache traffic is larger. This does not mean that batch 104 is automatically optimal. It only identifies a change in the dominant term of this model.
With an 8-bit KV cache, K is halved and the ridge batch is
approximately 208 at the same context. With 8-bit weights and a
16-bit KV cache, it is approximately 52. Quantizing weights moves the
ridge earlier because the fixed weight term becomes smaller.
Critical context length
The same relationship can be solved for context. The critical context length at a chosen batch is:
Ccritical = W ÷ (B × K)
For the 16-bit-weight, 16-bit-KV example at batch 8:
Ccritical = 140,000,000,000 ÷ (8 × 327,680) ≈ 53,406 tokens
At an average context below roughly 53k tokens, weights dominate the simplified traffic budget. Above it, cache reads dominate. If the KV cache is 8-bit, the critical context is approximately 106.8k tokens. If the weights are 8-bit while the cache remains 16-bit, it is approximately 26.7k tokens.
This threshold is useful for capacity planning, but it is not a quality or latency guarantee. Long-context implementations may use paged attention, sliding windows, prefix reuse, speculative decoding, or sparse attention patterns that change the amount of data actually read.
Make the estimate reproducible
Put the model and hardware assumptions in a small script rather than copying numbers into a spreadsheet. The following Python program uses decimal bytes for GB and TB. It prints the same quantities used above.
from dataclasses import dataclass
@dataclass
class Case:
layers: int
kv_heads: int
head_dim: int
parameters: int
weight_bytes: float
kv_bytes: float
bandwidth_bytes_s: float
batch: int
context: int
def estimate(case: Case):
kv_per_token = (
2
* case.layers
* case.kv_heads
* case.head_dim
* case.kv_bytes
)
weight_traffic = case.parameters * case.weight_bytes
kv_traffic = case.batch * case.context * kv_per_token
total_traffic = weight_traffic + kv_traffic
tokens_per_second = case.bandwidth_bytes_s / total_traffic
ridge_batch = weight_traffic / (case.context * kv_per_token)
critical_context = weight_traffic / (case.batch * kv_per_token)
return {
"kv_bytes_per_token": kv_per_token,
"weight_traffic_gb": weight_traffic / 1e9,
"kv_traffic_gb": kv_traffic / 1e9,
"total_traffic_gb": total_traffic / 1e9,
"aggregate_tokens_per_second": tokens_per_second,
"ridge_batch": ridge_batch,
"critical_context": critical_context,
}
case = Case(
layers=80,
kv_heads=8,
head_dim=128,
parameters=70_000_000_000,
weight_bytes=2,
kv_bytes=2,
bandwidth_bytes_s=1.6e12,
batch=8,
context=4096,
)
for key, value in estimate(case).items():
print(f"{key}: {value}")
To compare dtypes, run the same case with
weight_bytes=1 and/or kv_bytes=1. For a
quantized format with metadata, replace the nominal byte count with an
effective byte count:
effective bytes/scalar = payload bytes + scale/zero-point overhead
If the model uses tensor parallelism, calculate the local weight traffic and add communication separately. Do not divide the full model size by the number of devices and assume the rest of the system is free: all reductions, broadcasts, and synchronization can impose additional bandwidth or latency costs.
Choose usable bandwidth, not the headline number
A vendor's peak bandwidth is an upper bound. For a useful estimate, define a conservative utilization factor:
usable bandwidth = peak bandwidth × bandwidth efficiency
For example, if a device is advertised at 1.6 TB/s and you choose an assumed 70% efficiency, use 1.12 TB/s in the estimate. The efficiency value must be treated as an assumption until it is calibrated against a real workload. This article does not provide a verified efficiency value for any GPU.
A practical calibration sequence is:
- Measure a representative decode run with fixed model, prompt, batch, context, and output length.
- Record aggregate output tokens per second and per-sequence latency.
- Compute implied bandwidth as
traffic/token × measured tokens/s. - Use that implied value only for the same implementation and operating point.
- Repeat at several batches and context lengths to see whether the slope follows the cache term.
A benchmark should report warm-up policy, tokenization policy, prompt length, generated length, concurrency, quantization format, KV-cache format, device count, and whether input and output tokens are included. Without those details, two “tokens/s” figures may describe different workloads.
Convert the estimate into batching limits
Traffic is only one batching limit. The cache must also fit in available memory. A simple cache-capacity check is:
cache_bytes ≈ active_sequences × context × K
Reserve memory for weights, runtime workspaces, activations, temporary buffers, allocator fragmentation, and framework overhead before assigning the remainder to KV cache:
available_cache = device_memory − weights − reserve
The capacity-only upper bound is then:
batchmemory ≈ available_cache ÷ (context × K)
Use the smaller of the capacity bound and the latency or throughput target. A batch can fit while still missing a service-level objective because the KV term has made each token too expensive.
For variable-length requests, replace batch × context with
the sum of cached tokens across active sequences. This is the quantity
that matters for memory capacity and, approximately, for KV traffic.
Failure cases and common wrong turns
Using total parameter count as exact weight traffic
Parameter count multiplied by nominal dtype size omits scales, padding, duplicated tensors, embedding choices, tied weights, and runtime transformations. Treat it as a first-order estimate. Use the actual serialized and resident representation when sizing a production system.
Counting query heads instead of KV heads
Grouped-query attention stores fewer keys and values than query heads.
The cache formula must use Hkv. This is one of
the easiest ways to produce a large, systematic error.
Confusing aggregate and per-request throughput
The formula returns aggregate tokens per second across the active batch. Divide by active sequences only when they have comparable scheduling and context. Streaming latency can be dominated by queueing and synchronization even when aggregate throughput looks healthy.
Assuming quantized KV cache is free
KV quantization lowers storage and often lowers traffic, but it may add scale metadata, conversion work, or implementation-specific accuracy constraints. The one-byte example is a model input, not a universal property of all 8-bit caches.
Ignoring non-memory bottlenecks
The bandwidth estimate can be optimistic when attention kernels, dequantization, normalization, sampling, CPU scheduling, PCIe or interconnect transfers, or multi-tenant interference dominate. Compare the bandwidth estimate with a compute estimate and with an actual end-to-end measurement.
A compact decision procedure
- Identify the resident weight representation and calculate or measure
W. - Read the architecture configuration: layers, KV heads, head dimension, and KV dtype.
- Calculate
K = 2 × L × Hkv × D × Skv. - Estimate total traffic with the expected active-token count, not just the nominal batch size.
- Use a documented usable-bandwidth assumption to estimate aggregate decode speed.
- Calculate
Bridge = W ÷ (C × K)andCcritical = W ÷ (B × K). - Check that weights and KV cache fit simultaneously in device memory.
- Validate with a controlled benchmark and report the conditions that produced it.
The model is most valuable as a comparison tool. It explains why a smaller weight dtype often improves decode speed at every batch, why KV dtype matters more as context grows, and why a configuration that fits in memory may still be too slow at the target concurrency.
Limits of the estimate
This method does not predict exact latency. It assumes a simplified streaming pattern and does not model cache locality, kernel fusion, asynchronous scheduling, prefix sharing, speculative decoding, sparse attention, request arrival patterns, or output-sampling cost. It also does not establish the quality or safety of any quantization format.
Use it as an evidence-bounded planning calculation: state every input, show the units, preserve the distinction between a derived example and a measured result, and then test the operating points that matter to your workload. When the benchmark disagrees, the disagreement is useful evidence that another term has become dominant.