Systems · Inference performance

Estimating LLM Decode Speed from Memory Bandwidth

Измеряем скорость decode LLM через пропускную способность памяти
Temporary fallback cover; replace in editorial pass.
Advanced 20-minute read

A useful first estimate for generation speed is often not the number of tensor cores. During autoregressive decode, a large language model (LLM) repeatedly streams its weights and reads the KV cache. If you can account for those bytes, you can estimate a traffic budget, an idealized tokens-per-second ceiling, the batch at which the cache becomes as expensive as the weights, and the context length where that transition occurs.

The short answer

For one generated token, approximate memory traffic as:

traffic/token ≈ weight_bytes + batch × context × kv_bytes/token

Then divide usable memory bandwidth by that traffic:

decode_tokens/s ≈ usable_bandwidth / traffic/token

This is a bandwidth-bound estimate, not a benchmark result. It assumes that the relevant weights and cache are streamed efficiently, that transfers overlap with computation, and that arithmetic, kernel launch overhead, communication, and scheduling do not become the limiting factors.

The estimate becomes especially useful when comparing data types. Lower-precision weights reduce the fixed cost paid by every generated token. Lower-precision KV cache reduces the cost that grows with batch size and context length.

What decode is actually moving

In a typical decoder-only transformer, one decode step consumes one new token for every active sequence. The model must apply every layer to the current hidden state. The weight matrices are reused across the batch, while attention reads the keys and values stored for previous tokens.

During prefill, many tokens are processed together and matrix multiplication can have high reuse. During decode, the new-token dimension is small. The workload often has low arithmetic intensity: relatively few operations are performed for each byte fetched from memory.

The model can therefore approach a roofline-model limit set by memory bandwidth before it reaches the accelerator's advertised compute throughput. This does not make compute irrelevant. It means that compute should be checked as a second bound after the traffic estimate.

Notation

Symbol Meaning
W Bytes occupied by model weights that must be read per decode step
B Active batch size: sequences producing a token in this step
C Average cached context length per active sequence
K KV-cache bytes read per sequence and cached token
BW Usable device memory bandwidth in bytes per second
T Estimated aggregate decode rate in tokens per second

Derive the KV-cache term

Let a model have L transformer layers, Hkv key/value heads, head dimension D, and Skv bytes per cache scalar. Each cached token stores both a key and a value:

K = 2 × L × Hkv × D × Skv

The factor of two accounts for keys and values. For grouped-query attention or multi-query attention, use the number of KV heads, not the number of query heads. Confusing those two values can overestimate cache traffic by a large factor.

For a decode step, a simple attention-read model is:

KV traffic/token ≈ B × C × K

This treats the full cached prefix as read for every active sequence. Implementations can differ in layout, padding, page size, quantization metadata, and cache reuse, so this term is an approximation rather than a byte-for-byte trace.

Worked case: an 80-layer GQA model

The following is a reproducible numerical example, not a claim about a measured product or a particular vendor's implementation. Assume:

  • 80 transformer layers
  • 8 KV heads
  • 128 values per head
  • 70 billion parameters
  • 16-bit weights, so W = 70 × 109 × 2 = 140 GB before format-specific overhead
  • 1.6 TB/s usable memory bandwidth
  • batch B = 8
  • cached context C = 4096 tokens

With a 16-bit KV cache:

K = 2 × 80 × 8 × 128 × 2 = 327,680 bytes/token

The cache traffic for the batch is:

8 × 4096 × 327,680 = 10,737,418,240 bytes ≈ 10.74 GB

The total simplified traffic budget is therefore about 140 + 10.74 = 150.74 GB per aggregate decode step. The bandwidth estimate is:

T ≈ 1,600 GB/s ÷ 150.74 GB ≈ 10.6 aggregate tokens/s

If all eight sequences are equally active, that corresponds to about 1.3 tokens/s per sequence. These numbers are arithmetic from the assumptions above. They are not measured throughput, and they omit quantization scales, allocator padding, cache metadata, kernel effects, and any multi-device traffic.

Compare KV-cache dtypes

Weights KV cache Weight traffic KV traffic at B=8, C=4096 Estimated aggregate rate
16-bit 16-bit 140.00 GB 10.74 GB 10.6 tokens/s
16-bit 8-bit 140.00 GB 5.37 GB 11.0 tokens/s
8-bit 16-bit 70.00 GB 10.74 GB 19.8 tokens/s
8-bit 8-bit 70.00 GB 5.37 GB 21.2 tokens/s

The table assumes exactly one byte per scalar for the 8-bit cases and ignores scales and other metadata. Real quantized formats may have block scales, zero points, alignment requirements, or dequantization behavior that changes both capacity and traffic.

Ridge batch: when cache traffic catches the weights

The ridge batch is the batch size at which the estimated KV-cache traffic equals the weight traffic:

Bridge = W ÷ (C × K)

In the worked case with 16-bit weights, a 16-bit KV cache, and C = 4096:

Bridge = 140,000,000,000 ÷ (4096 × 327,680) ≈ 104

Below a batch of roughly 104, the simplified weight-read term is larger than the cache-read term. Above it, context-dependent cache traffic is larger. This does not mean that batch 104 is automatically optimal. It only identifies a change in the dominant term of this model.

With an 8-bit KV cache, K is halved and the ridge batch is approximately 208 at the same context. With 8-bit weights and a 16-bit KV cache, it is approximately 52. Quantizing weights moves the ridge earlier because the fixed weight term becomes smaller.

Critical context length

The same relationship can be solved for context. The critical context length at a chosen batch is:

Ccritical = W ÷ (B × K)

For the 16-bit-weight, 16-bit-KV example at batch 8:

Ccritical = 140,000,000,000 ÷ (8 × 327,680) ≈ 53,406 tokens

At an average context below roughly 53k tokens, weights dominate the simplified traffic budget. Above it, cache reads dominate. If the KV cache is 8-bit, the critical context is approximately 106.8k tokens. If the weights are 8-bit while the cache remains 16-bit, it is approximately 26.7k tokens.

This threshold is useful for capacity planning, but it is not a quality or latency guarantee. Long-context implementations may use paged attention, sliding windows, prefix reuse, speculative decoding, or sparse attention patterns that change the amount of data actually read.

Make the estimate reproducible

Put the model and hardware assumptions in a small script rather than copying numbers into a spreadsheet. The following Python program uses decimal bytes for GB and TB. It prints the same quantities used above.

from dataclasses import dataclass

@dataclass
class Case:
    layers: int
    kv_heads: int
    head_dim: int
    parameters: int
    weight_bytes: float
    kv_bytes: float
    bandwidth_bytes_s: float
    batch: int
    context: int

def estimate(case: Case):
    kv_per_token = (
        2
        * case.layers
        * case.kv_heads
        * case.head_dim
        * case.kv_bytes
    )
    weight_traffic = case.parameters * case.weight_bytes
    kv_traffic = case.batch * case.context * kv_per_token
    total_traffic = weight_traffic + kv_traffic
    tokens_per_second = case.bandwidth_bytes_s / total_traffic
    ridge_batch = weight_traffic / (case.context * kv_per_token)
    critical_context = weight_traffic / (case.batch * kv_per_token)

    return {
        "kv_bytes_per_token": kv_per_token,
        "weight_traffic_gb": weight_traffic / 1e9,
        "kv_traffic_gb": kv_traffic / 1e9,
        "total_traffic_gb": total_traffic / 1e9,
        "aggregate_tokens_per_second": tokens_per_second,
        "ridge_batch": ridge_batch,
        "critical_context": critical_context,
    }

case = Case(
    layers=80,
    kv_heads=8,
    head_dim=128,
    parameters=70_000_000_000,
    weight_bytes=2,
    kv_bytes=2,
    bandwidth_bytes_s=1.6e12,
    batch=8,
    context=4096,
)

for key, value in estimate(case).items():
    print(f"{key}: {value}")

To compare dtypes, run the same case with weight_bytes=1 and/or kv_bytes=1. For a quantized format with metadata, replace the nominal byte count with an effective byte count:

effective bytes/scalar = payload bytes + scale/zero-point overhead

If the model uses tensor parallelism, calculate the local weight traffic and add communication separately. Do not divide the full model size by the number of devices and assume the rest of the system is free: all reductions, broadcasts, and synchronization can impose additional bandwidth or latency costs.

Choose usable bandwidth, not the headline number

A vendor's peak bandwidth is an upper bound. For a useful estimate, define a conservative utilization factor:

usable bandwidth = peak bandwidth × bandwidth efficiency

For example, if a device is advertised at 1.6 TB/s and you choose an assumed 70% efficiency, use 1.12 TB/s in the estimate. The efficiency value must be treated as an assumption until it is calibrated against a real workload. This article does not provide a verified efficiency value for any GPU.

A practical calibration sequence is:

  1. Measure a representative decode run with fixed model, prompt, batch, context, and output length.
  2. Record aggregate output tokens per second and per-sequence latency.
  3. Compute implied bandwidth as traffic/token × measured tokens/s.
  4. Use that implied value only for the same implementation and operating point.
  5. Repeat at several batches and context lengths to see whether the slope follows the cache term.

A benchmark should report warm-up policy, tokenization policy, prompt length, generated length, concurrency, quantization format, KV-cache format, device count, and whether input and output tokens are included. Without those details, two “tokens/s” figures may describe different workloads.

Convert the estimate into batching limits

Traffic is only one batching limit. The cache must also fit in available memory. A simple cache-capacity check is:

cache_bytes ≈ active_sequences × context × K

Reserve memory for weights, runtime workspaces, activations, temporary buffers, allocator fragmentation, and framework overhead before assigning the remainder to KV cache:

available_cache = device_memory − weights − reserve

The capacity-only upper bound is then:

batchmemory ≈ available_cache ÷ (context × K)

Use the smaller of the capacity bound and the latency or throughput target. A batch can fit while still missing a service-level objective because the KV term has made each token too expensive.

For variable-length requests, replace batch × context with the sum of cached tokens across active sequences. This is the quantity that matters for memory capacity and, approximately, for KV traffic.

Failure cases and common wrong turns

Using total parameter count as exact weight traffic

Parameter count multiplied by nominal dtype size omits scales, padding, duplicated tensors, embedding choices, tied weights, and runtime transformations. Treat it as a first-order estimate. Use the actual serialized and resident representation when sizing a production system.

Counting query heads instead of KV heads

Grouped-query attention stores fewer keys and values than query heads. The cache formula must use Hkv. This is one of the easiest ways to produce a large, systematic error.

Confusing aggregate and per-request throughput

The formula returns aggregate tokens per second across the active batch. Divide by active sequences only when they have comparable scheduling and context. Streaming latency can be dominated by queueing and synchronization even when aggregate throughput looks healthy.

Assuming quantized KV cache is free

KV quantization lowers storage and often lowers traffic, but it may add scale metadata, conversion work, or implementation-specific accuracy constraints. The one-byte example is a model input, not a universal property of all 8-bit caches.

Ignoring non-memory bottlenecks

The bandwidth estimate can be optimistic when attention kernels, dequantization, normalization, sampling, CPU scheduling, PCIe or interconnect transfers, or multi-tenant interference dominate. Compare the bandwidth estimate with a compute estimate and with an actual end-to-end measurement.

A compact decision procedure

  1. Identify the resident weight representation and calculate or measure W.
  2. Read the architecture configuration: layers, KV heads, head dimension, and KV dtype.
  3. Calculate K = 2 × L × Hkv × D × Skv.
  4. Estimate total traffic with the expected active-token count, not just the nominal batch size.
  5. Use a documented usable-bandwidth assumption to estimate aggregate decode speed.
  6. Calculate Bridge = W ÷ (C × K) and Ccritical = W ÷ (B × K).
  7. Check that weights and KV cache fit simultaneously in device memory.
  8. Validate with a controlled benchmark and report the conditions that produced it.

The model is most valuable as a comparison tool. It explains why a smaller weight dtype often improves decode speed at every batch, why KV dtype matters more as context grows, and why a configuration that fits in memory may still be too slow at the target concurrency.

Limits of the estimate

This method does not predict exact latency. It assumes a simplified streaming pattern and does not model cache locality, kernel fusion, asynchronous scheduling, prefix sharing, speculative decoding, sparse attention, request arrival patterns, or output-sampling cost. It also does not establish the quality or safety of any quantization format.

Use it as an evidence-bounded planning calculation: state every input, show the units, preserve the distinction between a derived example and a measured result, and then test the operating points that matter to your workload. When the benchmark disagrees, the disagreement is useful evidence that another term has become dominant.

We publish what works for us—and implement the same solutions for your business. We design AI automation, Telegram bots, chats, and AI agents for real-world processes. Discuss your project →