How much does compressing the KV cache cost on hard science questions?
While it answers, a language model stores the attention keys and values of every token it has read and written so far: the KV cache. Every new token is computed by attending over all of it. Compressing the KV cache lets one GPU serve more and longer conversations. The question is whether the model gets less accurate. Here a small model (MiniCPM5-2B) answers GPQA Diamond, 198 graduate-level biology, chemistry and physics questions written to be hard even for experts who can search the web, with an uncompressed KV cache, with tkv compression, and with the 4-bit compression options built into vLLM.
Reference: the MiniCPM5-2B model card reports 70.2% on GPQA Diamond, taken from Artificial Analysis's independent evaluation (their own prompt and repeats). Our uncompressed KV cache scores 69.1% [63.9–74.0], in agreement. We follow the model card's recommended settings throughout; see "Reproduce these results" below.
Measured against the uncompressed KV cache on the same questions and seeds: tkv K8V8 -1.5 points [-3.0, -0.1]; tkv K5V5 -0.7 points [-3.9, +2.5]; tkv K4V4 -1.4 points [-2.9, +0.2].
A gap this size sits at the edge of what 15 repeats can separate from chance. We therefore also measured directly how far each setting moves the model's next-token probabilities from an exact 32-bit (FP32) reference, on GPQA-length answers of 6,800 to 35,000 tokens. The uncompressed bf16 KV cache scores 0.9929 (1 = identical); tkv K8V8 0.9917 and tkv K4V4 0.9824. K8V8's per-token error is about a ninth of K4V4's and does not grow with answer length, yet both lose the same ~1.5 points here. A per-token difference that small cannot account for 1.5 points, and the two tkv rows are measured against the same bf16 runs, so a lucky bf16 baseline would lower both alike. We therefore read the gap as chance, not a cost of compression.
Accuracy
How to read this chart. Each dot is one way of storing the KV cache. Further left is a smaller KV cache; higher up means more questions answered correctly. Blue dots are ours (tkv), orange are the KV cache formats built into vLLM, each named by its --kv-cache-dtype. The dotted grey line is the uncompressed KV cache (auto, bf16); the dotted blue line joins our best result at each size; thin vertical lines are 95% ranges1.
| KV cache dtype | KV cache bits per number (measured) | Correct [95% range] | Difference from bf16, same questions and seeds2 | Repeats (each) | Hit context limit3 |
|---|---|---|---|---|---|
| auto (bf16) | 16 (uncompressed) | 69.1% [63.9–74.0] | reference | 15 (69.2 / 66.2 / 71.2 / 71.7 / 67.2 / 68.7 / 68.2 / 66.7 / 69.2 / 69.7 / 70.7 / 67.2 / 69.2 / 69.2 / 72.2) | 3 of 2970 |
| tkv K8V8 | 8.13 (8-bit keys and values, plus a scale per vector) | 67.6% [62.2–72.7] | -1.5 points [-3.0, -0.1]: worse (2970 paired answers) | 15 (63.1 / 68.2 / 70.2 / 71.2 / 64.6 / 64.1 / 70.2 / 69.2 / 68.2 / 69.2 / 69.7 / 66.7 / 65.2 / 69.7 / 64.1) | 0 of 2970 |
| tkv K5V5 | 5.13 (5-bit keys and values, plus a scale per vector) | 68.2% [62.5–73.6] | -0.7 points [-3.9, +2.5]: no measurable difference (594 paired answers) | 3 (68.2 / 67.7 / 68.7) | 1 of 594 |
| tkv K4V4 | 4.13 (4-bit keys and values, plus a scale per vector) | 67.7% [62.4–72.7] | -1.4 points [-2.9, +0.2]: no measurable difference (2970 paired answers) | 15 (70.2 / 64.6 / 67.7 / 66.7 / 67.2 / 67.2 / 67.7 / 69.7 / 69.2 / 65.2 / 67.7 / 68.2 / 70.2 / 63.6 / 70.7) | 5 of 2970 |
| turboquant_4bit_nc | 5.31 (4-bit keys and values, but vLLM keeps the first 2 and last 2 of the 42 layers uncompressed) | 68.0% [62.1–73.6] | -0.8 points [-3.7, +2.0]: no measurable difference (594 paired answers) | 3 (66.7 / 67.2 / 70.2) | 0 of 594 |
| int4_per_token_head | 4.25 (4-bit keys and values, plus a scale per token and head) | 67.3% [61.6–72.7] | -1.5 points [-5.1, +2.0]: no measurable difference (594 paired answers) | 3 (67.2 / 67.2 / 67.7) | 10 of 594 |
- 95% range: bootstrap over the 198 questions (5,000 resamples), each question scored as its average over its repeats.
- Paired difference: the same questions and seeds under both settings, so question difficulty cancels out. A range that includes 0 means no measurable difference.
- Responses that reached the 131,072-token window. Every one was runaway thinking that never finished; none ran out of room for the answer. They count as wrong. See "Answers that never stop" below.
Why this model and this test
We picked a setup that is hard on a compressed KV cache, so that passing it means something. Easier setups would flatter any compression method, ours included.
- A classic dense model. MiniCPM5-2B is a standard transformer (the Llama design): all 42 of its layers attend over the full KV cache of the conversation, with no shortcut layers. That makes it a clean reference with nowhere for errors to hide.
- Full-precision weights. At 2 billion parameters the model fits unquantized (bf16 weights, about 5 GB) on a single NVIDIA GPU, so weight quantization plays no part in these results. KV cache quantization is generally independent of weight quantization: any error from quantizing weights or activations would affect every KV cache format in a similar way, so the comparison between formats would hold either way.
- Errors compound. Each layer's output feeds the next, so a small error in an early layer's attention is carried through every layer after it, and each of those layers attends over its own compressed KV cache again.
- Little room for error. A 2-billion-parameter model has less spare capacity to absorb noise than a large one. Its KV cache is also tightly shared: 16 query heads attend over just 2 key/value heads, so every compressed number is read by 8 heads.
- A test that leans on the KV cache. GPQA Diamond makes the model reason at length before answering: about 8,000 tokens per answer on average, up to 130,000. Every one of those tokens is written into the compressed KV cache and attended over by every token after it.
- More forgiving setups. Hybrid models keep a KV cache in only some layers (Qwen3.5-0.8B: 6 of its 24; the rest use linear attention with a small fixed-size state that KV compression doesn't touch). Short prompts with short answers barely exercise the KV cache at all.
Answers that never stop
With the model card's repetition penalty, answers that think until the 131,072-token window is full are rare: auto (bf16) 3 of 2,970, tkv K8V8 0 of 2,970, tkv K5V5 1 of 594, tkv K4V4 5 of 2,970, turboquant_4bit_nc 0 of 594, int4_per_token_head 10 of 594. 10 of them collapsed into meaningless repetition: tkv K5V5 1, int4_per_token_head 9. They count as wrong.
int8 prefill off. tkv K4V4 with its int8 prompt-reading kernel switched off (TKV_PREFILL_INT8=0) scored 68.3%, +0.6 points [-0.7, +1.9]: no measurable difference (2970 paired answers) against the default, over the same 15 seeds. These prompts are short, so this tests int8 prompt reading only at short context.
How we measured
Questions: all 198 GPQA Diamond questions, the four options in a fixed per-question order (seeded from the question id, the same in every run), and the standard simple-evals prompt ("…The last line of your response should be of the following format: 'Answer: $LETTER'…"). The answer is the last such line after the model's thinking. Responses with no answer line count as wrong.
Repeats: the uncompressed KV cache, tkv K4V4 and tkv K8V8 asked every question 15 times (seeds 21 to 35), 2,970 answers each; the other settings 3 times (seeds 21, 22, 23), 594 answers each. Paired differences use only the seeds both settings ran. A single pass of bf16 swung from 66.2% to 72.2% with the seed alone, which is why one pass isn't enough.
Model and serving: MiniCPM5-2B in vLLM with its default thinking mode and the model card's recommended sampling: temperature 1.0, top-p 0.95, min-p 0 and no top-k (read from the model's generation config), plus the card's repetition_penalty 1.05 for repetitive output; 64 to 256 requests at a time, sized so the KV cache never ran short. No limit on response length; the full 131,072-token window; prompts never shortened. Only --kv-cache-dtype changes between settings.
Hardware: NVIDIA RTX 4090 for every setting. Accuracy only: these runs were not set up as speed measurements.
The page contains no question text. The question set stays private, as its authors ask.
Reproduce these results
Model: openbmb/MiniCPM5-2B, revision 12a3808a956f869c767195e9266b59c4d21d92e2, bf16 weights, default chat template (thinking on).
Sampling: exactly the model card's recommendation, read from the model's generation_config.json by --generation-config auto: temperature 1.0, top-p 0.95, min-p 0, no top-k, plus the card's repetition_penalty: 1.05, sent with each request along with its seed. No max_tokens, no system prompt.
Software: vLLM 0.28.0 and torch 2.13.0+cu130 (our image vllm-turbo:8c2f68ef60), with turbo-attn main mounted over the image's copy (c76018f4b; the K8V8 seeds 24 to 35 on 47e75ad97, which adds a decode speed fix only). One NVIDIA RTX 4090 (24 GB) per server.
Server, uncompressed (auto, bf16):
vllm serve openbmb/MiniCPM5-2B --served-model-name m --dtype bfloat16 --trust-remote-code --max-model-len 131072 --max-num-seqs 256 --gpu-memory-utilization 0.85 --generation-config auto --kv-cache-dtype auto
vLLM's built-in formats: the same command with --kv-cache-dtype turboquant_4bit_nc or --kv-cache-dtype int4_per_token_head.
tkv (turbo-attn): the same command with --kv-cache-dtype tkv --attention-backend TURBO_ATTN and:
VLLM_PLUGINS=tkv TKV_BITS=4 # 5 for K5V5, 8 for K8V8 TKV_CALIBRATION_FILE=calibrations/minicpm5-2b_v59.json # from the turbo-attn repo # everything else at its default, e.g. int8 prefill (TKV_PREFILL_INT8) and the fresh diagonal (TKV_FRESH_DIAGONAL) both on
The calibration file is calibrations/minicpm5-2b_v59.json in the turbo-attn repository: the same rotations, codebooks and scales these runs loaded.
Questions: gpqa_diamond.csv from the gated dataset Idavidrein/gpqa (198 rows, sha256 41d1213cd7a4998605a26c2798500652572007161b3a92817ba46b35befcd305).
Client: gpqa_client.py holds the exact prompt, option shuffle, request and answer extraction used here. Run it against the server:
python gpqa_client.py http://localhost:8000/v1/chat/completions gpqa_diamond.csv out.json 21,22,23 256 --repetition-penalty 1.05
Seeds 21 to 35 (or 21, 22, 23); 64 to 256 requests in flight; no time limit that could cut an answer short. Score = share of all answers correct (pass@1 averaged over seeds); 95% ranges are a bootstrap over questions.