While a model writes, it keeps a running memory of everything it has read, the KV cache. Compressing it lets one GPU serve more conversations. We compared our compression (turbo-attn) against the KV-cache quantizations currently available in vLLM, all measured the same way in the same engine.
"Words unchanged" = how often the model writes the same next word as with uncompressed memory, under normal sampling. "Closer" compares each method's gap to uncompressed: ours 0.6 points at 4 bits vs 1.9 (int4) and 2.3 (vLLM 4-bit). All numbers: Qwen3.5-0.8B in vLLM 0.28.0 with our vllm-turbo patches; method and provenance at the bottom.
turbo-attn is not one trick. Each piece below removes a different source of error; all of them were switched on for the measurements on this page (details at the bottom).
Every model is calibrated on its own activations, captured from real text. Rotations, scales and codebooks are all fit from that data, not from generic assumptions.
A fast Walsh–Hadamard transform spreads outlier channels across the whole vector so every coordinate quantizes alike. Queries are rotated the same way, so attention scores need no un-rotation.
A rotation fitted per layer and KV head on top of Hadamard: for keys, the directions the queries actually probe; for values, the directions that actually reach the output. Bits go where attention looks.
Each vector's length is stored once as a bf16 number and only its direction is quantized, with a key-side norm correction so the decoded key keeps its true length.
Calibrated scales stretch each channel to fill the codebook's range before quantizing, so quiet channels are not wasted and loud ones do not clip.
Instead of evenly spaced levels, each layer and bit width gets Lloyd-Max levels fitted to its calibrated value distribution (5–8 bits).
At 2–4 bits, pairs of channels are quantized together against a 2-D codebook (4b entries per pair), which captures how neighbouring values move together.
While a prompt is being read, its newest chunk attends to its own fresh values instead of the compressed codes; only older context is read from the codes. (With the fast prompt-reading kernel used here, those fresh values are held at 8 bits.)
Each conversation's newest 128 positions, the ones decoding leans on hardest, are also kept at 8 bits beside the codes and read from there.
These are built into turbo-attn and can be switched on, but every result on this page used the default, faster setting. They are extra room to trade a little speed or effort for more faithfulness. We are not claiming they win on this test.
Gives each layer its own bit width under a fixed memory budget, chosen by calibration, instead of the same width everywhere. Enabled with a fractional budget such as TKV_BITS=4.5 and a SmartMix-calibrated file. These results used one width for every layer.
Adds up the attention scores without the rounding the fast tensor-core path allows, at some cost in speed. TKV_MMA_PRECISION=exact. These results used fast.
Keeps the decoded memory in 32-bit instead of 16-bit while attending, using the non-tensor-core kernel. TKV_DECODE_SMEM_HALF=0. These results used the 16-bit tile.
Reads the prompt with bf16 arithmetic instead of the fast 8-bit kernel, keeps the newest chunk at bf16, or spends a time budget on finer splits. TKV_PREFILL_INT8=0, TKV_PREFILL_INT8_FRESH_S8=0, TKV_PREFILL_INT8_SPLIT_FIDELITY. These results used the 8-bit kernel with the newest chunk at 8 bits.
How to read this chart. Each dot is one way of compressing the memory. Further left uses less memory. Higher up means the model writes more like the original. Top-left is best. Blue dots are ours, orange are the quantizations currently available in vLLM, and the dotted grey line is uncompressed memory.
Showing how often the model writes the same next word as the original. Our K4V4 setting matches 98.92% of the time; uncompressed memory matches 99.53%.
| codec | memory per number | fits in the same memory | words unchanged¹ |
|---|---|---|---|
| K2V2ours | 2.06 bits | 7.76× | 95.72% |
| K3V2ours | 2.56 bits | 6.24× | 96.99% |
| K3V3ours | 3.06 bits | 5.22× | 98.02% |
| vLLM 3-bitother | 3.09 bits | 5.17× | 93.62% |
| K4V3ours | 3.56 bits | 4.49× | 98.59% |
| vLLM K3/V4other | 3.59 bits | 4.45× | 96.64% |
| K4V4ours | 4.06 bits | 3.94× | 98.92% |
| vLLM 4-bitother | 4.09 bits | 3.91× | 97.20% |
| vLLM int4other | 4.12 bits | 3.88× | 97.63% |
| K5V4ours | 4.56 bits | 3.51× | 99.15% |
| K6V4ours | 5.06 bits | 3.16× | 99.24% |
| K6V5ours | 5.56 bits | 2.88× | 99.34% |
| K6V6ours | 6.06 bits | 2.64× | 99.40% |
| vLLM K8/V4other | 6.06 bits | 2.64× | 97.46% |
| FP8other | 8.00 bits | 2.00× | 99.24% |
| K8V8ours | 8.06 bits | 1.98× | 99.48% |
| vLLM int8other | 8.12 bits | 1.97× | 99.46% |
| Uncompressed (bf16)baseline | 16.00 bits | 1.00× (baseline) | 99.53% |
Small differences are within the margin of error, for example K8V8 against uncompressed memory.
oscar:4e855d3b37) over Hadamard, per-channel scales, key norm correction, vq2 codebooks at 2–4 bits and Lloyd-Max codebooks at 5–8 bits, fresh diagonal on (TKV_FRESH_DIAGONAL=1), fidelity ring on (128 positions at 8 bits). Kernels: tensor-core attention in fast mode with a 16-bit working tile, and the 8-bit prompt-reading kernel (newest chunk at 8 bits). Where two variants use the same memory, only the better one is shown (K5V4, not K4V5, K6V4, not K5V5, FP8, not FP8 e5m2, vLLM int8, not vLLM fp8 per-token-head).registry.arbi.work/vllm-turbo:latest, sha256:6f59874954cc…), eager mode, max length 16,384, GPU memory fraction 0.6.fp8 (e4m3), fp8_e5m2, the per-token-head int4/int8/fp8 modes, and the built-in low-bit presets (vLLM names them turboquant_*; shown here as "vLLM 3-bit", "vLLM 4-bit", "vLLM K3/V4", "vLLM K8/V4")./mnt/k8scache/paper_kv_ablation/ci/n32-main-20261003-run37145188480/.