tkv advantage over each engine's own bf16 KV Same engine, same workload, KV cache swapped: bf16 → tkv. Higher is better on every bar.
Decode tok/s per stream: tkv vs bf16 0% 20% 40% 60% 80% % vs bf16 arbi-serve c1: tkv vs bf16 +56.9% +57% vLLM c1: tkv vs bf16 +57.6% +58% SGLang c1: tkv vs bf16 +56.2% +56% c1@126k arbi-serve c8: tkv vs bf16 +35.9% +36% vLLM c8: tkv vs bf16 +30.4% +30% SGLang c8: tkv vs bf16 +33.9% +34% c8@14k arbi-serve c64: tkv vs bf16 +33.5% +34% SGLang c64: tkv vs bf16 +41.1% +41% c64@1.6k vLLM c256: tkv vs bf16 +21.3% +21% SGLang c256: tkv vs bf16 +29.8% +30% c256@0.8k concurrency @ median prompt length Prefill tok/s: tkv vs bf16 0% 20% 40% 60% 80% % vs bf16 arbi-serve c1: tkv vs bf16 +69.0% +69% vLLM c1: tkv vs bf16 +63.3% +63% SGLang c1: tkv vs bf16 +64.3% +64% c1@126k arbi-serve c8: tkv vs bf16 +26.4% +26% vLLM c8: tkv vs bf16 +15.4% +15% SGLang c8: tkv vs bf16 +29.7% +30% c8@14k arbi-serve c64: tkv vs bf16 +3.1% +3% SGLang c64: tkv vs bf16 +8.8% +9% c64@1.6k vLLM c256: tkv vs bf16 +1.1% +1% SGLang c256: tkv vs bf16 +4.2% +4% c256@0.8k concurrency @ median prompt length TTFT reduction: tkv vs bf16 0% 10% 20% 30% 40% 50% % lower TTFT arbi-serve c1: tkv vs bf16 +40.8% +41% vLLM c1: tkv vs bf16 +38.7% +39% SGLang c1: tkv vs bf16 +39.1% +39% c1@126k arbi-serve c8: tkv vs bf16 +24.4% +24% vLLM c8: tkv vs bf16 +12.4% +12% SGLang c8: tkv vs bf16 +22.3% +22% c8@14k arbi-serve c64: tkv vs bf16 +2.9% +3% SGLang c64: tkv vs bf16 +8.8% +9% c64@1.6k vLLM c256: tkv vs bf16 +0.4% +0.4% SGLang c256: tkv vs bf16 +4.2% +4% c256@0.8k concurrency @ median prompt length
Engine parity One colour per engine: the engines track each other, with the tkv lines (solid) above the bf16 lines (dashed).
Decode tok/s per stream 0 50 100 150 tok/s per stream c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 80.6 tok/s per stream arbi-serve bf16 c8: 61.5 tok/s per stream arbi-serve bf16 c64: 55.8 tok/s per stream vLLM bf16 c1: 85.0 tok/s per stream vLLM bf16 c8: 70.7 tok/s per stream vLLM bf16 c64: 67.7 tok/s per stream vLLM bf16 c256: 27.0 tok/s per stream SGLang bf16 c1: 84.5 tok/s per stream SGLang bf16 c8: 69.2 tok/s per stream SGLang bf16 c64: 67.0 tok/s per stream SGLang bf16 c256: 23.5 tok/s per stream arbi-serve tkv c1: 126 tok/s per stream arbi-serve tkv c8: 83.5 tok/s per stream arbi-serve tkv c64: 74.5 tok/s per stream vLLM tkv c1: 134 tok/s per stream vLLM tkv c8: 92.3 tok/s per stream vLLM tkv c256: 32.7 tok/s per stream SGLang tkv c1: 132 tok/s per stream SGLang tkv c8: 92.7 tok/s per stream SGLang tkv c64: 94.5 tok/s per stream SGLang tkv c256: 30.5 tok/s per stream Prefill tok/s 0 2.5k 5k 7.5k 10k 12.5k prefill tok/s c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 5773 prefill tok/s arbi-serve bf16 c8: 4697 prefill tok/s arbi-serve bf16 c64: 885 prefill tok/s vLLM bf16 c1: 5857 prefill tok/s vLLM bf16 c8: 4278 prefill tok/s vLLM bf16 c64: 806 prefill tok/s vLLM bf16 c256: 210 prefill tok/s SGLang bf16 c1: 5738 prefill tok/s SGLang bf16 c8: 4473 prefill tok/s SGLang bf16 c64: 869 prefill tok/s SGLang bf16 c256: 234 prefill tok/s arbi-serve tkv c1: 9757 prefill tok/s arbi-serve tkv c8: 5935 prefill tok/s arbi-serve tkv c64: 912 prefill tok/s vLLM tkv c1: 9562 prefill tok/s vLLM tkv c8: 4936 prefill tok/s vLLM tkv c256: 212 prefill tok/s SGLang tkv c1: 9428 prefill tok/s SGLang tkv c8: 5804 prefill tok/s SGLang tkv c64: 945 prefill tok/s SGLang tkv c256: 244 prefill tok/s Scaling with batch Total decode throughput over all concurrent streams.
Total decode tok/s (all streams) 0 2k 4k 6k 8k 10k tok/s, all streams c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 81 tok/s, all streams arbi-serve bf16 c8: 627 tok/s, all streams arbi-serve bf16 c64: 3960 tok/s, all streams vLLM bf16 c1: 85 tok/s, all streams vLLM bf16 c8: 721 tok/s, all streams vLLM bf16 c64: 4685 tok/s, all streams vLLM bf16 c256: 7984 tok/s, all streams SGLang bf16 c1: 84 tok/s, all streams SGLang bf16 c8: 709 tok/s, all streams SGLang bf16 c64: 4644 tok/s, all streams SGLang bf16 c256: 6806 tok/s, all streams arbi-serve tkv c1: 126 tok/s, all streams arbi-serve tkv c8: 800 tok/s, all streams arbi-serve tkv c64: 5186 tok/s, all streams vLLM tkv c1: 134 tok/s, all streams vLLM tkv c8: 886 tok/s, all streams vLLM tkv c256: 9350 tok/s, all streams SGLang tkv c1: 132 tok/s, all streams SGLang tkv c8: 885 tok/s, all streams SGLang tkv c64: 6226 tok/s, all streams SGLang tkv c256: 8505 tok/s, all streams Validity: 21 of 21 cells are VALID under the all-N-admitted rule (each cell's own [validity] verdict). decode tok/s per stream = 1000 / median TPOT. prefill tok/s = prompt tokens / TTFT. TTFT = median ms. total decode = output tokens of all streams over their decode spans. c<N>@<len>: concurrency N at the cells' median prompt length, which the matrix varies with N. Percentages compare tkv with the same engine's bf16 at the same cell. Bars and y-axes start at 0. Charts show point values; the table below carries the run's spread.