tkv advantage over each engine's own bf16 KV Same engine, same workload, KV cache swapped: bf16 → tkv. Higher is better on every bar.
Decode tok/s per stream: tkv vs bf16 0% 20% 40% 60% 80% % vs bf16 arbi-serve c1: tkv vs bf16 +55.9% +56% vLLM c1: tkv vs bf16 +57.9% +58% SGLang c1: tkv vs bf16 +55.7% +56% c1@126k arbi-serve c8: tkv vs bf16 +39.7% +40% vLLM c8: tkv vs bf16 +31.4% +31% SGLang c8: tkv vs bf16 +34.0% +34% c8@14k arbi-serve c64: tkv vs bf16 +42.9% +43% vLLM c64: tkv vs bf16 +42.5% +42% SGLang c64: tkv vs bf16 +41.0% +41% c64@1.6k arbi-serve c256: tkv vs bf16 +16.1% +16% vLLM c256: tkv vs bf16 +17.7% +18% SGLang c256: tkv vs bf16 +29.2% +29% c256@0.8k concurrency @ median prompt length Prefill tok/s: tkv vs bf16 0% 20% 40% 60% 80% % vs bf16 arbi-serve c1: tkv vs bf16 +66.7% +67% vLLM c1: tkv vs bf16 +65.5% +66% SGLang c1: tkv vs bf16 +66.9% +67% c1@126k arbi-serve c8: tkv vs bf16 +24.5% +25% vLLM c8: tkv vs bf16 +18.9% +19% SGLang c8: tkv vs bf16 +30.9% +31% c8@14k arbi-serve c64: tkv vs bf16 +1.7% +2% vLLM c64: tkv vs bf16 +4.9% +5% SGLang c64: tkv vs bf16 +8.8% +9% c64@1.6k arbi-serve c256: tkv vs bf16 +0.5% +0.5% vLLM c256: tkv vs bf16 -0.4% -0.4% SGLang c256: tkv vs bf16 +4.0% +4% c256@0.8k concurrency @ median prompt length TTFT reduction: tkv vs bf16 0% 10% 20% 30% 40% 50% % lower TTFT arbi-serve c1: tkv vs bf16 +40.0% +40% vLLM c1: tkv vs bf16 +39.6% +40% SGLang c1: tkv vs bf16 +40.1% +40% c1@126k arbi-serve c8: tkv vs bf16 +17.5% +17% vLLM c8: tkv vs bf16 +15.3% +15% SGLang c8: tkv vs bf16 +23.2% +23% c8@14k arbi-serve c64: tkv vs bf16 +2.3% +2% vLLM c64: tkv vs bf16 +7.3% +7% SGLang c64: tkv vs bf16 +9.0% +9% c64@1.6k arbi-serve c256: tkv vs bf16 +0.2% +0.2% vLLM c256: tkv vs bf16 -1.4% -1% SGLang c256: tkv vs bf16 +4.8% +5% c256@0.8k concurrency @ median prompt length
Engine parity One colour per engine: the engines track each other, with the tkv lines (solid) above the bf16 lines (dashed).
Decode tok/s per stream 0 50 100 150 tok/s per stream c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 85.4 tok/s per stream arbi-serve bf16 c8: 68.3 tok/s per stream arbi-serve bf16 c64: 68.7 tok/s per stream arbi-serve bf16 c256: 27.4 tok/s per stream vLLM bf16 c1: 85.3 tok/s per stream vLLM bf16 c8: 70.8 tok/s per stream vLLM bf16 c64: 67.9 tok/s per stream vLLM bf16 c256: 27.2 tok/s per stream SGLang bf16 c1: 84.7 tok/s per stream SGLang bf16 c8: 69.2 tok/s per stream SGLang bf16 c64: 67.1 tok/s per stream SGLang bf16 c256: 23.4 tok/s per stream arbi-serve tkv c1: 133 tok/s per stream arbi-serve tkv c8: 95.3 tok/s per stream arbi-serve tkv c64: 98.1 tok/s per stream arbi-serve tkv c256: 31.9 tok/s per stream vLLM tkv c1: 135 tok/s per stream vLLM tkv c8: 93.0 tok/s per stream vLLM tkv c64: 96.8 tok/s per stream vLLM tkv c256: 32.0 tok/s per stream SGLang tkv c1: 132 tok/s per stream SGLang tkv c8: 92.7 tok/s per stream SGLang tkv c64: 94.6 tok/s per stream SGLang tkv c256: 30.3 tok/s per stream Prefill tok/s 0 2.5k 5k 7.5k 10k 12.5k prefill tok/s c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 5777 prefill tok/s arbi-serve bf16 c8: 4452 prefill tok/s arbi-serve bf16 c64: 934 prefill tok/s arbi-serve bf16 c256: 220 prefill tok/s vLLM bf16 c1: 5811 prefill tok/s vLLM bf16 c8: 4240 prefill tok/s vLLM bf16 c64: 799 prefill tok/s vLLM bf16 c256: 210 prefill tok/s SGLang bf16 c1: 5658 prefill tok/s SGLang bf16 c8: 4443 prefill tok/s SGLang bf16 c64: 867 prefill tok/s SGLang bf16 c256: 234 prefill tok/s arbi-serve tkv c1: 9632 prefill tok/s arbi-serve tkv c8: 5545 prefill tok/s arbi-serve tkv c64: 950 prefill tok/s arbi-serve tkv c256: 221 prefill tok/s vLLM tkv c1: 9619 prefill tok/s vLLM tkv c8: 5040 prefill tok/s vLLM tkv c64: 838 prefill tok/s vLLM tkv c256: 209 prefill tok/s SGLang tkv c1: 9440 prefill tok/s SGLang tkv c8: 5815 prefill tok/s SGLang tkv c64: 944 prefill tok/s SGLang tkv c256: 244 prefill tok/s Scaling with batch Total decode throughput over all concurrent streams.
Total decode tok/s (all streams) 0 2k 4k 6k 8k 10k tok/s, all streams c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 85 tok/s, all streams arbi-serve bf16 c8: 697 tok/s, all streams arbi-serve bf16 c64: 4736 tok/s, all streams arbi-serve bf16 c256: 8173 tok/s, all streams vLLM bf16 c1: 85 tok/s, all streams vLLM bf16 c8: 721 tok/s, all streams vLLM bf16 c64: 4702 tok/s, all streams vLLM bf16 c256: 8035 tok/s, all streams SGLang bf16 c1: 85 tok/s, all streams SGLang bf16 c8: 708 tok/s, all streams SGLang bf16 c64: 4648 tok/s, all streams SGLang bf16 c256: 6798 tok/s, all streams arbi-serve tkv c1: 133 tok/s, all streams arbi-serve tkv c8: 890 tok/s, all streams arbi-serve tkv c64: 6494 tok/s, all streams arbi-serve tkv c256: 9299 tok/s, all streams vLLM tkv c1: 135 tok/s, all streams vLLM tkv c8: 892 tok/s, all streams vLLM tkv c64: 6368 tok/s, all streams vLLM tkv c256: 9333 tok/s, all streams SGLang tkv c1: 132 tok/s, all streams SGLang tkv c8: 885 tok/s, all streams SGLang tkv c64: 6233 tok/s, all streams SGLang tkv c256: 8477 tok/s, all streams Validity: 24 of 24 cells are VALID under the all-N-admitted rule (each cell's own [validity] verdict). decode tok/s per stream = 1000 / median TPOT. prefill tok/s = prompt tokens / TTFT. TTFT = median ms. total decode = output tokens of all streams over their decode spans. c<N>@<len>: concurrency N at the cells' median prompt length, which the matrix varies with N. Percentages compare tkv with the same engine's bf16 at the same cell. Bars and y-axes start at 0. Charts show point values; the table below carries the run's spread.