tkv advantage over each engine's own bf16 KV Same engine, same workload, KV cache swapped: bf16 → tkv. Higher is better on every bar.
Decode tok/s per stream: tkv vs bf16 0% 20% 40% 60% 80% % vs bf16 arbi-serve c1: tkv vs bf16 +56.3% +56% vLLM c1: tkv vs bf16 +58.1% +58% SGLang c1: tkv vs bf16 +56.1% +56% c1@126k arbi-serve c8: tkv vs bf16 +36.7% +37% vLLM c8: tkv vs bf16 +31.4% +31% SGLang c8: tkv vs bf16 +33.9% +34% c8@14k arbi-serve c64: tkv vs bf16 +43.6% +44% vLLM c64: tkv vs bf16 +42.5% +42% SGLang c64: tkv vs bf16 +41.1% +41% c64@1.6k arbi-serve c256: tkv vs bf16 +20.1% +20% vLLM c256: tkv vs bf16 +22.2% +22% SGLang c256: tkv vs bf16 +29.4% +29% c256@0.8k concurrency @ median prompt length Prefill tok/s: tkv vs bf16 0% 20% 40% 60% 80% % vs bf16 arbi-serve c1: tkv vs bf16 +66.9% +67% vLLM c1: tkv vs bf16 +64.9% +65% SGLang c1: tkv vs bf16 +65.2% +65% c1@126k arbi-serve c8: tkv vs bf16 +11.8% +12% vLLM c8: tkv vs bf16 +18.4% +18% SGLang c8: tkv vs bf16 +29.8% +30% c8@14k arbi-serve c64: tkv vs bf16 +2.8% +3% vLLM c64: tkv vs bf16 +3.0% +3% SGLang c64: tkv vs bf16 +9.8% +10% c64@1.6k arbi-serve c256: tkv vs bf16 +1.6% +2% vLLM c256: tkv vs bf16 +2.4% +2% SGLang c256: tkv vs bf16 +3.5% +4% c256@0.8k concurrency @ median prompt length TTFT reduction: tkv vs bf16 0% 10% 20% 30% 40% 50% % lower TTFT arbi-serve c1: tkv vs bf16 +40.1% +40% vLLM c1: tkv vs bf16 +39.3% +39% SGLang c1: tkv vs bf16 +39.5% +39% c1@126k arbi-serve c8: tkv vs bf16 +10.7% +11% vLLM c8: tkv vs bf16 +15.0% +15% SGLang c8: tkv vs bf16 +22.6% +23% c8@14k arbi-serve c64: tkv vs bf16 +1.0% +1% vLLM c64: tkv vs bf16 +2.9% +3% SGLang c64: tkv vs bf16 +9.3% +9% c64@1.6k arbi-serve c256: tkv vs bf16 -0.2% -0.2% vLLM c256: tkv vs bf16 +2.1% +2% SGLang c256: tkv vs bf16 +4.0% +4% c256@0.8k concurrency @ median prompt length
Engine parity One colour per engine: the engines track each other, with the tkv lines (solid) above the bf16 lines (dashed).
Decode tok/s per stream 0 50 100 150 tok/s per stream c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 85.4 tok/s per stream arbi-serve bf16 c8: 68.7 tok/s per stream arbi-serve bf16 c64: 68.7 tok/s per stream arbi-serve bf16 c256: 27.5 tok/s per stream vLLM bf16 c1: 85.1 tok/s per stream vLLM bf16 c8: 70.8 tok/s per stream vLLM bf16 c64: 67.9 tok/s per stream vLLM bf16 c256: 27.2 tok/s per stream SGLang bf16 c1: 84.5 tok/s per stream SGLang bf16 c8: 69.3 tok/s per stream SGLang bf16 c64: 67.2 tok/s per stream SGLang bf16 c256: 23.6 tok/s per stream arbi-serve tkv c1: 134 tok/s per stream arbi-serve tkv c8: 94.0 tok/s per stream arbi-serve tkv c64: 98.6 tok/s per stream arbi-serve tkv c256: 33.0 tok/s per stream vLLM tkv c1: 135 tok/s per stream vLLM tkv c8: 93.0 tok/s per stream vLLM tkv c64: 96.7 tok/s per stream vLLM tkv c256: 33.2 tok/s per stream SGLang tkv c1: 132 tok/s per stream SGLang tkv c8: 92.9 tok/s per stream SGLang tkv c64: 94.8 tok/s per stream SGLang tkv c256: 30.5 tok/s per stream Prefill tok/s 0 2.5k 5k 7.5k 10k 12.5k prefill tok/s c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 5787 prefill tok/s arbi-serve bf16 c8: 4553 prefill tok/s arbi-serve bf16 c64: 933 prefill tok/s arbi-serve bf16 c256: 219 prefill tok/s vLLM bf16 c1: 5861 prefill tok/s vLLM bf16 c8: 4304 prefill tok/s vLLM bf16 c64: 820 prefill tok/s vLLM bf16 c256: 210 prefill tok/s SGLang bf16 c1: 5744 prefill tok/s SGLang bf16 c8: 4508 prefill tok/s SGLang bf16 c64: 872 prefill tok/s SGLang bf16 c256: 238 prefill tok/s arbi-serve tkv c1: 9660 prefill tok/s arbi-serve tkv c8: 5091 prefill tok/s arbi-serve tkv c64: 958 prefill tok/s arbi-serve tkv c256: 222 prefill tok/s vLLM tkv c1: 9664 prefill tok/s vLLM tkv c8: 5097 prefill tok/s vLLM tkv c64: 844 prefill tok/s vLLM tkv c256: 215 prefill tok/s SGLang tkv c1: 9489 prefill tok/s SGLang tkv c8: 5850 prefill tok/s SGLang tkv c64: 957 prefill tok/s SGLang tkv c256: 246 prefill tok/s Scaling with batch Total decode throughput over all concurrent streams.
Total decode tok/s (all streams) 0 2.5k 5k 7.5k 10k 12.5k tok/s, all streams c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 85 tok/s, all streams arbi-serve bf16 c8: 697 tok/s, all streams arbi-serve bf16 c64: 4738 tok/s, all streams arbi-serve bf16 c256: 8212 tok/s, all streams vLLM bf16 c1: 85 tok/s, all streams vLLM bf16 c8: 722 tok/s, all streams vLLM bf16 c64: 4706 tok/s, all streams vLLM bf16 c256: 8036 tok/s, all streams SGLang bf16 c1: 85 tok/s, all streams SGLang bf16 c8: 710 tok/s, all streams SGLang bf16 c64: 4648 tok/s, all streams SGLang bf16 c256: 6843 tok/s, all streams arbi-serve tkv c1: 134 tok/s, all streams arbi-serve tkv c8: 903 tok/s, all streams arbi-serve tkv c64: 6529 tok/s, all streams arbi-serve tkv c256: 9557 tok/s, all streams vLLM tkv c1: 135 tok/s, all streams vLLM tkv c8: 892 tok/s, all streams vLLM tkv c64: 6367 tok/s, all streams vLLM tkv c256: 9480 tok/s, all streams SGLang tkv c1: 132 tok/s, all streams SGLang tkv c8: 886 tok/s, all streams SGLang tkv c64: 6243 tok/s, all streams SGLang tkv c256: 8539 tok/s, all streams Validity: 24 of 24 cells are VALID under the all-N-admitted rule (each cell's own [validity] verdict). decode tok/s per stream = 1000 / median TPOT. prefill tok/s = prompt tokens / TTFT. TTFT = median ms. total decode = output tokens of all streams over their decode spans. c<N>@<len>: concurrency N at the cells' median prompt length, which the matrix varies with N. Percentages compare tkv with the same engine's bf16 at the same cell. Bars and y-axes start at 0. Charts show point values; the table below carries the run's spread.