tkv advantage over each engine's own bf16 KV Same engine, same workload, KV cache swapped: bf16 → tkv. Higher is better on every bar.
Decode tok/s per stream: tkv vs bf16 0% 20% 40% 60% 80% % vs bf16 arbi-serve c1: tkv vs bf16 +56.1% +56% vLLM c1: tkv vs bf16 +58.1% +58% SGLang c1: tkv vs bf16 +55.2% +55% c1@126k arbi-serve c8: tkv vs bf16 +38.6% +39% vLLM c8: tkv vs bf16 +32.3% +32% SGLang c8: tkv vs bf16 +34.6% +35% c8@14k arbi-serve c64: tkv vs bf16 +44.5% +45% vLLM c64: tkv vs bf16 +42.8% +43% SGLang c64: tkv vs bf16 +41.5% +41% c64@1.6k arbi-serve c256: tkv vs bf16 +12.7% +13% vLLM c256: tkv vs bf16 +23.1% +23% SGLang c256: tkv vs bf16 +24.4% +24% c256@0.8k concurrency @ median prompt length Prefill tok/s: tkv vs bf16 0% 20% 40% 60% 80% % vs bf16 arbi-serve c1: tkv vs bf16 +67.0% +67% vLLM c1: tkv vs bf16 +67.7% +68% SGLang c1: tkv vs bf16 +70.5% +70% c1@126k arbi-serve c8: tkv vs bf16 +22.9% +23% vLLM c8: tkv vs bf16 +20.0% +20% SGLang c8: tkv vs bf16 +32.3% +32% c8@14k arbi-serve c64: tkv vs bf16 +5.0% +5% vLLM c64: tkv vs bf16 +4.8% +5% SGLang c64: tkv vs bf16 +10.1% +10% c64@1.6k arbi-serve c256: tkv vs bf16 +2.8% +3% vLLM c256: tkv vs bf16 +2.7% +3% SGLang c256: tkv vs bf16 +3.5% +3% c256@0.8k concurrency @ median prompt length TTFT reduction: tkv vs bf16 0% 10% 20% 30% 40% 50% % lower TTFT arbi-serve c1: tkv vs bf16 +40.1% +40% vLLM c1: tkv vs bf16 +40.4% +40% SGLang c1: tkv vs bf16 +41.3% +41% c1@126k arbi-serve c8: tkv vs bf16 +11.6% +12% vLLM c8: tkv vs bf16 +16.3% +16% SGLang c8: tkv vs bf16 +24.0% +24% c8@14k arbi-serve c64: tkv vs bf16 +4.8% +5% vLLM c64: tkv vs bf16 +6.3% +6% SGLang c64: tkv vs bf16 +9.9% +10% c64@1.6k arbi-serve c256: tkv vs bf16 -0.3% -0.3% vLLM c256: tkv vs bf16 +2.6% +3% SGLang c256: tkv vs bf16 +4.1% +4% c256@0.8k concurrency @ median prompt length
Engine parity One colour per engine: the engines track each other, with the tkv lines (solid) above the bf16 lines (dashed).
Decode tok/s per stream 0 50 100 150 tok/s per stream c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 85.4 tok/s per stream arbi-serve bf16 c8: 68.0 tok/s per stream arbi-serve bf16 c64: 68.3 tok/s per stream arbi-serve bf16 c256: 27.5 tok/s per stream vLLM bf16 c1: 85.3 tok/s per stream vLLM bf16 c8: 70.8 tok/s per stream vLLM bf16 c64: 67.9 tok/s per stream vLLM bf16 c256: 27.2 tok/s per stream SGLang bf16 c1: 84.7 tok/s per stream SGLang bf16 c8: 69.5 tok/s per stream SGLang bf16 c64: 67.4 tok/s per stream SGLang bf16 c256: 23.8 tok/s per stream arbi-serve tkv c1: 133 tok/s per stream arbi-serve tkv c8: 94.3 tok/s per stream arbi-serve tkv c64: 98.7 tok/s per stream arbi-serve tkv c256: 31.0 tok/s per stream vLLM tkv c1: 135 tok/s per stream vLLM tkv c8: 93.6 tok/s per stream vLLM tkv c64: 97.0 tok/s per stream vLLM tkv c256: 33.5 tok/s per stream SGLang tkv c1: 131 tok/s per stream SGLang tkv c8: 93.5 tok/s per stream SGLang tkv c64: 95.3 tok/s per stream SGLang tkv c256: 29.7 tok/s per stream Prefill tok/s 0 2.5k 5k 7.5k 10k 12.5k prefill tok/s c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 5884 prefill tok/s arbi-serve bf16 c8: 4374 prefill tok/s arbi-serve bf16 c64: 925 prefill tok/s arbi-serve bf16 c256: 219 prefill tok/s vLLM bf16 c1: 5851 prefill tok/s vLLM bf16 c8: 4256 prefill tok/s vLLM bf16 c64: 816 prefill tok/s vLLM bf16 c256: 211 prefill tok/s SGLang bf16 c1: 5756 prefill tok/s SGLang bf16 c8: 4544 prefill tok/s SGLang bf16 c64: 894 prefill tok/s SGLang bf16 c256: 243 prefill tok/s arbi-serve tkv c1: 9827 prefill tok/s arbi-serve tkv c8: 5376 prefill tok/s arbi-serve tkv c64: 971 prefill tok/s arbi-serve tkv c256: 225 prefill tok/s vLLM tkv c1: 9809 prefill tok/s vLLM tkv c8: 5106 prefill tok/s vLLM tkv c64: 855 prefill tok/s vLLM tkv c256: 217 prefill tok/s SGLang tkv c1: 9811 prefill tok/s SGLang tkv c8: 6014 prefill tok/s SGLang tkv c64: 984 prefill tok/s SGLang tkv c256: 251 prefill tok/s Scaling with batch Total decode throughput over all concurrent streams.
Total decode tok/s (all streams) 0 2.5k 5k 7.5k 10k 12.5k tok/s, all streams c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 85 tok/s, all streams arbi-serve bf16 c8: 697 tok/s, all streams arbi-serve bf16 c64: 4726 tok/s, all streams arbi-serve bf16 c256: 8228 tok/s, all streams vLLM bf16 c1: 85 tok/s, all streams vLLM bf16 c8: 722 tok/s, all streams vLLM bf16 c64: 4706 tok/s, all streams vLLM bf16 c256: 8057 tok/s, all streams SGLang bf16 c1: 85 tok/s, all streams SGLang bf16 c8: 711 tok/s, all streams SGLang bf16 c64: 4665 tok/s, all streams SGLang bf16 c256: 6918 tok/s, all streams arbi-serve tkv c1: 134 tok/s, all streams arbi-serve tkv c8: 898 tok/s, all streams arbi-serve tkv c64: 6531 tok/s, all streams arbi-serve tkv c256: 9173 tok/s, all streams vLLM tkv c1: 135 tok/s, all streams vLLM tkv c8: 896 tok/s, all streams vLLM tkv c64: 6386 tok/s, all streams vLLM tkv c256: 9565 tok/s, all streams SGLang tkv c1: 131 tok/s, all streams SGLang tkv c8: 890 tok/s, all streams SGLang tkv c64: 6273 tok/s, all streams SGLang tkv c256: 8470 tok/s, all streams Validity: 24 of 24 cells are VALID under the all-N-admitted rule (each cell's own [validity] verdict). decode tok/s per stream = 1000 / median TPOT. prefill tok/s = prompt tokens / TTFT. TTFT = median ms. total decode = output tokens of all streams over their decode spans. c<N>@<len>: concurrency N at the cells' median prompt length, which the matrix varies with N. Percentages compare tkv with the same engine's bf16 at the same cell. Bars and y-axes start at 0. Charts show point values; the table below carries the run's spread.