tkv advantage over each engine's own bf16 KV Same engine, same workload, KV cache swapped: bf16 → tkv. Higher is better on every bar.
Decode tok/s per stream: tkv vs bf16 0% 20% 40% 60% 80% % vs bf16 arbi-serve c1: tkv vs bf16 +56.9% +57% vLLM c1: tkv vs bf16 +58.3% +58% SGLang c1: tkv vs bf16 +56.7% +57% c1@126k arbi-serve c8: tkv vs bf16 +36.5% +37% vLLM c8: tkv vs bf16 +31.1% +31% SGLang c8: tkv vs bf16 +33.9% +34% c8@14k arbi-serve c64: tkv vs bf16 +32.8% +33% SGLang c64: tkv vs bf16 +41.1% +41% c64@1.6k vLLM c256: tkv vs bf16 +21.1% +21% SGLang c256: tkv vs bf16 +29.5% +29% c256@0.8k concurrency @ median prompt length Prefill tok/s: tkv vs bf16 0% 20% 40% 60% 80% % vs bf16 arbi-serve c1: tkv vs bf16 +67.6% +68% vLLM c1: tkv vs bf16 +63.6% +64% SGLang c1: tkv vs bf16 +63.2% +63% c1@126k arbi-serve c8: tkv vs bf16 +22.5% +22% vLLM c8: tkv vs bf16 +15.1% +15% SGLang c8: tkv vs bf16 +29.3% +29% c8@14k arbi-serve c64: tkv vs bf16 +4.3% +4% SGLang c64: tkv vs bf16 +8.8% +9% c64@1.6k vLLM c256: tkv vs bf16 +0.6% +0.6% SGLang c256: tkv vs bf16 +4.6% +5% c256@0.8k concurrency @ median prompt length TTFT reduction: tkv vs bf16 0% 10% 20% 30% 40% 50% % lower TTFT arbi-serve c1: tkv vs bf16 +40.3% +40% vLLM c1: tkv vs bf16 +38.9% +39% SGLang c1: tkv vs bf16 +38.7% +39% c1@126k arbi-serve c8: tkv vs bf16 +16.6% +17% vLLM c8: tkv vs bf16 +10.9% +11% SGLang c8: tkv vs bf16 +22.4% +22% c8@14k arbi-serve c64: tkv vs bf16 +6.8% +7% SGLang c64: tkv vs bf16 +8.9% +9% c64@1.6k vLLM c256: tkv vs bf16 +0.6% +0.6% SGLang c256: tkv vs bf16 +4.8% +5% c256@0.8k concurrency @ median prompt length
Engine parity One colour per engine: the engines track each other, with the tkv lines (solid) above the bf16 lines (dashed).
Decode tok/s per stream 0 50 100 150 tok/s per stream c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 80.6 tok/s per stream arbi-serve bf16 c8: 61.2 tok/s per stream arbi-serve bf16 c64: 55.9 tok/s per stream vLLM bf16 c1: 85.0 tok/s per stream vLLM bf16 c8: 70.8 tok/s per stream vLLM bf16 c64: 67.9 tok/s per stream vLLM bf16 c256: 27.1 tok/s per stream SGLang bf16 c1: 84.3 tok/s per stream SGLang bf16 c8: 69.1 tok/s per stream SGLang bf16 c64: 67.1 tok/s per stream SGLang bf16 c256: 23.4 tok/s per stream arbi-serve tkv c1: 126 tok/s per stream arbi-serve tkv c8: 83.6 tok/s per stream arbi-serve tkv c64: 74.2 tok/s per stream vLLM tkv c1: 135 tok/s per stream vLLM tkv c8: 92.9 tok/s per stream vLLM tkv c256: 32.8 tok/s per stream SGLang tkv c1: 132 tok/s per stream SGLang tkv c8: 92.5 tok/s per stream SGLang tkv c64: 94.6 tok/s per stream SGLang tkv c256: 30.3 tok/s per stream Prefill tok/s 0 2.5k 5k 7.5k 10k 12.5k prefill tok/s c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 5766 prefill tok/s arbi-serve bf16 c8: 4715 prefill tok/s arbi-serve bf16 c64: 882 prefill tok/s vLLM bf16 c1: 5858 prefill tok/s vLLM bf16 c8: 4362 prefill tok/s vLLM bf16 c64: 817 prefill tok/s vLLM bf16 c256: 212 prefill tok/s SGLang bf16 c1: 5730 prefill tok/s SGLang bf16 c8: 4464 prefill tok/s SGLang bf16 c64: 867 prefill tok/s SGLang bf16 c256: 233 prefill tok/s arbi-serve tkv c1: 9663 prefill tok/s arbi-serve tkv c8: 5774 prefill tok/s arbi-serve tkv c64: 919 prefill tok/s vLLM tkv c1: 9587 prefill tok/s vLLM tkv c8: 5019 prefill tok/s vLLM tkv c256: 213 prefill tok/s SGLang tkv c1: 9352 prefill tok/s SGLang tkv c8: 5772 prefill tok/s SGLang tkv c64: 943 prefill tok/s SGLang tkv c256: 244 prefill tok/s Scaling with batch Total decode throughput over all concurrent streams.
Total decode tok/s (all streams) 0 2k 4k 6k 8k 10k tok/s, all streams c1@126k c8@14k c64@1.6k c256@0.8k concurrency (log scale) @ median prompt length arbi-serve bf16 c1: 81 tok/s, all streams arbi-serve bf16 c8: 627 tok/s, all streams arbi-serve bf16 c64: 3961 tok/s, all streams vLLM bf16 c1: 85 tok/s, all streams vLLM bf16 c8: 722 tok/s, all streams vLLM bf16 c64: 4708 tok/s, all streams vLLM bf16 c256: 8030 tok/s, all streams SGLang bf16 c1: 84 tok/s, all streams SGLang bf16 c8: 708 tok/s, all streams SGLang bf16 c64: 4644 tok/s, all streams SGLang bf16 c256: 6790 tok/s, all streams arbi-serve tkv c1: 126 tok/s, all streams arbi-serve tkv c8: 802 tok/s, all streams arbi-serve tkv c64: 5172 tok/s, all streams vLLM tkv c1: 135 tok/s, all streams vLLM tkv c8: 891 tok/s, all streams vLLM tkv c256: 9396 tok/s, all streams SGLang tkv c1: 132 tok/s, all streams SGLang tkv c8: 884 tok/s, all streams SGLang tkv c64: 6230 tok/s, all streams SGLang tkv c256: 8474 tok/s, all streams Validity: 21 of 21 cells are VALID under the all-N-admitted rule (each cell's own [validity] verdict). decode tok/s per stream = 1000 / median TPOT. prefill tok/s = prompt tokens / TTFT. TTFT = median ms. total decode = output tokens of all streams over their decode spans. c<N>@<len>: concurrency N at the cells' median prompt length, which the matrix varies with N. Percentages compare tkv with the same engine's bf16 at the same cell. Bars and y-axes start at 0. Charts show point values; the table below carries the run's spread.