Matrix A - minicpm5_2b - run 38337063638 (partial: 16 of 24 cells)

tkv advantage over each engine's own bf16 KV

Same engine, same workload, KV cache swapped: bf16 → tkv. Higher is better on every bar.

arbi-servevLLM
Decode tok/s per stream: tkv vs bf16
0%20%40%60%80%% vs bf16arbi-serve c1: tkv vs bf16 +56.5%+57%vLLM c1: tkv vs bf16 +57.5%+58%c1@126karbi-serve c8: tkv vs bf16 +36.2%+36%vLLM c8: tkv vs bf16 +30.9%+31%c8@14karbi-serve c64: tkv vs bf16 +43.6%+44%vLLM c64: tkv vs bf16 +42.3%+42%c64@1.6karbi-serve c256: tkv vs bf16 +14.9%+15%vLLM c256: tkv vs bf16 +19.2%+19%c256@0.8kconcurrency @ median prompt length
Prefill tok/s: tkv vs bf16
0%20%40%60%80%% vs bf16arbi-serve c1: tkv vs bf16 +66.6%+67%vLLM c1: tkv vs bf16 +62.4%+62%c1@126karbi-serve c8: tkv vs bf16 +22.5%+22%vLLM c8: tkv vs bf16 +16.4%+16%c8@14karbi-serve c64: tkv vs bf16 +3.3%+3%vLLM c64: tkv vs bf16 +2.4%+2%c64@1.6karbi-serve c256: tkv vs bf16 +1.7%+2%vLLM c256: tkv vs bf16 -0.6%-0.6%c256@0.8kconcurrency @ median prompt length
TTFT reduction: tkv vs bf16
0%10%20%30%40%50%% lower TTFTarbi-serve c1: tkv vs bf16 +40.0%+40%vLLM c1: tkv vs bf16 +38.4%+38%c1@126karbi-serve c8: tkv vs bf16 +4.3%+4%vLLM c8: tkv vs bf16 +13.0%+13%c8@14karbi-serve c64: tkv vs bf16 +2.1%+2%vLLM c64: tkv vs bf16 +2.1%+2%c64@1.6karbi-serve c256: tkv vs bf16 +1.8%+2%vLLM c256: tkv vs bf16 -1.6%-2%c256@0.8kconcurrency @ median prompt length

Engine parity

One colour per engine: the engines track each other, with the tkv lines (solid) above the bf16 lines (dashed).

arbi-servevLLMtkv KVbf16 KV
Decode tok/s per stream
050100150tok/s per streamc1@126kc8@14kc64@1.6kc256@0.8kconcurrency (log scale) @ median prompt lengtharbi-serve bf16 c1: 85.2 tok/s per streamarbi-serve bf16 c8: 69.3 tok/s per streamarbi-serve bf16 c64: 68.5 tok/s per streamarbi-serve bf16 c256: 27.7 tok/s per streamvLLM bf16 c1: 85.1 tok/s per streamvLLM bf16 c8: 70.4 tok/s per streamvLLM bf16 c64: 67.7 tok/s per streamvLLM bf16 c256: 26.7 tok/s per streamarbi-serve tkv c1: 133 tok/s per streamarbi-serve tkv c8: 94.4 tok/s per streamarbi-serve tkv c64: 98.4 tok/s per streamarbi-serve tkv c256: 31.9 tok/s per streamvLLM tkv c1: 134 tok/s per streamvLLM tkv c8: 92.2 tok/s per streamvLLM tkv c64: 96.3 tok/s per streamvLLM tkv c256: 31.8 tok/s per stream
Prefill tok/s
02.5k5k7.5k10k12.5kprefill tok/sc1@126kc8@14kc64@1.6kc256@0.8kconcurrency (log scale) @ median prompt lengtharbi-serve bf16 c1: 5760 prefill tok/sarbi-serve bf16 c8: 4469 prefill tok/sarbi-serve bf16 c64: 926 prefill tok/sarbi-serve bf16 c256: 216 prefill tok/svLLM bf16 c1: 5778 prefill tok/svLLM bf16 c8: 4210 prefill tok/svLLM bf16 c64: 800 prefill tok/svLLM bf16 c256: 207 prefill tok/sarbi-serve tkv c1: 9599 prefill tok/sarbi-serve tkv c8: 5472 prefill tok/sarbi-serve tkv c64: 957 prefill tok/sarbi-serve tkv c256: 219 prefill tok/svLLM tkv c1: 9383 prefill tok/svLLM tkv c8: 4901 prefill tok/svLLM tkv c64: 819 prefill tok/svLLM tkv c256: 206 prefill tok/s

Scaling with batch

Total decode throughput over all concurrent streams.

arbi-servevLLMtkv KVbf16 KV
Total decode tok/s (all streams)
02k4k6k8k10ktok/s, all streamsc1@126kc8@14kc64@1.6kc256@0.8kconcurrency (log scale) @ median prompt lengtharbi-serve bf16 c1: 85 tok/s, all streamsarbi-serve bf16 c8: 694 tok/s, all streamsarbi-serve bf16 c64: 4734 tok/s, all streamsarbi-serve bf16 c256: 8215 tok/s, all streamsvLLM bf16 c1: 85 tok/s, all streamsvLLM bf16 c8: 719 tok/s, all streamsvLLM bf16 c64: 4691 tok/s, all streamsvLLM bf16 c256: 7912 tok/s, all streamsarbi-serve tkv c1: 133 tok/s, all streamsarbi-serve tkv c8: 894 tok/s, all streamsarbi-serve tkv c64: 6506 tok/s, all streamsarbi-serve tkv c256: 9312 tok/s, all streamsvLLM tkv c1: 134 tok/s, all streamsvLLM tkv c8: 885 tok/s, all streamsvLLM tkv c64: 6340 tok/s, all streamsvLLM tkv c256: 9139 tok/s, all streams

Canonical table

Matrix A - minicpm5_2b - run 38337063638 (partial: 16 of 24 cells)
       │                                     c1@126k                                      │                                      c8@14k                                      │                                     c64@1.6k                                     │                                    c256@0.8k                                     
engine │ prefill tok/s decode/stream  decode total          TTFT          TPOT agg@900 tok/s │ prefill tok/s decode/stream  decode total          TTFT          TPOT agg@900 tok/s │ prefill tok/s decode/stream  decode total          TTFT          TPOT agg@900 tok/s │ prefill tok/s decode/stream  decode total          TTFT          TPOT agg@900 tok/s 
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
arbi   │     9599/5760      133/85.2        133/85   13442/22400      7.5/11.7         45/27 │     5472/4469     94.4/69.3       894/694     2718/2839     10.6/14.4       522/390 │       957/926     98.4/68.5     6506/4734     2251/2299     10.2/14.6     4898/3700 │       219/216     31.9/27.7     9312/8215     4543/4628     31.4/36.0     6319/5742 
vllm   │     9383/5778      134/85.1        134/85   13751/22328      7.5/11.8         44/27 │     4901/4210     92.2/70.4       885/719     3441/3957     10.8/14.2       497/398 │       819/800     96.3/67.7     6340/4691     2975/3039     10.4/14.8     4585/3547 │       206/207     31.8/26.7     9139/7912     4972/4894     31.4/37.4     6092/5517 
cells: tkv/bf16 · TTFT,TPOT=ms median · prefill=prompt tok / TTFT · decode/stream=1000/TPOT · decode total=all streams over their decode spans · agg=all streams @ 900 tok, prefill INCLUDED · best / worst>20% / worst-close / plain mid