ywkang/kernbench2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gqa decode: fix cube_base output-store; measure batch scaling on one SIP

Browse Source 
Concurrency fix (completes the cube_base change): the decode kernel's
output store o_base still used the global cube_id while every load used
the user-local cube_local. For a single user (cube_base=0) they coincide,
so it was masked; a second batched user (cube_base>0) overshot its o shard
into an unmapped VA, mis-decoded to a phantom sip0.cube0.pe8 and raised a
RoutingError. Switch o_base to cube_local (one line); single-user results
are byte-identical (decode smoke/correctness pass).

Batch harness (un-skipped): create all users' tensors first, then submit
all launches deferred and drain together, so deploys don't drive an
already-submitted launch and serialize the batch. Concurrent users on
disjoint CUBE groups now overlap to within 3-5% of the single-user
latency. Swept B in {1,2,capacity} at 8K (high-B dense runs are the
expensive ones; throughput is near-linear in B).

Measured result: aggregate throughput at a full SIP rises from 0.86
(1-kv-per-cube, 2 users) to 1.41 requests/us (8-kv-per-cube, 16 users) —
the dense mapping wins throughput, the spread mapping wins latency.
S5.2 batch paragraph updated projected -> measured; add batch_scaling
figure and plot_batch_scaling.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yangwook Kang
2026-07-22 17:37:10 -07:00
parent 8c108154a7
commit 13f5cbc317
7 changed files with 132 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sweeps/decode_batch_sweep.csv
+12
View File
@@ -0,0 +1,12 @@
mode,kv_per_cube,C,S_kv,B,capacity,agg_latency_us,mean_user_latency_us,throughput_users_per_us
A1,1,8,8192,1,2,2.2471645000005376,2.2471645000005376,0.44500524994932983
A1,1,8,8192,2,2,2.3131845000012543,2.2471645000012357,0.864608940617973
A2,2,4,8192,1,4,3.3555127500005475,3.3555127500005475,0.2980170467240325
A2,2,4,8192,2,4,3.3935227500010514,3.355512750001042,0.5893580645656141
A2,2,4,8192,4,4,3.4695427500010703,3.3555127500011586,1.1528896711241752
A4,4,2,8192,1,8,5.586135500000557,5.586135500000557,0.17901463364071643
A4,4,2,8192,2,8,5.602295500000823,5.586135500000673,0.35699652044411195
A4,4,2,8192,8,8,5.783285499986028,5.586135499984957,1.3832967436968704
B,8,1,8192,1,16,10.816091000000947,10.816091000000947,0.09245484343649775
B,8,1,8192,2,16,10.848411000001535,10.816091000001236,0.18435879687815265
B,8,1,8192,16,16,11.38492099986691,10.816090999974403,1.4053676788962384
1 mode kv_per_cube C S_kv B capacity agg_latency_us mean_user_latency_us throughput_users_per_us
2 A1 1 8 8192 1 2 2.2471645000005376 2.2471645000005376 0.44500524994932983
3 A1 1 8 8192 2 2 2.3131845000012543 2.2471645000012357 0.864608940617973
4 A2 2 4 8192 1 4 3.3555127500005475 3.3555127500005475 0.2980170467240325
5 A2 2 4 8192 2 4 3.3935227500010514 3.355512750001042 0.5893580645656141
6 A2 2 4 8192 4 4 3.4695427500010703 3.3555127500011586 1.1528896711241752
7 A4 4 2 8192 1 8 5.586135500000557 5.586135500000557 0.17901463364071643
8 A4 4 2 8192 2 8 5.602295500000823 5.586135500000673 0.35699652044411195
9 A4 4 2 8192 8 8 5.783285499986028 5.586135499984957 1.3832967436968704
10 B 8 1 8192 1 16 10.816091000000947 10.816091000000947 0.09245484343649775
11 B 8 1 8192 2 16 10.848411000001535 10.816091000001236 0.18435879687815265
12 B 8 1 8192 16 16 11.38492099986691 10.816090999974403 1.4053676788962384