ywkang/kernbench2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gqa short-context: exclude KV deploy from wall; correct HBM peak; batch scaffold

Browse Source 
Metric fixes (test harnesses, deploy artifact + wrong constant):
- wall = max(t_end) - min(t_start): exclude the one-time KV-cache deploy
  from the measured decode/prefill step (it was 92-99% of wall, mapping-
  invariant, and masked the real per-mapping separation).
- PEAK_PE_HBM_BPS 128 -> 256 B/ns (8 channels/PE x 32 GB/s); the old value
  was half the modeled per-PE HBM BW, so hbm_bw_util read >1.0 once wall
  was corrected. All six short-context sweep CSVs regenerated/repatched.

Result: the four KV mappings now separate along a {64,32,16,8}-active-PE
ladder (decode 8-kv/1-kv = 4.8x at 8K, 7.4x at 64K), not "modest/tied" as
before; decode is bandwidth-bound at 46-76% of the 256 GB/s per-PE ceiling.

Report (S5.2 rewrite):
- Replace the tied-wall / 8x-per-PE-util claims (both deploy artifacts)
  with the corrected separation and a density trade-off (per-CUBE KV
  footprint, Fig 16 top panel switched to a wall-invariant metric).
- Add a projected latency-vs-batched-throughput analysis (marked
  projected, not measured): dense mappings win throughput, 1-kv wins
  latency; converges at long context.
- Regenerate Fig 15/16/17/18; fix plot script hardcoded ROOT path.

Batch experiment (Part 2, cube_base):
- Add backward-compatible cube_base=0 scalar to the decode kernel so a
  batched user placed at DPPolicy.cube_start addresses 0-based shards.
  Default preserves single-user behavior (32 decode tests pass unchanged).
- New batch harness (skipped): concurrent B>=2 launches hit a sim routing
  issue (sip0.cube0.pe8); single-user path verified. Concurrency fix next.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yangwook Kang
2026-07-22 16:25:04 -07:00
parent cb6d21f145
commit 711a9a257f
23 changed files with 403 additions and 187 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sweeps/short_context_prefill_sweep.csv
+16 -16
View File
@@ -1,17 +1,17 @@
mode,kv_per_cube,C,S_kv,wall_us,n_pe,gemm_util,math_count,hbm_bw_util,hbm_read_mb,hbm_write_kb,ipcq_kb,kv_cache_per_cube_mb
A1,1,8,8192,1100.677128999906,64,0.0019053289513740923,5248,0.001875209310359147,16.0625,64.0,114688.0,2.0
A1,1,8,16384,2177.674413966317,64,0.0019260473343055045,10880,0.0018882528874050511,32.0625,64.0,229376.0,4.0
A1,1,8,32768,4334.294263847072,64,0.001935403433484189,22144,0.0018937339046091134,64.0625,64.0,458752.0,8.0
A1,1,8,65536,8649.037693613021,64,0.0019397783423315907,44672,0.0018961647042087428,128.0625,64.0,917504.0,16.0
A2,2,4,8192,1093.6804565019731,32,0.003835036070239803,2624,0.003774411415563731,16.0625,64.0,49152.0,4.0
A2,2,4,16384,2175.2205604341793,32,0.0038564401939640384,5440,0.0037807660287830624,32.0625,64.0,98304.0,8.0
A2,2,4,32768,4338.30076834928,32,0.003867232102121915,11072,0.003783970009586559,64.0625,64.0,196608.0,16.0
A2,2,4,65536,8664.461183990821,32,0.003872650738157816,22336,0.0037855787340364574,128.0625,64.0,393216.0,32.0
A4,4,2,8192,1103.915289502884,16,0.0075989598837576024,1312,0.007478834724463186,16.0625,64.0,16384.0,8.0
A4,4,2,16384,2197.52387042685,16,0.007634600117789289,2720,0.007484787865719574,32.0625,64.0,32768.0,16.0
A4,4,2,32768,4384.131109368463,16,0.0076536105246211355,5536,0.007488827131524694,64.0625,64.0,65536.0,32.0
A4,4,2,65536,8757.668308885037,16,0.007662868886230499,11168,0.007490578277947104,128.0625,64.0,131072.0,64.0
B,8,1,8192,1125.2657520018113,8,0.014909558893223249,656,0.014673867013748253,16.0625,64.0,0.0,16.0
B,8,1,16384,2241.999191943168,8,0.01496629977413786,1360,0.014672619026008048,32.0625,64.0,0.0,32.0
B,8,1,32768,4475.466071825993,8,0.014994832476216706,2768,0.014671991463273241,64.0625,64.0,0.0,64.0
B,8,1,65536,8942.39983160376,8,0.01500913966355881,5584,0.014671676783710771,128.0625,64.0,0.0,128.0
A1,1,8,8192,34.765078999497696,64,0.06032352177393696,5248,0.029685,16.0625,64.0,114688.0,2.0
A1,1,8,16384,62.2566439945763,64,0.06737118692698127,10880,0.0330246,32.0625,64.0,229376.0,4.0
A1,1,8,32768,117.23649402130023,64,0.07155287327558905,22144,0.0350062,64.0625,64.0,458752.0,8.0
A1,1,8,65536,227.190164006345,64,0.07384657726472504,44672,0.0360931,128.0625,64.0,917504.0,16.0
A2,2,4,8192,34.98345650027832,32,0.11989392757596813,2624,0.0589993,16.0625,64.0,49152.0,4.0
A2,2,4,16384,65.32612049105158,32,0.12841123790818756,5440,0.0629457,32.0625,64.0,98304.0,8.0
A2,2,4,32768,126.01144852332584,32,0.13314041062637966,11072,0.0651369,64.0625,64.0,196608.0,16.0
A2,2,4,65536,247.3821043861434,32,0.1356380732680083,22336,0.0662942,128.0625,64.0,393216.0,32.0
A4,4,2,8192,46.571509501108665,16,0.18012317165281877,1312,0.0886379,16.0625,64.0,16384.0,8.0
A4,4,2,16384,88.98265048367577,16,0.18854479956273557,2720,0.0924225,32.0625,64.0,32768.0,16.0
A4,4,2,32768,173.195009542468,16,0.19373786859461295,5536,0.0947833,64.0625,64.0,65536.0,32.0
A4,4,2,65536,341.9424492809754,16,0.19625777419912674,11168,0.0959226,128.0625,64.0,131072.0,64.0
B,8,1,8192,68.50162199999485,8,0.24491706196386726,656,0.120523,16.0625,64.0,0.0,16.0
B,8,1,16384,134.03762199999207,8,0.25033592434218427,1360,0.122712,32.0625,64.0,0.0,32.0
B,8,1,32768,265.1096219999986,8,0.25313628186615866,2768,0.123843,64.0625,64.0,0.0,64.0
B,8,1,65536,527.2536220000293,8,0.2545600872134325,5584,0.124418,128.0625,64.0,0.0,128.0
1 mode kv_per_cube C S_kv wall_us n_pe gemm_util math_count hbm_bw_util hbm_read_mb hbm_write_kb ipcq_kb kv_cache_per_cube_mb
2 A1 1 8 8192 1100.677128999906 34.765078999497696 64 0.0019053289513740923 0.06032352177393696 5248 0.001875209310359147 0.029685 16.0625 64.0 114688.0 2.0
3 A1 1 8 16384 2177.674413966317 62.2566439945763 64 0.0019260473343055045 0.06737118692698127 10880 0.0018882528874050511 0.0330246 32.0625 64.0 229376.0 4.0
4 A1 1 8 32768 4334.294263847072 117.23649402130023 64 0.001935403433484189 0.07155287327558905 22144 0.0018937339046091134 0.0350062 64.0625 64.0 458752.0 8.0
5 A1 1 8 65536 8649.037693613021 227.190164006345 64 0.0019397783423315907 0.07384657726472504 44672 0.0018961647042087428 0.0360931 128.0625 64.0 917504.0 16.0
6 A2 2 4 8192 1093.6804565019731 34.98345650027832 32 0.003835036070239803 0.11989392757596813 2624 0.003774411415563731 0.0589993 16.0625 64.0 49152.0 4.0
7 A2 2 4 16384 2175.2205604341793 65.32612049105158 32 0.0038564401939640384 0.12841123790818756 5440 0.0037807660287830624 0.0629457 32.0625 64.0 98304.0 8.0
8 A2 2 4 32768 4338.30076834928 126.01144852332584 32 0.003867232102121915 0.13314041062637966 11072 0.003783970009586559 0.0651369 64.0625 64.0 196608.0 16.0
9 A2 2 4 65536 8664.461183990821 247.3821043861434 32 0.003872650738157816 0.1356380732680083 22336 0.0037855787340364574 0.0662942 128.0625 64.0 393216.0 32.0
10 A4 4 2 8192 1103.915289502884 46.571509501108665 16 0.0075989598837576024 0.18012317165281877 1312 0.007478834724463186 0.0886379 16.0625 64.0 16384.0 8.0
11 A4 4 2 16384 2197.52387042685 88.98265048367577 16 0.007634600117789289 0.18854479956273557 2720 0.007484787865719574 0.0924225 32.0625 64.0 32768.0 16.0
12 A4 4 2 32768 4384.131109368463 173.195009542468 16 0.0076536105246211355 0.19373786859461295 5536 0.007488827131524694 0.0947833 64.0625 64.0 65536.0 32.0
13 A4 4 2 65536 8757.668308885037 341.9424492809754 16 0.007662868886230499 0.19625777419912674 11168 0.007490578277947104 0.0959226 128.0625 64.0 131072.0 64.0
14 B 8 1 8192 1125.2657520018113 68.50162199999485 8 0.014909558893223249 0.24491706196386726 656 0.014673867013748253 0.120523 16.0625 64.0 0.0 16.0
15 B 8 1 16384 2241.999191943168 134.03762199999207 8 0.01496629977413786 0.25033592434218427 1360 0.014672619026008048 0.122712 32.0625 64.0 0.0 32.0
16 B 8 1 32768 4475.466071825993 265.1096219999986 8 0.014994832476216706 0.25313628186615866 2768 0.014671991463273241 0.123843 64.0625 64.0 0.0 64.0
17 B 8 1 65536 8942.39983160376 527.2536220000293 8 0.01500913966355881 0.2545600872134325 5584 0.014671676783710771 0.124418 128.0625 64.0 0.0 128.0