ywkang/kernbench2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gqa short-context: exclude KV deploy from wall; correct HBM peak; batch scaffold

Browse Source 
Metric fixes (test harnesses, deploy artifact + wrong constant):
- wall = max(t_end) - min(t_start): exclude the one-time KV-cache deploy
  from the measured decode/prefill step (it was 92-99% of wall, mapping-
  invariant, and masked the real per-mapping separation).
- PEAK_PE_HBM_BPS 128 -> 256 B/ns (8 channels/PE x 32 GB/s); the old value
  was half the modeled per-PE HBM BW, so hbm_bw_util read >1.0 once wall
  was corrected. All six short-context sweep CSVs regenerated/repatched.

Result: the four KV mappings now separate along a {64,32,16,8}-active-PE
ladder (decode 8-kv/1-kv = 4.8x at 8K, 7.4x at 64K), not "modest/tied" as
before; decode is bandwidth-bound at 46-76% of the 256 GB/s per-PE ceiling.

Report (S5.2 rewrite):
- Replace the tied-wall / 8x-per-PE-util claims (both deploy artifacts)
  with the corrected separation and a density trade-off (per-CUBE KV
  footprint, Fig 16 top panel switched to a wall-invariant metric).
- Add a projected latency-vs-batched-throughput analysis (marked
  projected, not measured): dense mappings win throughput, 1-kv wins
  latency; converges at long context.
- Regenerate Fig 15/16/17/18; fix plot script hardcoded ROOT path.

Batch experiment (Part 2, cube_base):
- Add backward-compatible cube_base=0 scalar to the decode kernel so a
  batched user placed at DPPolicy.cube_start addresses 0-based shards.
  Default preserves single-user behavior (32 decode tests pass unchanged).
- New batch harness (skipped): concurrent B>=2 launches hit a sim routing
  issue (sip0.cube0.pe8); single-user path verified. Concurrency fix next.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yangwook Kang
2026-07-22 16:25:04 -07:00
parent cb6d21f145
commit 711a9a257f
23 changed files with 403 additions and 187 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sweeps/short_context_decode_sweep.csv
+16 -16
View File
@@ -1,17 +1,17 @@
mode,kv_per_cube,C,S_kv,wall_us,n_pe,gemm_util,math_count,hbm_bw_util,hbm_read_mb,hbm_write_kb,ipcq_kb,kv_cache_per_cube_mb
A1,1,8,8192,146.95153450000072,64,0.0017838806576053131,656,0.013997812319543966,16.0625,8.0,57.75,2.0
A1,1,8,16384,279.3832665000025,64,0.0018765905580817512,1360,0.014693077546933053,32.0625,8.0,57.75,4.0
A1,1,8,32768,544.246730500006,64,0.0019266555796063503,2768,0.015068533333155061,64.0625,8.0,57.75,8.0
A1,1,8,65536,1074.081058500367,64,0.0019525081309290412,5584,0.01526234902874828,128.0625,8.0,57.75,16.0
A2,2,4,8192,139.87083275000066,32,0.003748372621310069,656,0.029355655637933423,16.03125,8.0,24.75,4.0
A2,2,4,16384,273.6622967500018,32,0.0038316421825465713,1360,0.029971245938539923,32.03125,8.0,24.75,8.0
A2,2,4,32768,541.3526247504072,32,0.0038739112070761647,2768,0.030283403553383558,64.03125,8.0,24.75,16.0
A2,2,4,65536,1077.4482007518286,32,0.003892812663356761,5584,0.030421880121130614,128.03125,8.0,24.75,32.0
A4,4,2,8192,139.89951550000075,16,0.007495208230366452,656,0.05864209015076937,16.015625,8.0,8.25,8.0
A4,4,2,16384,276.4641435001754,16,0.007585620230706115,1360,0.05930606331952627,32.015625,8.0,8.25,16.0
A4,4,2,32768,549.9508595010393,16,0.007626688689616412,2768,0.059605325518975856,64.015625,8.0,8.25,32.0
A4,4,2,65536,1097.305291501857,16,0.0076447348472311214,5584,0.059735426874945555,128.015625,8.0,8.25,64.0
B,8,1,8192,144.06747100006032,8,0.014556735017573057,656,0.11383555139933797,16.0078125,8.0,0.0,16.0
B,8,1,16384,286.24975700042637,8,0.014652602831705778,1360,0.1145293548666704,32.0078125,8.0,0.0,32.0
B,8,1,32768,570.8048290010414,8,0.014696105522939606,2768,0.11484135499470417,64.0078125,8.0,0.0,64.0
B,8,1,65536,1139.9149730017834,8,0.014717953880200899,5584,0.11499805082373886,128.0078125,8.0,0.0,128.0
A1,1,8,8192,2.2471645000005376,64,0.11665545624270428,656,0.457688,16.0625,8.0,57.75,2.0
A1,1,8,16384,3.6068965000013704,64,0.14535709577464812,1360,0.569049,32.0625,8.0,57.75,4.0
A1,1,8,32768,6.326360500001814,64,0.1657471147905734,2768,0.648161,64.0625,8.0,57.75,8.0
A1,1,8,65536,11.765288499959512,64,0.17824909265982342,5584,0.696668,128.0625,8.0,57.75,16.0
A2,2,4,8192,3.3555127500005475,32,0.15624676139283236,656,0.611829,16.03125,8.0,24.75,4.0
A2,2,4,16384,6.074976750001311,32,0.17260576347056114,1360,0.675064,32.03125,8.0,24.75,8.0
A2,2,4,32768,11.513904750002315,32,0.18214081543451005,2768,0.711922,64.03125,8.0,24.75,16.0
A2,2,4,65536,22.391760749934242,32,0.18731461303283142,5584,0.731921,128.03125,8.0,24.75,32.0
A4,4,2,8192,5.586135500000557,16,0.1877104484844272,656,0.734318,16.015625,8.0,8.25,8.0
A4,4,2,16384,11.025063500001211,16,0.19021677290108474,1360,0.743578,32.015625,8.0,8.25,16.0
A4,4,2,32768,21.902919500003218,16,0.19149520227204342,2768,0.748302,64.015625,8.0,8.25,32.0
A4,4,2,65536,43.65863149988279,16,0.19214088284049563,5584,0.750688,128.015625,8.0,8.25,64.0
B,8,1,8192,10.816091000000947,8,0.1938918598225168,656,0.75813,16.0078125,8.0,0.0,16.0
B,8,1,16384,21.693947000002314,8,0.1933398288471476,1360,0.755602,32.0078125,8.0,0.0,32.0
B,8,1,32768,43.449659000006385,8,0.19306499045255035,2768,0.754344,64.0078125,8.0,0.0,64.0
B,8,1,65536,86.96108299976913,8,0.19292786406575965,5584,0.753716,128.0078125,8.0,0.0,128.0
1 mode kv_per_cube C S_kv wall_us n_pe gemm_util math_count hbm_bw_util hbm_read_mb hbm_write_kb ipcq_kb kv_cache_per_cube_mb
2 A1 1 8 8192 146.95153450000072 2.2471645000005376 64 0.0017838806576053131 0.11665545624270428 656 0.013997812319543966 0.457688 16.0625 8.0 57.75 2.0
3 A1 1 8 16384 279.3832665000025 3.6068965000013704 64 0.0018765905580817512 0.14535709577464812 1360 0.014693077546933053 0.569049 32.0625 8.0 57.75 4.0
4 A1 1 8 32768 544.246730500006 6.326360500001814 64 0.0019266555796063503 0.1657471147905734 2768 0.015068533333155061 0.648161 64.0625 8.0 57.75 8.0
5 A1 1 8 65536 1074.081058500367 11.765288499959512 64 0.0019525081309290412 0.17824909265982342 5584 0.01526234902874828 0.696668 128.0625 8.0 57.75 16.0
6 A2 2 4 8192 139.87083275000066 3.3555127500005475 32 0.003748372621310069 0.15624676139283236 656 0.029355655637933423 0.611829 16.03125 8.0 24.75 4.0
7 A2 2 4 16384 273.6622967500018 6.074976750001311 32 0.0038316421825465713 0.17260576347056114 1360 0.029971245938539923 0.675064 32.03125 8.0 24.75 8.0
8 A2 2 4 32768 541.3526247504072 11.513904750002315 32 0.0038739112070761647 0.18214081543451005 2768 0.030283403553383558 0.711922 64.03125 8.0 24.75 16.0
9 A2 2 4 65536 1077.4482007518286 22.391760749934242 32 0.003892812663356761 0.18731461303283142 5584 0.030421880121130614 0.731921 128.03125 8.0 24.75 32.0
10 A4 4 2 8192 139.89951550000075 5.586135500000557 16 0.007495208230366452 0.1877104484844272 656 0.05864209015076937 0.734318 16.015625 8.0 8.25 8.0
11 A4 4 2 16384 276.4641435001754 11.025063500001211 16 0.007585620230706115 0.19021677290108474 1360 0.05930606331952627 0.743578 32.015625 8.0 8.25 16.0
12 A4 4 2 32768 549.9508595010393 21.902919500003218 16 0.007626688689616412 0.19149520227204342 2768 0.059605325518975856 0.748302 64.015625 8.0 8.25 32.0
13 A4 4 2 65536 1097.305291501857 43.65863149988279 16 0.0076447348472311214 0.19214088284049563 5584 0.059735426874945555 0.750688 128.015625 8.0 8.25 64.0
14 B 8 1 8192 144.06747100006032 10.816091000000947 8 0.014556735017573057 0.1938918598225168 656 0.11383555139933797 0.75813 16.0078125 8.0 0.0 16.0
15 B 8 1 16384 286.24975700042637 21.693947000002314 8 0.014652602831705778 0.1933398288471476 1360 0.1145293548666704 0.755602 32.0078125 8.0 0.0 32.0
16 B 8 1 32768 570.8048290010414 43.449659000006385 8 0.014696105522939606 0.19306499045255035 2768 0.11484135499470417 0.754344 64.0078125 8.0 0.0 64.0
17 B 8 1 65536 1139.9149730017834 86.96108299976913 8 0.014717953880200899 0.19292786406575965 5584 0.11499805082373886 0.753716 128.0078125 8.0 0.0 128.0