ywkang/kernbench2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gqa short-context: exclude KV deploy from wall; correct HBM peak; batch scaffold

Browse Source 
Metric fixes (test harnesses, deploy artifact + wrong constant):
- wall = max(t_end) - min(t_start): exclude the one-time KV-cache deploy
  from the measured decode/prefill step (it was 92-99% of wall, mapping-
  invariant, and masked the real per-mapping separation).
- PEAK_PE_HBM_BPS 128 -> 256 B/ns (8 channels/PE x 32 GB/s); the old value
  was half the modeled per-PE HBM BW, so hbm_bw_util read >1.0 once wall
  was corrected. All six short-context sweep CSVs regenerated/repatched.

Result: the four KV mappings now separate along a {64,32,16,8}-active-PE
ladder (decode 8-kv/1-kv = 4.8x at 8K, 7.4x at 64K), not "modest/tied" as
before; decode is bandwidth-bound at 46-76% of the 256 GB/s per-PE ceiling.

Report (S5.2 rewrite):
- Replace the tied-wall / 8x-per-PE-util claims (both deploy artifacts)
  with the corrected separation and a density trade-off (per-CUBE KV
  footprint, Fig 16 top panel switched to a wall-invariant metric).
- Add a projected latency-vs-batched-throughput analysis (marked
  projected, not measured): dense mappings win throughput, 1-kv wins
  latency; converges at long context.
- Regenerate Fig 15/16/17/18; fix plot script hardcoded ROOT path.

Batch experiment (Part 2, cube_base):
- Add backward-compatible cube_base=0 scalar to the decode kernel so a
  batched user placed at DPPolicy.cube_start addresses 0-based shards.
  Default preserves single-user behavior (32 decode tests pass unchanged).
- New batch harness (skipped): concurrent B>=2 launches hit a sim routing
  issue (sip0.cube0.pe8); single-user path verified. Concurrency fix next.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yangwook Kang
2026-07-22 16:25:04 -07:00
parent cb6d21f145
commit 711a9a257f
23 changed files with 403 additions and 187 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sweeps/short_context_prefill_composite_fused_sweep.csv
+16 -16
View File
@@ -1,17 +1,17 @@
variant,mode,kv_per_cube,C,S_kv,wall_us,n_pe,gemm_util,math_count,math_pipeline_us,hbm_bw_util,hbm_read_mb,hbm_write_kb,ipcq_kb,kv_cache_per_cube_mb
composite_fused,A1,1,8,8192,1101.4436329998505,64,0.011331194467560951,4800,97.32531200143696,0.0018739043362378583,16.0625,64.0,114688.0,2.0
composite_fused,A1,1,8,16384,2178.44091796638,64,0.012139279878164511,9920,208.55424000307917,0.0018875884886695197,32.0625,64.0,229376.0,4.0
composite_fused,A1,1,8,32768,4335.060767847123,64,0.012542561892233649,20160,431.01209600543973,0.0018933990639481287,64.0625,64.0,458752.0,8.0
composite_fused,A1,1,8,65536,8649.804197613514,64,0.012743513431742019,40640,875.9278079059123,0.0018959966752224023,128.0625,64.0,917504.0,16.0
composite_fused,A2,2,4,8192,1093.9720835019473,32,0.01853321150177446,2400,96.40870400077105,0.0037734052470386023,16.0625,64.0,49152.0,4.0
composite_fused,A2,2,4,16384,2175.4760434340433,32,0.019695389489835093,4960,206.59008000165224,0.00378032202414797,32.0625,64.0,98304.0,8.0
composite_fused,A2,2,4,32768,4338.482933349601,32,0.020281493171481216,10080,426.9528320015669,0.003783811127574436,64.0625,64.0,196608.0,16.0
composite_fused,A2,2,4,65536,8664.499802991075,32,0.020575802877008553,20320,867.678335950613,0.0037855618611332994,128.0625,64.0,393216.0,32.0
composite_fused,A4,4,2,8192,1102.9720360028073,16,0.03256574312686088,1200,92.28038400042057,0.007485230568419403,16.0625,64.0,16384.0,8.0
composite_fused,A4,4,2,16384,2196.357595426575,16,0.03449854256839416,2480,197.74368000090124,0.007488762319145705,32.0625,64.0,32768.0,16.0
composite_fused,A4,4,2,32768,4383.130075368471,16,0.03547127219906801,5040,408.6702720000148,0.00749053745507198,64.0625,64.0,65536.0,32.0
composite_fused,A4,4,2,65536,8756.675034884938,16,0.03595923415220853,10160,830.5234559737444,0.007491427937962982,128.0625,64.0,131072.0,64.0
composite_fused,B,8,1,8192,1125.1019960018205,8,0.12366810519999533,712,121.6138560003899,0.014676002761240575,16.0625,64.0,0.0,16.0
composite_fused,B,8,1,16384,2241.8354359431773,8,0.13192705729555512,1480,260.6011200008355,0.0146736907948643,32.0625,64.0,0.0,32.0
composite_fused,B,8,1,32768,4475.302315825972,8,0.1360796989728368,3016,538.575647996068,0.014672528326811126,64.0625,64.0,0.0,64.0
composite_fused,B,8,1,65536,8942.236075603707,8,0.13816184916758104,6088,1094.5247039788662,0.014671945460927953,128.0625,64.0,0.0,128.0
composite_fused,A1,1,8,8192,35.53158299944201,64,0.3512557265116541,4800,97.32531200143696,0.0290446,16.0625,64.0,114688.0,2.0
composite_fused,A1,1,8,16384,63.02314799463935,64,0.4196030322618738,9920,208.55424000307917,0.0326229,32.0625,64.0,229376.0,4.0
composite_fused,A1,1,8,32768,118.00299802135117,64,0.4607744625054221,20160,431.01209600543973,0.0347788,64.0625,64.0,458752.0,8.0
composite_fused,A1,1,8,65536,227.95666800683736,64,0.4835519703723695,40640,875.9278079059123,0.0359717,128.0625,64.0,917504.0,16.0
composite_fused,A2,2,4,8192,35.27508350025257,32,0.5747630902258046,2400,96.40870400077105,0.0585116,16.0625,64.0,49152.0,4.0
composite_fused,A2,2,4,16384,65.58160349091561,32,0.653336388872743,4960,206.59008000165224,0.0627005,32.0625,64.0,98304.0,8.0
composite_fused,A2,2,4,32768,126.19361352364719,32,0.69726913692688,10080,426.9528320015669,0.0650429,64.0625,64.0,196608.0,16.0
composite_fused,A2,2,4,65536,247.42072338639758,32,0.7205501525262498,20320,867.678335950613,0.0662839,128.0625,64.0,393216.0,32.0
composite_fused,A4,4,2,8192,45.62825600103196,16,0.7872118539828873,1200,92.28038400042057,0.0904703,16.0625,64.0,16384.0,8.0
composite_fused,A4,4,2,16384,87.81637548340065,16,0.8628360665552867,2480,197.74368000090124,0.09365,32.0625,64.0,32768.0,16.0
composite_fused,A4,4,2,32768,172.19397554247547,16,0.9029073142513345,5040,408.6702720000148,0.0953343,64.0625,64.0,65536.0,32.0
composite_fused,A4,4,2,65536,340.9491752808783,16,0.9235491703853554,10160,830.5234559737444,0.096202,128.0625,64.0,131072.0,64.0
composite_fused,B,8,1,8192,68.33786600000411,8,2.036048828365075,712,121.6138560003899,0.120811,16.0625,64.0,0.0,16.0
composite_fused,B,8,1,16384,133.87386600000133,8,2.209234414765302,1480,260.6011200008355,0.122862,32.0625,64.0,0.0,32.0
composite_fused,B,8,1,32768,264.9458659999771,8,2.2985744263323977,3016,538.575647996068,0.12392,64.0625,64.0,0.0,64.0
composite_fused,B,8,1,65536,527.0898659999762,8,2.3439567929361727,6088,1094.5247039788662,0.124457,128.0625,64.0,0.0,128.0
1 variant mode kv_per_cube C S_kv wall_us n_pe gemm_util math_count math_pipeline_us hbm_bw_util hbm_read_mb hbm_write_kb ipcq_kb kv_cache_per_cube_mb
2 composite_fused A1 1 8 8192 1101.4436329998505 35.53158299944201 64 0.011331194467560951 0.3512557265116541 4800 97.32531200143696 0.0018739043362378583 0.0290446 16.0625 64.0 114688.0 2.0
3 composite_fused A1 1 8 16384 2178.44091796638 63.02314799463935 64 0.012139279878164511 0.4196030322618738 9920 208.55424000307917 0.0018875884886695197 0.0326229 32.0625 64.0 229376.0 4.0
4 composite_fused A1 1 8 32768 4335.060767847123 118.00299802135117 64 0.012542561892233649 0.4607744625054221 20160 431.01209600543973 0.0018933990639481287 0.0347788 64.0625 64.0 458752.0 8.0
5 composite_fused A1 1 8 65536 8649.804197613514 227.95666800683736 64 0.012743513431742019 0.4835519703723695 40640 875.9278079059123 0.0018959966752224023 0.0359717 128.0625 64.0 917504.0 16.0
6 composite_fused A2 2 4 8192 1093.9720835019473 35.27508350025257 32 0.01853321150177446 0.5747630902258046 2400 96.40870400077105 0.0037734052470386023 0.0585116 16.0625 64.0 49152.0 4.0
7 composite_fused A2 2 4 16384 2175.4760434340433 65.58160349091561 32 0.019695389489835093 0.653336388872743 4960 206.59008000165224 0.00378032202414797 0.0627005 32.0625 64.0 98304.0 8.0
8 composite_fused A2 2 4 32768 4338.482933349601 126.19361352364719 32 0.020281493171481216 0.69726913692688 10080 426.9528320015669 0.003783811127574436 0.0650429 64.0625 64.0 196608.0 16.0
9 composite_fused A2 2 4 65536 8664.499802991075 247.42072338639758 32 0.020575802877008553 0.7205501525262498 20320 867.678335950613 0.0037855618611332994 0.0662839 128.0625 64.0 393216.0 32.0
10 composite_fused A4 4 2 8192 1102.9720360028073 45.62825600103196 16 0.03256574312686088 0.7872118539828873 1200 92.28038400042057 0.007485230568419403 0.0904703 16.0625 64.0 16384.0 8.0
11 composite_fused A4 4 2 16384 2196.357595426575 87.81637548340065 16 0.03449854256839416 0.8628360665552867 2480 197.74368000090124 0.007488762319145705 0.09365 32.0625 64.0 32768.0 16.0
12 composite_fused A4 4 2 32768 4383.130075368471 172.19397554247547 16 0.03547127219906801 0.9029073142513345 5040 408.6702720000148 0.00749053745507198 0.0953343 64.0625 64.0 65536.0 32.0
13 composite_fused A4 4 2 65536 8756.675034884938 340.9491752808783 16 0.03595923415220853 0.9235491703853554 10160 830.5234559737444 0.007491427937962982 0.096202 128.0625 64.0 131072.0 64.0
14 composite_fused B 8 1 8192 1125.1019960018205 68.33786600000411 8 0.12366810519999533 2.036048828365075 712 121.6138560003899 0.014676002761240575 0.120811 16.0625 64.0 0.0 16.0
15 composite_fused B 8 1 16384 2241.8354359431773 133.87386600000133 8 0.13192705729555512 2.209234414765302 1480 260.6011200008355 0.0146736907948643 0.122862 32.0625 64.0 0.0 32.0
16 composite_fused B 8 1 32768 4475.302315825972 264.9458659999771 8 0.1360796989728368 2.2985744263323977 3016 538.575647996068 0.014672528326811126 0.12392 64.0625 64.0 0.0 64.0
17 composite_fused B 8 1 65536 8942.236075603707 527.0898659999762 8 0.13816184916758104 2.3439567929361727 6088 1094.5247039788662 0.014671945460927953 0.124457 128.0625 64.0 0.0 128.0