ywkang/kernbench2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gqa short-context: exclude KV deploy from wall; correct HBM peak; batch scaffold

Browse Source 
Metric fixes (test harnesses, deploy artifact + wrong constant):
- wall = max(t_end) - min(t_start): exclude the one-time KV-cache deploy
  from the measured decode/prefill step (it was 92-99% of wall, mapping-
  invariant, and masked the real per-mapping separation).
- PEAK_PE_HBM_BPS 128 -> 256 B/ns (8 channels/PE x 32 GB/s); the old value
  was half the modeled per-PE HBM BW, so hbm_bw_util read >1.0 once wall
  was corrected. All six short-context sweep CSVs regenerated/repatched.

Result: the four KV mappings now separate along a {64,32,16,8}-active-PE
ladder (decode 8-kv/1-kv = 4.8x at 8K, 7.4x at 64K), not "modest/tied" as
before; decode is bandwidth-bound at 46-76% of the 256 GB/s per-PE ceiling.

Report (S5.2 rewrite):
- Replace the tied-wall / 8x-per-PE-util claims (both deploy artifacts)
  with the corrected separation and a density trade-off (per-CUBE KV
  footprint, Fig 16 top panel switched to a wall-invariant metric).
- Add a projected latency-vs-batched-throughput analysis (marked
  projected, not measured): dense mappings win throughput, 1-kv wins
  latency; converges at long context.
- Regenerate Fig 15/16/17/18; fix plot script hardcoded ROOT path.

Batch experiment (Part 2, cube_base):
- Add backward-compatible cube_base=0 scalar to the decode kernel so a
  batched user placed at DPPolicy.cube_start addresses 0-based shards.
  Default preserves single-user behavior (32 decode tests pass unchanged).
- New batch harness (skipped): concurrent B>=2 launches hit a sim routing
  issue (sip0.cube0.pe8); single-user path verified. Concurrency fix next.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yangwook Kang
2026-07-22 16:25:04 -07:00
parent cb6d21f145
commit 711a9a257f
23 changed files with 403 additions and 187 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sweeps/short_context_decode_composite_fused_sweep.csv
+16 -16
View File
@@ -1,17 +1,17 @@
variant,mode,kv_per_cube,C,S_kv,wall_us,n_pe,gemm_util,math_count,math_pipeline_us,hbm_bw_util,hbm_read_mb,hbm_write_kb,ipcq_kb,kv_cache_per_cube_mb
composite_fused,A1,1,8,8192,146.95153450000072,64,0.0017838806576053131,656,0.0,0.013997812319543966,16.0625,8.0,57.75,2.0
composite_fused,A1,1,8,16384,279.867342500003,64,0.007173570099630665,1296,13.903615999992937,0.0146676634841736,32.0625,8.0,57.75,4.0
composite_fused,A1,1,8,32768,544.5763825000042,64,0.010097125355742134,2576,41.71084799996763,0.015059411798858053,64.0625,8.0,57.75,8.0
composite_fused,A1,1,8,65536,1074.101862500392,64,0.011619635377527878,5136,97.32531200143696,0.015262053416273654,128.0625,8.0,57.75,16.0
composite_fused,A2,2,4,8192,140.35490875000028,32,0.01430408111743636,624,6.951807999995537,0.029254409671653128,16.03125,8.0,24.75,4.0
composite_fused,A2,2,4,16384,273.9919487500021,32,0.020068677291975947,1264,20.855423999989405,0.029935186188568382,32.03125,8.0,24.75,8.0
composite_fused,A2,2,4,32768,541.3734287504049,32,0.0230537210304644,2544,48.66265599996224,0.030282239817052973,64.03125,8.0,24.75,16.0
composite_fused,A2,2,4,65536,1076.851308751879,32,0.024557433125924955,5104,104.27712000153959,0.030438742780552714,128.03125,8.0,24.75,32.0
composite_fused,A4,4,2,8192,140.2291675000001,16,0.03921192786066565,608,10.427711999993306,0.058504233792873325,16.015625,8.0,8.25,8.0
composite_fused,A4,4,2,16384,276.4849475001752,16,0.04514051167338893,1248,24.33132799998764,0.05930160085835996,32.015625,8.0,8.25,16.0
composite_fused,A4,4,2,32768,549.3539675010354,16,0.04813782290440163,2528,52.13855999995954,0.059670088757369746,64.015625,8.0,8.25,32.0
composite_fused,A4,4,2,65536,1095.4730075019588,16,0.04963405545386165,5088,107.7530240015909,0.05983534012350623,128.015625,8.0,8.25,64.0
composite_fused,B,8,1,8192,144.10828500005923,8,0.08660620726926088,600,12.16566399999219,0.11380331116974475,16.0078125,8.0,0.0,16.0
composite_fused,B,8,1,16384,285.6728750004253,8,0.09256988084807763,1240,26.069279999986758,0.11476063311909189,32.0078125,8.0,0.0,32.0
composite_fused,B,8,1,32768,568.9925550010347,8,0.09555971782205692,2520,53.8765119999582,0.1152071313127828,64.0078125,8.0,0.0,64.0
composite_fused,B,8,1,65536,1135.6319150019874,8,0.09706392938508733,5080,109.49097600161657,0.11543176822374755,128.0078125,8.0,0.0,128.0
composite_fused,A1,1,8,8192,2.2471645000005376,64,0.11665545624270428,656,0.0,0.457688,16.0625,8.0,57.75,2.0
composite_fused,A1,1,8,16384,4.090972500001895,64,0.4907507933676371,1296,13.903615999992937,0.501714,32.0625,8.0,57.75,4.0
composite_fused,A1,1,8,32768,6.65601250000007,64,0.8261186408347434,2576,41.71084799996763,0.61606,64.0625,8.0,57.75,8.0
composite_fused,A1,1,8,65536,11.786092499984429,64,1.05893212704674,5136,97.32531200143696,0.695438,128.0625,8.0,57.75,16.0
composite_fused,A2,2,4,8192,3.8395887500001407,32,0.5228809986460942,624,6.951807999995537,0.534693,16.03125,8.0,24.75,4.0
composite_fused,A2,2,4,16384,6.404628750001605,32,0.858544064722257,1264,20.855423999989405,0.640318,32.03125,8.0,24.75,8.0
composite_fused,A2,2,4,32768,11.534708749999874,32,1.0820101547616419,2544,48.66265599996224,0.710638,64.03125,8.0,24.75,16.0
composite_fused,A2,2,4,65536,21.794868749984772,32,1.21334541192213,5104,104.27712000153959,0.751966,128.03125,8.0,24.75,32.0
composite_fused,A4,4,2,8192,5.915787499999919,16,0.9294884239792724,608,10.427711999993306,0.693399,16.015625,8.0,8.25,8.0
composite_fused,A4,4,2,16384,11.045867500001041,16,1.1298951395303998,1248,24.33132799998764,0.742178,32.015625,8.0,8.25,16.0
composite_fused,A4,4,2,32768,21.306027499999384,16,1.2411841672219757,2528,52.13855999995954,0.769266,64.015625,8.0,8.25,32.0
composite_fused,A4,4,2,65536,41.82634749998455,16,1.2999645260103314,5088,107.7530240015909,0.783573,128.015625,8.0,8.25,64.0
composite_fused,B,8,1,8192,10.856904999999854,8,1.149560763397397,600,12.16566399999219,0.75528,16.0078125,8.0,0.0,16.0
composite_fused,B,8,1,16384,21.117065000001226,8,1.2522906947682175,1240,26.069279999986758,0.776244,32.0078125,8.0,0.0,32.0
composite_fused,B,8,1,32768,41.637384999999774,8,1.3058641410537761,2520,53.8765119999582,0.787177,64.0078125,8.0,0.0,64.0
composite_fused,B,8,1,65536,82.67802499997313,8,1.33323087973183,5080,109.49097600161657,0.792762,128.0078125,8.0,0.0,128.0
1 variant mode kv_per_cube C S_kv wall_us n_pe gemm_util math_count math_pipeline_us hbm_bw_util hbm_read_mb hbm_write_kb ipcq_kb kv_cache_per_cube_mb
2 composite_fused A1 1 8 8192 146.95153450000072 2.2471645000005376 64 0.0017838806576053131 0.11665545624270428 656 0.0 0.013997812319543966 0.457688 16.0625 8.0 57.75 2.0
3 composite_fused A1 1 8 16384 279.867342500003 4.090972500001895 64 0.007173570099630665 0.4907507933676371 1296 13.903615999992937 0.0146676634841736 0.501714 32.0625 8.0 57.75 4.0
4 composite_fused A1 1 8 32768 544.5763825000042 6.65601250000007 64 0.010097125355742134 0.8261186408347434 2576 41.71084799996763 0.015059411798858053 0.61606 64.0625 8.0 57.75 8.0
5 composite_fused A1 1 8 65536 1074.101862500392 11.786092499984429 64 0.011619635377527878 1.05893212704674 5136 97.32531200143696 0.015262053416273654 0.695438 128.0625 8.0 57.75 16.0
6 composite_fused A2 2 4 8192 140.35490875000028 3.8395887500001407 32 0.01430408111743636 0.5228809986460942 624 6.951807999995537 0.029254409671653128 0.534693 16.03125 8.0 24.75 4.0
7 composite_fused A2 2 4 16384 273.9919487500021 6.404628750001605 32 0.020068677291975947 0.858544064722257 1264 20.855423999989405 0.029935186188568382 0.640318 32.03125 8.0 24.75 8.0
8 composite_fused A2 2 4 32768 541.3734287504049 11.534708749999874 32 0.0230537210304644 1.0820101547616419 2544 48.66265599996224 0.030282239817052973 0.710638 64.03125 8.0 24.75 16.0
9 composite_fused A2 2 4 65536 1076.851308751879 21.794868749984772 32 0.024557433125924955 1.21334541192213 5104 104.27712000153959 0.030438742780552714 0.751966 128.03125 8.0 24.75 32.0
10 composite_fused A4 4 2 8192 140.2291675000001 5.915787499999919 16 0.03921192786066565 0.9294884239792724 608 10.427711999993306 0.058504233792873325 0.693399 16.015625 8.0 8.25 8.0
11 composite_fused A4 4 2 16384 276.4849475001752 11.045867500001041 16 0.04514051167338893 1.1298951395303998 1248 24.33132799998764 0.05930160085835996 0.742178 32.015625 8.0 8.25 16.0
12 composite_fused A4 4 2 32768 549.3539675010354 21.306027499999384 16 0.04813782290440163 1.2411841672219757 2528 52.13855999995954 0.059670088757369746 0.769266 64.015625 8.0 8.25 32.0
13 composite_fused A4 4 2 65536 1095.4730075019588 41.82634749998455 16 0.04963405545386165 1.2999645260103314 5088 107.7530240015909 0.05983534012350623 0.783573 128.015625 8.0 8.25 64.0
14 composite_fused B 8 1 8192 144.10828500005923 10.856904999999854 8 0.08660620726926088 1.149560763397397 600 12.16566399999219 0.11380331116974475 0.75528 16.0078125 8.0 0.0 16.0
15 composite_fused B 8 1 16384 285.6728750004253 21.117065000001226 8 0.09256988084807763 1.2522906947682175 1240 26.069279999986758 0.11476063311909189 0.776244 32.0078125 8.0 0.0 32.0
16 composite_fused B 8 1 32768 568.9925550010347 41.637384999999774 8 0.09555971782205692 1.3058641410537761 2520 53.8765119999582 0.1152071313127828 0.787177 64.0078125 8.0 0.0 64.0
17 composite_fused B 8 1 65536 1135.6319150019874 82.67802499997313 8 0.09706392938508733 1.33323087973183 5080 109.49097600161657 0.11543176822374755 0.792762 128.0078125 8.0 0.0 128.0