ywkang
711a9a257f
Metric fixes (test harnesses, deploy artifact + wrong constant):
- wall = max(t_end) - min(t_start): exclude the one-time KV-cache deploy
from the measured decode/prefill step (it was 92-99% of wall, mapping-
invariant, and masked the real per-mapping separation).
- PEAK_PE_HBM_BPS 128 -> 256 B/ns (8 channels/PE x 32 GB/s); the old value
was half the modeled per-PE HBM BW, so hbm_bw_util read >1.0 once wall
was corrected. All six short-context sweep CSVs regenerated/repatched.
Result: the four KV mappings now separate along a {64,32,16,8}-active-PE
ladder (decode 8-kv/1-kv = 4.8x at 8K, 7.4x at 64K), not "modest/tied" as
before; decode is bandwidth-bound at 46-76% of the 256 GB/s per-PE ceiling.
Report (S5.2 rewrite):
- Replace the tied-wall / 8x-per-PE-util claims (both deploy artifacts)
with the corrected separation and a density trade-off (per-CUBE KV
footprint, Fig 16 top panel switched to a wall-invariant metric).
- Add a projected latency-vs-batched-throughput analysis (marked
projected, not measured): dense mappings win throughput, 1-kv wins
latency; converges at long context.
- Regenerate Fig 15/16/17/18; fix plot script hardcoded ROOT path.
Batch experiment (Part 2, cube_base):
- Add backward-compatible cube_base=0 scalar to the decode kernel so a
batched user placed at DPPolicy.cube_start addresses 0-based shards.
Default preserves single-user behavior (32 decode tests pass unchanged).
- New batch harness (skipped): concurrent B>=2 launches hit a sim routing
issue (sip0.cube0.pe8); single-user path verified. Concurrency fix next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2.1 KiB
2.1 KiB
| 1 | variant | mode | kv_per_cube | C | S_kv | wall_us | n_pe | gemm_util | math_count | math_pipeline_us | hbm_bw_util | hbm_read_mb | hbm_write_kb | ipcq_kb | kv_cache_per_cube_mb |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | composite_fused | A1 | 1 | 8 | 8192 | 35.53158299944201 | 64 | 0.3512557265116541 | 4800 | 97.32531200143696 | 0.0290446 | 16.0625 | 64.0 | 114688.0 | 2.0 |
| 3 | composite_fused | A1 | 1 | 8 | 16384 | 63.02314799463935 | 64 | 0.4196030322618738 | 9920 | 208.55424000307917 | 0.0326229 | 32.0625 | 64.0 | 229376.0 | 4.0 |
| 4 | composite_fused | A1 | 1 | 8 | 32768 | 118.00299802135117 | 64 | 0.4607744625054221 | 20160 | 431.01209600543973 | 0.0347788 | 64.0625 | 64.0 | 458752.0 | 8.0 |
| 5 | composite_fused | A1 | 1 | 8 | 65536 | 227.95666800683736 | 64 | 0.4835519703723695 | 40640 | 875.9278079059123 | 0.0359717 | 128.0625 | 64.0 | 917504.0 | 16.0 |
| 6 | composite_fused | A2 | 2 | 4 | 8192 | 35.27508350025257 | 32 | 0.5747630902258046 | 2400 | 96.40870400077105 | 0.0585116 | 16.0625 | 64.0 | 49152.0 | 4.0 |
| 7 | composite_fused | A2 | 2 | 4 | 16384 | 65.58160349091561 | 32 | 0.653336388872743 | 4960 | 206.59008000165224 | 0.0627005 | 32.0625 | 64.0 | 98304.0 | 8.0 |
| 8 | composite_fused | A2 | 2 | 4 | 32768 | 126.19361352364719 | 32 | 0.69726913692688 | 10080 | 426.9528320015669 | 0.0650429 | 64.0625 | 64.0 | 196608.0 | 16.0 |
| 9 | composite_fused | A2 | 2 | 4 | 65536 | 247.42072338639758 | 32 | 0.7205501525262498 | 20320 | 867.678335950613 | 0.0662839 | 128.0625 | 64.0 | 393216.0 | 32.0 |
| 10 | composite_fused | A4 | 4 | 2 | 8192 | 45.62825600103196 | 16 | 0.7872118539828873 | 1200 | 92.28038400042057 | 0.0904703 | 16.0625 | 64.0 | 16384.0 | 8.0 |
| 11 | composite_fused | A4 | 4 | 2 | 16384 | 87.81637548340065 | 16 | 0.8628360665552867 | 2480 | 197.74368000090124 | 0.09365 | 32.0625 | 64.0 | 32768.0 | 16.0 |
| 12 | composite_fused | A4 | 4 | 2 | 32768 | 172.19397554247547 | 16 | 0.9029073142513345 | 5040 | 408.6702720000148 | 0.0953343 | 64.0625 | 64.0 | 65536.0 | 32.0 |
| 13 | composite_fused | A4 | 4 | 2 | 65536 | 340.9491752808783 | 16 | 0.9235491703853554 | 10160 | 830.5234559737444 | 0.096202 | 128.0625 | 64.0 | 131072.0 | 64.0 |
| 14 | composite_fused | B | 8 | 1 | 8192 | 68.33786600000411 | 8 | 2.036048828365075 | 712 | 121.6138560003899 | 0.120811 | 16.0625 | 64.0 | 0.0 | 16.0 |
| 15 | composite_fused | B | 8 | 1 | 16384 | 133.87386600000133 | 8 | 2.209234414765302 | 1480 | 260.6011200008355 | 0.122862 | 32.0625 | 64.0 | 0.0 | 32.0 |
| 16 | composite_fused | B | 8 | 1 | 32768 | 264.9458659999771 | 8 | 2.2985744263323977 | 3016 | 538.575647996068 | 0.12392 | 64.0625 | 64.0 | 0.0 | 64.0 |
| 17 | composite_fused | B | 8 | 1 | 65536 | 527.0898659999762 | 8 | 2.3439567929361727 | 6088 | 1094.5247039788662 | 0.124457 | 128.0625 | 64.0 | 0.0 | 128.0 |