ywkang
711a9a257f
Metric fixes (test harnesses, deploy artifact + wrong constant):
- wall = max(t_end) - min(t_start): exclude the one-time KV-cache deploy
from the measured decode/prefill step (it was 92-99% of wall, mapping-
invariant, and masked the real per-mapping separation).
- PEAK_PE_HBM_BPS 128 -> 256 B/ns (8 channels/PE x 32 GB/s); the old value
was half the modeled per-PE HBM BW, so hbm_bw_util read >1.0 once wall
was corrected. All six short-context sweep CSVs regenerated/repatched.
Result: the four KV mappings now separate along a {64,32,16,8}-active-PE
ladder (decode 8-kv/1-kv = 4.8x at 8K, 7.4x at 64K), not "modest/tied" as
before; decode is bandwidth-bound at 46-76% of the 256 GB/s per-PE ceiling.
Report (S5.2 rewrite):
- Replace the tied-wall / 8x-per-PE-util claims (both deploy artifacts)
with the corrected separation and a density trade-off (per-CUBE KV
footprint, Fig 16 top panel switched to a wall-invariant metric).
- Add a projected latency-vs-batched-throughput analysis (marked
projected, not measured): dense mappings win throughput, 1-kv wins
latency; converges at long context.
- Regenerate Fig 15/16/17/18; fix plot script hardcoded ROOT path.
Batch experiment (Part 2, cube_base):
- Add backward-compatible cube_base=0 scalar to the decode kernel so a
batched user placed at DPPolicy.cube_start addresses 0-based shards.
Default preserves single-user behavior (32 decode tests pass unchanged).
- New batch harness (skipped): concurrent B>=2 launches hit a sim routing
issue (sip0.cube0.pe8); single-user path verified. Concurrency fix next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2.1 KiB
2.1 KiB
| 1 | variant | mode | kv_per_cube | C | S_kv | wall_us | n_pe | gemm_util | math_count | math_pipeline_us | hbm_bw_util | hbm_read_mb | hbm_write_kb | ipcq_kb | kv_cache_per_cube_mb |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | composite_fused | A1 | 1 | 8 | 8192 | 2.2471645000005376 | 64 | 0.11665545624270428 | 656 | 0.0 | 0.457688 | 16.0625 | 8.0 | 57.75 | 2.0 |
| 3 | composite_fused | A1 | 1 | 8 | 16384 | 4.090972500001895 | 64 | 0.4907507933676371 | 1296 | 13.903615999992937 | 0.501714 | 32.0625 | 8.0 | 57.75 | 4.0 |
| 4 | composite_fused | A1 | 1 | 8 | 32768 | 6.65601250000007 | 64 | 0.8261186408347434 | 2576 | 41.71084799996763 | 0.61606 | 64.0625 | 8.0 | 57.75 | 8.0 |
| 5 | composite_fused | A1 | 1 | 8 | 65536 | 11.786092499984429 | 64 | 1.05893212704674 | 5136 | 97.32531200143696 | 0.695438 | 128.0625 | 8.0 | 57.75 | 16.0 |
| 6 | composite_fused | A2 | 2 | 4 | 8192 | 3.8395887500001407 | 32 | 0.5228809986460942 | 624 | 6.951807999995537 | 0.534693 | 16.03125 | 8.0 | 24.75 | 4.0 |
| 7 | composite_fused | A2 | 2 | 4 | 16384 | 6.404628750001605 | 32 | 0.858544064722257 | 1264 | 20.855423999989405 | 0.640318 | 32.03125 | 8.0 | 24.75 | 8.0 |
| 8 | composite_fused | A2 | 2 | 4 | 32768 | 11.534708749999874 | 32 | 1.0820101547616419 | 2544 | 48.66265599996224 | 0.710638 | 64.03125 | 8.0 | 24.75 | 16.0 |
| 9 | composite_fused | A2 | 2 | 4 | 65536 | 21.794868749984772 | 32 | 1.21334541192213 | 5104 | 104.27712000153959 | 0.751966 | 128.03125 | 8.0 | 24.75 | 32.0 |
| 10 | composite_fused | A4 | 4 | 2 | 8192 | 5.915787499999919 | 16 | 0.9294884239792724 | 608 | 10.427711999993306 | 0.693399 | 16.015625 | 8.0 | 8.25 | 8.0 |
| 11 | composite_fused | A4 | 4 | 2 | 16384 | 11.045867500001041 | 16 | 1.1298951395303998 | 1248 | 24.33132799998764 | 0.742178 | 32.015625 | 8.0 | 8.25 | 16.0 |
| 12 | composite_fused | A4 | 4 | 2 | 32768 | 21.306027499999384 | 16 | 1.2411841672219757 | 2528 | 52.13855999995954 | 0.769266 | 64.015625 | 8.0 | 8.25 | 32.0 |
| 13 | composite_fused | A4 | 4 | 2 | 65536 | 41.82634749998455 | 16 | 1.2999645260103314 | 5088 | 107.7530240015909 | 0.783573 | 128.015625 | 8.0 | 8.25 | 64.0 |
| 14 | composite_fused | B | 8 | 1 | 8192 | 10.856904999999854 | 8 | 1.149560763397397 | 600 | 12.16566399999219 | 0.75528 | 16.0078125 | 8.0 | 0.0 | 16.0 |
| 15 | composite_fused | B | 8 | 1 | 16384 | 21.117065000001226 | 8 | 1.2522906947682175 | 1240 | 26.069279999986758 | 0.776244 | 32.0078125 | 8.0 | 0.0 | 32.0 |
| 16 | composite_fused | B | 8 | 1 | 32768 | 41.637384999999774 | 8 | 1.3058641410537761 | 2520 | 53.8765119999582 | 0.787177 | 64.0078125 | 8.0 | 0.0 | 64.0 |
| 17 | composite_fused | B | 8 | 1 | 65536 | 82.67802499997313 | 8 | 1.33323087973183 | 5080 | 109.49097600161657 | 0.792762 | 128.0078125 | 8.0 | 0.0 | 128.0 |