ywkang
711a9a257f
Metric fixes (test harnesses, deploy artifact + wrong constant):
- wall = max(t_end) - min(t_start): exclude the one-time KV-cache deploy
from the measured decode/prefill step (it was 92-99% of wall, mapping-
invariant, and masked the real per-mapping separation).
- PEAK_PE_HBM_BPS 128 -> 256 B/ns (8 channels/PE x 32 GB/s); the old value
was half the modeled per-PE HBM BW, so hbm_bw_util read >1.0 once wall
was corrected. All six short-context sweep CSVs regenerated/repatched.
Result: the four KV mappings now separate along a {64,32,16,8}-active-PE
ladder (decode 8-kv/1-kv = 4.8x at 8K, 7.4x at 64K), not "modest/tied" as
before; decode is bandwidth-bound at 46-76% of the 256 GB/s per-PE ceiling.
Report (S5.2 rewrite):
- Replace the tied-wall / 8x-per-PE-util claims (both deploy artifacts)
with the corrected separation and a density trade-off (per-CUBE KV
footprint, Fig 16 top panel switched to a wall-invariant metric).
- Add a projected latency-vs-batched-throughput analysis (marked
projected, not measured): dense mappings win throughput, 1-kv wins
latency; converges at long context.
- Regenerate Fig 15/16/17/18; fix plot script hardcoded ROOT path.
Batch experiment (Part 2, cube_base):
- Add backward-compatible cube_base=0 scalar to the decode kernel so a
batched user placed at DPPolicy.cube_start addresses 0-based shards.
Default preserves single-user behavior (32 decode tests pass unchanged).
- New batch harness (skipped): concurrent B>=2 launches hit a sim routing
issue (sip0.cube0.pe8); single-user path verified. Concurrency fix next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1.6 KiB
1.6 KiB
| 1 | mode | kv_per_cube | C | S_kv | wall_us | n_pe | gemm_util | math_count | hbm_bw_util | hbm_read_mb | hbm_write_kb | ipcq_kb | kv_cache_per_cube_mb |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | A1 | 1 | 8 | 8192 | 2.2471645000005376 | 64 | 0.11665545624270428 | 656 | 0.457688 | 16.0625 | 8.0 | 57.75 | 2.0 |
| 3 | A1 | 1 | 8 | 16384 | 3.6068965000013704 | 64 | 0.14535709577464812 | 1360 | 0.569049 | 32.0625 | 8.0 | 57.75 | 4.0 |
| 4 | A1 | 1 | 8 | 32768 | 6.326360500001814 | 64 | 0.1657471147905734 | 2768 | 0.648161 | 64.0625 | 8.0 | 57.75 | 8.0 |
| 5 | A1 | 1 | 8 | 65536 | 11.765288499959512 | 64 | 0.17824909265982342 | 5584 | 0.696668 | 128.0625 | 8.0 | 57.75 | 16.0 |
| 6 | A2 | 2 | 4 | 8192 | 3.3555127500005475 | 32 | 0.15624676139283236 | 656 | 0.611829 | 16.03125 | 8.0 | 24.75 | 4.0 |
| 7 | A2 | 2 | 4 | 16384 | 6.074976750001311 | 32 | 0.17260576347056114 | 1360 | 0.675064 | 32.03125 | 8.0 | 24.75 | 8.0 |
| 8 | A2 | 2 | 4 | 32768 | 11.513904750002315 | 32 | 0.18214081543451005 | 2768 | 0.711922 | 64.03125 | 8.0 | 24.75 | 16.0 |
| 9 | A2 | 2 | 4 | 65536 | 22.391760749934242 | 32 | 0.18731461303283142 | 5584 | 0.731921 | 128.03125 | 8.0 | 24.75 | 32.0 |
| 10 | A4 | 4 | 2 | 8192 | 5.586135500000557 | 16 | 0.1877104484844272 | 656 | 0.734318 | 16.015625 | 8.0 | 8.25 | 8.0 |
| 11 | A4 | 4 | 2 | 16384 | 11.025063500001211 | 16 | 0.19021677290108474 | 1360 | 0.743578 | 32.015625 | 8.0 | 8.25 | 16.0 |
| 12 | A4 | 4 | 2 | 32768 | 21.902919500003218 | 16 | 0.19149520227204342 | 2768 | 0.748302 | 64.015625 | 8.0 | 8.25 | 32.0 |
| 13 | A4 | 4 | 2 | 65536 | 43.65863149988279 | 16 | 0.19214088284049563 | 5584 | 0.750688 | 128.015625 | 8.0 | 8.25 | 64.0 |
| 14 | B | 8 | 1 | 8192 | 10.816091000000947 | 8 | 0.1938918598225168 | 656 | 0.75813 | 16.0078125 | 8.0 | 0.0 | 16.0 |
| 15 | B | 8 | 1 | 16384 | 21.693947000002314 | 8 | 0.1933398288471476 | 1360 | 0.755602 | 32.0078125 | 8.0 | 0.0 | 32.0 |
| 16 | B | 8 | 1 | 32768 | 43.449659000006385 | 8 | 0.19306499045255035 | 2768 | 0.754344 | 64.0078125 | 8.0 | 0.0 | 64.0 |
| 17 | B | 8 | 1 | 65536 | 86.96108299976913 | 8 | 0.19292786406575965 | 5584 | 0.753716 | 128.0078125 | 8.0 | 0.0 | 128.0 |