ywkang
711a9a257f
Metric fixes (test harnesses, deploy artifact + wrong constant):
- wall = max(t_end) - min(t_start): exclude the one-time KV-cache deploy
from the measured decode/prefill step (it was 92-99% of wall, mapping-
invariant, and masked the real per-mapping separation).
- PEAK_PE_HBM_BPS 128 -> 256 B/ns (8 channels/PE x 32 GB/s); the old value
was half the modeled per-PE HBM BW, so hbm_bw_util read >1.0 once wall
was corrected. All six short-context sweep CSVs regenerated/repatched.
Result: the four KV mappings now separate along a {64,32,16,8}-active-PE
ladder (decode 8-kv/1-kv = 4.8x at 8K, 7.4x at 64K), not "modest/tied" as
before; decode is bandwidth-bound at 46-76% of the 256 GB/s per-PE ceiling.
Report (S5.2 rewrite):
- Replace the tied-wall / 8x-per-PE-util claims (both deploy artifacts)
with the corrected separation and a density trade-off (per-CUBE KV
footprint, Fig 16 top panel switched to a wall-invariant metric).
- Add a projected latency-vs-batched-throughput analysis (marked
projected, not measured): dense mappings win throughput, 1-kv wins
latency; converges at long context.
- Regenerate Fig 15/16/17/18; fix plot script hardcoded ROOT path.
Batch experiment (Part 2, cube_base):
- Add backward-compatible cube_base=0 scalar to the decode kernel so a
batched user placed at DPPolicy.cube_start addresses 0-based shards.
Default preserves single-user behavior (32 decode tests pass unchanged).
- New batch harness (skipped): concurrent B>=2 launches hit a sim routing
issue (sip0.cube0.pe8); single-user path verified. Concurrency fix next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1.8 KiB
1.8 KiB
| 1 | variant | mode | kv_per_cube | C | S_kv | wall_us | n_pe | gemm_util | math_count | math_pipeline_us | hbm_bw_util | hbm_read_mb | hbm_write_kb | ipcq_kb | kv_cache_per_cube_mb |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | composite | A1 | 1 | 8 | 8192 | 35.312546999435405 | 64 | 0.19162004939598534 | 5248 | 0.0 | 0.0292247 | 16.0625 | 64.0 | 114688.0 | 2.0 |
| 3 | composite | A1 | 1 | 8 | 16384 | 62.80411199463298 | 64 | 0.21548245124172694 | 10880 | 0.0 | 0.0327367 | 32.0625 | 64.0 | 229376.0 | 4.0 |
| 4 | composite | A1 | 1 | 8 | 32768 | 117.78396202135924 | 64 | 0.2297967187462748 | 22144 | 0.0 | 0.0348435 | 64.0625 | 64.0 | 458752.0 | 8.0 |
| 5 | composite | A1 | 1 | 8 | 65536 | 227.7376320068799 | 64 | 0.23769780826259174 | 44672 | 0.0 | 0.0360063 | 128.0625 | 64.0 | 917504.0 | 16.0 |
| 6 | composite | A2 | 2 | 4 | 8192 | 35.873466499721395 | 32 | 0.21785371648964683 | 2624 | 0.0 | 0.0575356 | 16.0625 | 64.0 | 49152.0 | 4.0 |
| 7 | composite | A2 | 2 | 4 | 16384 | 67.67155399355386 | 32 | 0.23097350479268527 | 5440 | 0.0 | 0.0607641 | 32.0625 | 64.0 | 98304.0 | 8.0 |
| 8 | composite | A2 | 2 | 4 | 32768 | 129.35661752446276 | 32 | 0.24166271963282046 | 11072 | 0.0 | 0.0634525 | 64.0625 | 64.0 | 196608.0 | 16.0 |
| 9 | composite | A2 | 2 | 4 | 65536 | 254.34501247150266 | 32 | 0.24581313146006306 | 22336 | 0.0 | 0.0644793 | 128.0625 | 64.0 | 393216.0 | 32.0 |
| 10 | composite | A4 | 4 | 2 | 8192 | 46.88723350110906 | 16 | 0.2114076532174909 | 1312 | 0.0 | 0.088041 | 16.0625 | 64.0 | 16384.0 | 8.0 |
| 11 | composite | A4 | 4 | 2 | 16384 | 89.10037348339473 | 16 | 0.22249783278807841 | 2720 | 0.0 | 0.0923004 | 32.0625 | 64.0 | 32768.0 | 16.0 |
| 12 | composite | A4 | 4 | 2 | 32768 | 173.47797354247328 | 16 | 0.22855512537871095 | 5536 | 0.0 | 0.0946287 | 64.0625 | 64.0 | 65536.0 | 32.0 |
| 13 | composite | A4 | 4 | 2 | 65536 | 342.2399932809714 | 16 | 0.23170453934016594 | 11168 | 0.0 | 0.0958392 | 128.0625 | 64.0 | 131072.0 | 64.0 |
| 14 | composite | B | 8 | 1 | 8192 | 68.73804599999497 | 8 | 0.3341725483628459 | 656 | 0.0 | 0.120108 | 16.0625 | 64.0 | 0.0 | 16.0 |
| 15 | composite | B | 8 | 1 | 16384 | 134.27404599999218 | 8 | 0.3421415930417734 | 1360 | 0.0 | 0.122496 | 32.0625 | 64.0 | 0.0 | 32.0 |
| 16 | composite | B | 8 | 1 | 32768 | 265.34604599999824 | 8 | 0.3462703641501576 | 2768 | 0.0 | 0.123733 | 64.0625 | 64.0 | 0.0 | 64.0 |
| 17 | composite | B | 8 | 1 | 65536 | 527.4900460000299 | 8 | 0.3483723443539211 | 5584 | 0.0 | 0.124363 | 128.0625 | 64.0 | 0.0 | 128.0 |