ywkang
e3f08972da
§2 platform:
- Add a "memory-centric AHBM" sentence to the device-and-execution
intro: each PE is paired with dedicated HBM bandwidth and on-PE
TCM, so the performance question is about feeding compute from
locally-attached memory + moving the unavoidable inter-PE traffic
efficiently. Makes the AHBM character of the platform visible
before §2.3 starts unpacking the latency model.
- Promote "Congestion and contention modeling" from \paragraph to
\subsubsection: this is the platform's core differentiator over a
peak-BW roofline, so it deserves its own heading.
- Rename "Control-plane (issue) cost model" to "Command dispatch
overhead model" -- describes what it actually charges (the PE_CPU
paying a fixed + per-byte cost to push a command to one of the
accelerator engines) without the more abstract "control-plane"
framing.
- Add a third Accuracy cross-check from kernbench probe: a
PE→HBM DMA-distance sweep that confirms monotonic hop progression,
bandwidth saturation matching the per-edge model, and the built-in
invariants (D2H >= H2D, cross-CUBE best < worst). New
Table~\ref{tab:probe-pe-dma} reports per-traversal latency and
utilisation at 32KiB / 1MiB across five hop classes.
- Captured probe output as figures/probe_pe_dma_summary.txt for
reproducibility.
§5 PE_IPCQ / all-reduce:
- Re-ran milestone-1h-ccl on current sim_engine (post-ADR-0064 Rev2
and the IPCQ slot-wrap Phase-2 race fix). Updated the topology-
comparison table and the buffer-kind caption to the fresh raw
latencies. Headline ratios are preserved:
torus vs mesh saving at 96 KB/PE: 24.8% (was 24.8%) -> ~25%
torus vs ring saving at 96 KB/PE: 19.3% (was 19.3%) -> ~19%
TCM vs HBM saving at 64 KB/PE: 13.4% (was 13.9%) -> ~13%
TCM vs SRAM saving at 64 KB/PE: 37.2% (was 38.3%) -> ~37%
The Executive Summary's "up to ~14%" / "up to ~38%" framing stays
consistent with these post values.
Artifacts refreshed in src/kernbench/benches/1H_milestone_output/:
- ccl/summary.csv + per-topology PNGs + buffer-kind CSV/PNG
- ccl/comparison_mesh_vs_ring_vs_2DTorus_vs_theoretical_vs_fsim.png
- gemm/* (re-run yields identical pe_window structure; PNGs
refreshed)
Paper figures synced to the fresh artifacts:
- figures/allreduce_comparison.png
- figures/allreduce_buffer_kind.png
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
58 lines
4.1 KiB
Plaintext
58 lines
4.1 KiB
Plaintext
|
|
=== H2D Write Latency (IO->HBM, data=32768B) ===
|
|
Case Target Hops Actual Ovhd Drain Wire Ovhd% Drain% Eff.BW BN.BW Util%
|
|
-------------------------------------------------------------------------------------------------------------------
|
|
h2d-1hop cube0.pe0 1 289.53 50.0 256.0 0.18 17.3% 88.4% 113.17 128.0 88.4%
|
|
h2d-2hop cube4.pe0 2 326.04 78.0 256.0 0.20 23.9% 78.5% 100.50 128.0 78.5%
|
|
h2d-3hop cube8.pe0 3 362.56 106.0 256.0 0.20 29.2% 70.6% 90.38 128.0 70.6%
|
|
h2d-4hop cube12.pe0 4 399.06 134.0 256.0 0.21 33.6% 64.1% 82.11 128.0 64.1%
|
|
[v] Monotonic increase: PASS
|
|
|
|
BW Saturation (Util% by data size):
|
|
Case 4KB 16KB 64KB 256KB 1MB
|
|
------------------------------------------------------------------
|
|
h2d-1hop 38.9% 71.8% 91.1% 97.6% 99.4%
|
|
h2d-2hop 29.0% 62.1% 86.8% 96.3% 99.1%
|
|
h2d-3hop 23.2% 54.7% 82.8% 95.1% 98.7%
|
|
h2d-4hop 19.3% 48.8% 79.2% 93.8% 98.4%
|
|
|
|
=== D2H Read Latency (HBM->IO, data=32768B) ===
|
|
Case Source Hops Actual Ovhd Drain Wire Ovhd% Drain% Eff.BW BN.BW Util%
|
|
-------------------------------------------------------------------------------------------------------------------
|
|
d2h-1hop cube0.pe0 1 571.20 23.0 256.0 0.04 4.0% 44.8% 57.37 128.0 44.8%
|
|
d2h-2hop cube4.pe0 2 635.72 51.0 256.0 0.05 8.0% 40.3% 51.54 128.0 40.3%
|
|
d2h-3hop cube8.pe0 3 700.24 79.0 256.0 0.06 11.3% 36.6% 46.80 128.0 36.6%
|
|
d2h-4hop cube12.pe0 4 764.76 107.0 256.0 0.07 14.0% 33.5% 42.85 128.0 33.5%
|
|
[v] Monotonic increase: PASS
|
|
|
|
BW Saturation (Util% by data size):
|
|
Case 4KB 16KB 64KB 256KB 1MB
|
|
------------------------------------------------------------------
|
|
d2h-1hop 9.2% 28.9% 61.9% 86.7% 96.3%
|
|
d2h-2hop 7.8% 25.2% 57.4% 84.4% 95.6%
|
|
d2h-3hop 6.7% 22.4% 53.5% 82.2% 94.9%
|
|
d2h-4hop 5.9% 20.1% 50.2% 80.1% 94.2%
|
|
[v] D2H >= H2D (reverse data path): PASS
|
|
|
|
=== PE DMA Latency (pe_dma -> router -> HBM, data=32768B) ===
|
|
Case Target Actual Ovhd Drain Wire Ovhd% Drain% Eff.BW BN.BW Util%
|
|
----------------------------------------------------------------------------------------------------------------------------
|
|
pe-local-hbm c0.pe0->c0.slice0 141.00 2.0 128.0 0.00 1.4% 90.8% 232.40 256.0 90.8%
|
|
pe-same-half-hbm c0.pe0->c0.slice1 147.87 4.0 128.0 0.03 2.7% 86.6% 221.60 256.0 86.6%
|
|
pe-cross-half-hbm c0.pe0->c0.slice4 161.17 10.0 128.0 0.09 6.2% 79.4% 203.31 256.0 79.4%
|
|
pe-cross-cube-hbm-best c0.pe0->c1.slice0 330.52 30.0 256.0 0.01 9.1% 77.5% 99.14 128.0 77.5%
|
|
pe-cross-cube-hbm-worst c0.pe0->c15.slice0 677.12 180.0 256.0 0.06 26.6% 37.8% 48.39 128.0 37.8%
|
|
* Local BN: 256.0 GB/s, Remote BN: 256.0 GB/s
|
|
[v] Cross-cube best < worst: PASS (330.52ns < 677.12ns)
|
|
|
|
BW Saturation (Util% by data size):
|
|
Case 4KB 16KB 64KB 256KB 1MB
|
|
------------------------------------------------------------------
|
|
pe-local-hbm 88.9% 97.0% 99.2% 99.8% 100.0%
|
|
pe-same-half-hbm 79.9% 94.1% 98.5% 99.6% 99.9%
|
|
pe-cross-half-hbm 61.3% 86.4% 96.2% 99.0% 99.8%
|
|
pe-cross-cube-hbm-best 51.6% 81.0% 94.5% 98.6% 99.6%
|
|
pe-cross-cube-hbm-worst 15.1% 41.6% 74.0% 91.9% 97.8%
|
|
|
|
============================================================
|