Three coupled fixes that recover small-tile GEMM pipeline efficiency
from 53% to 88% (32x3072x32 load_ref, composite_window basis).
1. PE_DMA channel-hold (ADR-0014 D4 clarified): both the
_handle_with_hooks (PeInternalTxn) and _pipeline_process
(TileToken) paths used to hold the cap=1 DMA channel through the
full HBM round-trip, which double-serialized with the HBM_CTRL's
own per-PC `available_at` model and prevented back-to-back tile
DMAs from amortizing their per-request head latency. Channel is
now released after the request is enqueued onto the next hop;
HBM serialization is HBM_CTRL's responsibility alone.
Tests: new test_pe_dma_back_to_back_pipelining as the oracle
(asserts wall < 75% of strict-serialized N x single_op). Existing
test_pe_dma_record_start_after_channel_acquire rewritten to assert
t_start clustering (channel released fast) instead of the old
round-trip-hold invariant. test_pe_dma_same_channel_serializes
still passes — HBM_CTRL preserves ordering.
Probe regression: PE→local-HBM 32 KiB stays at 141 ns
(single-request, unaffected).
2. milestone_1h_gemm bench: matmul_composite was reading MATMUL_M/K/N
env vars at module load, so every sweep row replayed the cached
256³ result; values now read inside run(). Drops the stale
sys.modules deletion hack.
3. Analytic ideal-pipeline model: dropped the (n_mn-1)·dma_w_per_pair
penalty (over-pessimistic for under-tile shapes — it pushed
measured > theoretical) and replaced the D_STAGES-derived head
with empirical T_PIPELINE_FILL=60 ns / T_PIPELINE_TAIL=30 ns.
Max analytic-vs-measured gap across all 7 swept shapes now 2.2 ppt
(was 9-44 ppt under the old constants).
Paper updates:
- §3 (GEMM): 78%→88% measured at 48 tiles, 23%→15% at 1 tile,
stage breakdown numbers refreshed (DMA in / Fetch / GEMM all
~785 ns at K=3072), analytic-vs-measured agreement tightened
to "within 2.2 ppt".
- §2.4 (Accuracy): GEMM tracking claim refreshed accordingly.
- §5 (GQA): restore long-ctx 4-cases figures into figures/
(they were dropped from bench output dir as derived artifacts in
92b9221 / e45626c but §5 still cites them by name).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Probe table (Table 1, tab:probe-pe-dma) promoted to table* so it
spans both columns and no longer gets clipped at the column edge.
- Fig 1 (hw-arch) row 1 (SIP, CUBE subfigures) trimmed: widths
0.495 -> 0.42 textwidth and a height=0.6\linewidth cap added so
the figure occupies less vertical space and floats can land
earlier in the document.
- Fig 1 and Fig 2 source blocks relocated to the top of §2 so they
enter LaTeX's float queue before any §2 body text. As a result
Fig 1 now lands on page 3 and Fig 2 on page 4 (was 4 and 5).
- Table 2 (tab:hw, modeled hardware config) reverted to plain
\begin{table}; the previous \begin{table*} reshape was not
requested.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- subfig widths 0.46 -> 0.495 textwidth so the \hfill gap between
(a) SIP and (b) CUBE shrinks to ~2% textwidth instead of ~8%.
- Removed the height=0.85\linewidth cap on both subfigures; with the
cap, the (roughly 1:1) SIP and CUBE diagrams were being scaled
down to 85% even when the column had room. Let them render at
their natural aspect ratio.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§2 platform:
- Accuracy promoted from \paragraph to \subsection (sec:accuracy);
it sits at the same heading level as Why / Device / Latency / HW
config and so reads as a first-class component of the platform
description rather than a tail-end footnote.
- Three architecture diagrams (SIP, CUBE, PE) collapsed into one
figure* using subcaption: Fig.~\ref{fig:hw-arch}(a) SIP and (b)
CUBE side-by-side on row 1, (c) PE wide on row 2, total height
capped at ~40% of the page. Subfigure cross-references rewritten
to Fig.~ref{fig:hw-arch}\subref{...} in the body text. main.tex
now pulls in the subcaption package.
- Table 1 (modeled hardware configuration) promoted to table*
(two-column / full page width) and the row layout rewritten as a
4-column tabular so Hierarchy + PE + Command-issue sit on the
left and Memory + Interconnect sit on the right. The previous
single-column rendering was getting cut at the right edge of the
printed column.
§6 GQA:
- Removed the historical four-panel headline table (tab:gqa), the
two figures (fig:gqa-lat, fig:gqa-break), and the prose paragraph
that cited their per-panel numbers. Reason: the underlying
milestone-gqa-headline bench has been simplified in collaborator
commit 65c365f ("drop misleading single_user_/multi_user_
panels") and no longer reproduces that data, so the section was
left referencing a dataset the current bench cannot regenerate.
- The previously-added 4-cases long-context decode comparison is
now §6's only Results subsection, retitled to
"Results: long-context decode and parallelism strategies".
The headline GQA result that survives into the paper is therefore
the parallelism trade-off study (Case 4 / Cube-SP × PE-SP at 34 us
buying an 8x KV-memory reduction over the latency leaders).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§2 platform:
- Add a "memory-centric AHBM" sentence to the device-and-execution
intro: each PE is paired with dedicated HBM bandwidth and on-PE
TCM, so the performance question is about feeding compute from
locally-attached memory + moving the unavoidable inter-PE traffic
efficiently. Makes the AHBM character of the platform visible
before §2.3 starts unpacking the latency model.
- Promote "Congestion and contention modeling" from \paragraph to
\subsubsection: this is the platform's core differentiator over a
peak-BW roofline, so it deserves its own heading.
- Rename "Control-plane (issue) cost model" to "Command dispatch
overhead model" -- describes what it actually charges (the PE_CPU
paying a fixed + per-byte cost to push a command to one of the
accelerator engines) without the more abstract "control-plane"
framing.
- Add a third Accuracy cross-check from kernbench probe: a
PE→HBM DMA-distance sweep that confirms monotonic hop progression,
bandwidth saturation matching the per-edge model, and the built-in
invariants (D2H >= H2D, cross-CUBE best < worst). New
Table~\ref{tab:probe-pe-dma} reports per-traversal latency and
utilisation at 32KiB / 1MiB across five hop classes.
- Captured probe output as figures/probe_pe_dma_summary.txt for
reproducibility.
§5 PE_IPCQ / all-reduce:
- Re-ran milestone-1h-ccl on current sim_engine (post-ADR-0064 Rev2
and the IPCQ slot-wrap Phase-2 race fix). Updated the topology-
comparison table and the buffer-kind caption to the fresh raw
latencies. Headline ratios are preserved:
torus vs mesh saving at 96 KB/PE: 24.8% (was 24.8%) -> ~25%
torus vs ring saving at 96 KB/PE: 19.3% (was 19.3%) -> ~19%
TCM vs HBM saving at 64 KB/PE: 13.4% (was 13.9%) -> ~13%
TCM vs SRAM saving at 64 KB/PE: 37.2% (was 38.3%) -> ~37%
The Executive Summary's "up to ~14%" / "up to ~38%" framing stays
consistent with these post values.
Artifacts refreshed in src/kernbench/benches/1H_milestone_output/:
- ccl/summary.csv + per-topology PNGs + buffer-kind CSV/PNG
- ccl/comparison_mesh_vs_ring_vs_2DTorus_vs_theoretical_vs_fsim.png
- gemm/* (re-run yields identical pe_window structure; PNGs
refreshed)
Paper figures synced to the fresh artifacts:
- figures/allreduce_comparison.png
- figures/allreduce_buffer_kind.png
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Reordered §2 subsections to follow the SIP → CUBE → PE → graph
flow: Why KernBench → Device and execution model → Latency model
→ Modeled hardware configuration. Readers now meet the device
hierarchy before the graph abstraction that re-uses it.
- §2.2 Device and execution model: starts with the SIP/CUBE/PE
hierarchy paragraphs (each anchoring fig:sip-arch, fig:cube-arch,
fig:pe-arch); then the runtime-API/sim-engine/components bullet
list; then the atomic-vs-composite command distinction (corrects
the prior over-narrow framing that read every PE command as
composite -- atomic single-engine commands exist too, and PE_CPU
itself runs control-plane work directly).
- §2.3 Latency model: opens with the four-contribution decomposition
(per-node overhead, per-edge transmission, drain, queuing delay)
and the latency_model schematic; retains existing The hardware as
a graph / From graph to DES / Latency contributions / Congestion
/ Control-plane cost model / Accuracy paragraphs. Accuracy
paragraph now closes on KernBench's sufficiency for *relative*
HW/SW design trade-offs given analytic + external-simulator
agreement.
- New figures and assets:
- figures/sip_architecture.pdf (SIP-level graph view)
- figures/cube_architecture.pdf (CUBE-level zoom-in)
- figures/latency_model.png (conceptual latency-model
schematic with per-node /
per-edge / drain / queuing-delay
colour coding)
- figures/pe_architecture.png (carried over)
- Source-of-truth generator for the latency schematic:
scripts/paper/paper_latency_model_diagram.py (a report-only
harness under scripts/paper/ per the /paper isolation rule).
- main.tex preamble: \usepackage{tikz} added (kept from prior
sequence-diagram draft -- harmless now that the latency model is
a PNG; left in to keep paragraph numbering stable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mid-pass through §2. Captured progress so far:
- §2 intro previews the three new threads (graph view, DES engine,
congestion) the rewritten latency section weaves together.
- §2.3 retitled to "Latency model: a graph traversed by events" and
restructured into bold-led paragraphs:
- The hardware as a graph (nodes = components, edges = links).
- From graph to discrete-event simulation (node/edge events,
deterministic ordering, correlation-ID trace).
- Latency contributions (per-node fixed, per-edge size-aware,
per-service occupancy).
- Congestion: per-edge FIFO BW occupancy, HBM per-PC parallelism,
component serial workers — the mechanisms that surface real
bottlenecks instead of peak-BW roofline.
- Control-plane cost model (FIXED + b·R) — unchanged.
- Accuracy: extended with a second cross-check from the all-reduce
study (torus vs. analytic startup-plus-per-packet model + FSIM
external single-device reference). Existing GEMM analytic-vs-
measured 10-20% check retained.
- §2.4 Hardware-configuration table unchanged.
Still TODO: cross-check accuracy numbers against a fresh
milestone-1h-gemm/ccl re-run; the current artifacts predate the
ADR-0064 Rev2 cost-model and IPCQ Phase-2 race fix and may need a
refresh before §2.3 final lock-in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New `/paper` slash-command skill that synthesizes ADR/SPEC content and live
KernBench benchmark results into a sectioned LaTeX technical paper compiled
to PDF with Tectonic (auto-installed). The skill negotiates a TOC, grounds
every number in committed artifacts or fresh bench runs, and keeps
report-only benches isolated.
This commit also includes the first generated report:
- docs/report/1H-codesign-paper/ — main.tex + per-section .tex, figures,
toc.md contract, and the built 8-page main.pdf. Covers the platform
(source-level kernels, latency model + accuracy, HW config from
topology.yaml), GEMM via composite command, All-Reduce via PE_IPCQ, and
fused GQA combining both, plus discussion/conclusion/2H future work.
- scripts/paper/ — isolated report harnesses (not registered benches):
paper_gqa_latency.py harvests per-panel GQA end-to-end latency + engine
occupancy (the milestone only emitted op-counts); paper_plot_gqa.py
renders the GQA figures.
GEMM/All-Reduce reuse committed milestone figures/CSVs; GQA results are
generated fresh. Honest flags retained: PE_CPU dispatch cost is 0 in this
config, and the proposed two-composite softmax_merge decode is marked
designed-not-measured.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ywkang