paper(1H): reflect D8 single-op cost model in §2/§3.4 + figure/diagram regen

Two strands bundled as the 1H-codesign-paper refresh unit:

(A) This session — single-op cost-model reflection (depends on 2d8271c):
  - §2 Table 2 (tab:hw): split "FIXED per command" into "FIXED per
    single-op command" (8 cycles) and "FIXED per composite command"
    (40 cycles); §2 dispatch-overhead prose updated to the two-class split.
  - §3.4 (sec:gemm-vs-async): rename paragraph headers + prose to
    async-full / async-tiled; "atomic" -> "single-op" throughout; reframe
    mechanism #3 from the old DMA-only fast-path to the single-op
    fast-path. Headline narrative now: even with EVERY single-op cmd
    (96 DMA + 48 dot + 47 add) charged the light 8-cycle FIXED, composite
    still wins ~2.8x at K=3072 purely on command-count structure (1 vs
    192 commands) -- down from the pre-D8 ~6.3x, and explicitly NOT a
    modelling artifact. Numbers refreshed from the regenerated sweep:
    async-full 3.83->3.91, async-tiled 1.14->~2.53, under-tile corner
    1.06->1.21, depth-2 vs depth-inf spread <1%. New figure wired in.
  - build/main.pdf rebuilt (tectonic); pdftotext-verified (no broken
    refs; Table 2 split, single-op terms, 2.8x/2.53/192-host-commands
    all present).

(B) Prior-session paper work riding along uncommitted: §4 all-reduce
    deep-edit, §5 GQA, §6 discussion trims; milestone_1h_ccl.py plot
    label "FSIM" -> "H2 2025 SW queue baseline"; regenerated diagrams
    under docs/diagrams/** and gemm output PNGs under
    1H_milestone_output/gemm/. (Composite-window gemm plots are
    unaffected by D8 — D8 only changes single-op dispatch FIXED, which
    the composite window excludes.)

All TODO items for the D8 single-op extension are now complete and
pushed across 3 commits (2d8271c cost-model+ADR+tests, 821bbf2 bench
harness, this paper refresh). Full regression green (826 passed, 1
skipped). No remaining work.

NOTE for review (carried from 2d8271c): ADR-0065's "2x CPU-offload win"
headline for GQA decode opt2 may want a refresh to the post-D8 ~1.87x.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/report/1H-codesign-paper/sections/04-allreduce.tex

@@ -1,46 +1,145 @@
 \section{PE\_IPCQ and Collective Communication}
 \label{sec:allreduce}
 
-\subsection{Why it is needed}
-
 Distributing a transformer across devices turns every tensor-parallel
 layer into a collective: partial results computed on different PEs, CUBEs,
-and SIPs must be summed and redistributed with an all-reduce. If that
-collective is handled by the host or by a generic DMA path, three problems
-appear. The reduction traffic competes with the kernel's own compute DMA
-on the same links, causing head-of-line blocking; there is no efficient
-peer-to-peer ring primitive, so data takes extra hops; and the ordering is
-hard to make deterministic. The hardware question is how to perform
-collectives \emph{on the device}, overlapped with compute and reproducible
-run-to-run.
+and SIPs must be summed and redistributed with an all-reduce. Underneath
+the algorithm this is fundamentally a PE-to-PE problem---many short
+messages flowing between neighbors as the reduction proceeds. The natural
+software realization is a per-direction ring buffer whose head and tail
+pointers the producer and consumer update atomically and poll, but our
+H2 2025 report measured this scheme end-to-end and found that the
+atomic-pointer traffic together with the consumer's polling loop dominate
+the per-message cost: the queue itself becomes the bottleneck well before
+the link runs out of bandwidth, and a pure-SW collective spends most of
+its time on metadata rather than on actually moving partials. A second,
+orthogonal problem is link sharing---if the collective rides the kernel's
+generic DMA path it competes with the GEMM's compute traffic on the same
+wires, so a large tile transfer head-of-line-blocks a pending reduction.
+This section asks how a dedicated hardware primitive can lift queue
+management off the software path entirely and, in the same design, stop
+collective traffic from stalling behind compute traffic, so the all-reduce
+runs overlapped with compute and at the interconnect's physical limit.
 
 \subsection{Design}
 
-KernBench models a dedicated per-PE collective engine, \textbf{PE\_IPCQ}
-(inter-PE communication queue). It is a control-plane block: it owns the
-ring-buffer address arithmetic, head/tail pointers, peer-pointer caches,
-backpressure, and the four-direction (N/S/E/W) neighbor map, with eight
-ring buffers per PE (four directions $\times$ \{tx, rx\}). Crucially,
-PE\_IPCQ does \emph{not} move data itself---it delegates the actual
-transfer to PE\_DMA, keeping a clean control/data split. To stop
-collective traffic from blocking compute, PE\_DMA is extended into a
-two-channel virtual-channel model: \texttt{vc\_compute} carries tile
-load/store for GEMM and vector math, \texttt{vc\_comm} carries IPCQ sends,
-each with an independent state machine. The same physical link is shared
-but progresses in chunks (\SI{256}{\byte}), so a large GEMM DMA does not
-lock the link end-to-end against a pending reduction. On top of this
-substrate the collective runs a hierarchical local-reduce / global
-all-reduce-broadcast schedule across whatever inter-device topology the
-configuration specifies.
+The proposed block is \textbf{PE\_IPCQ} (inter-PE communication queue),
+a small controller dropped into every PE next to PE\_DMA, PE\_GEMM,
+PE\_MATH, and PE\_TCM. It is a control-plane block---it holds no
+payload data---and consists of three pieces: a per-direction
+\emph{QPair register file} (\textasciitilde\SI{576}{\byte} of flip-flops
+covering up to eight directions), a combinational slot-address
+generator and backpressure comparator, and a credit injector/receiver
+wired to the NoC. Each QPair holds the local pointers
+(\texttt{my\_head}, \texttt{my\_tail}), shadowed views of the peer's
+(\texttt{peer\_head\_cache}, \texttt{peer\_tail\_cache}), the local and
+peer rx-buffer physical bases, ring depth and slot size (both
+power-of-two), and the peer's credit-target address. The ring data
+itself lives in a reserved slot region of TCM (or PE-local HBM, or
+cube-shared SRAM), addressed by the QPair registers; PE\_IPCQ never
+touches the bytes, only the pointers. The PE\_CPU sees the controller
+as an MMIO peripheral and the N/S/E/W direction labels as logical
+ports---one kernel image runs across a 1D ring, 2D mesh, or 2D torus
+because the topology only changes which peers the QPair registers
+point at.
+
+\emph{Initialization.} The host CCL backend brings the whole machine
+to a usable state before any kernel runs.
+\texttt{init\_process\_group(backend="ahbm")} loads \texttt{ccl.yaml},
+resolves the algorithm + topology + buffer\_kind + slot configuration,
+allocates an rx ring-buffer region on every participating PE so every
+rank now knows every other rank's \texttt{rx\_base\_pa}, and fans out
+an \texttt{IpcqInitMsg} that writes the QPair register file on each
+PE\_IPCQ over MMIO. The same fan-out wires the per-direction
+credit-return channel: each PE\_IPCQ records its peer's credit-target
+address and a back-pointer that the credit receiver will use to update
+the right QPair. After init, every register the runtime needs is
+preloaded; nothing in the kernel path allocates memory, walks a table,
+or talks to the host.
+
+\emph{Send.} When the kernel executes \texttt{tl.send(dir="E",
+src\_addr, nbytes)}, the PE\_CPU performs a single MMIO write into
+PE\_IPCQ describing the request (direction, source address/space,
+length, sender handle). The controller evaluates backpressure in one
+combinational compare---\texttt{(my\_head $-$ peer\_tail\_cache) $<$
+n\_slots}---and, on a hit, the slot-address generator returns
+\texttt{dst = peer\_rx\_base\_pa + (my\_head \% n\_slots) $\times$
+slot\_size} in one to two cycles. PE\_IPCQ then emits one
+\texttt{IpcqDmaToken} on its dedicated port to PE\_DMA's
+\texttt{vc\_comm} channel, carrying the data descriptor (\texttt{src,
+dst, nbytes}) plus a small piggyback header (\texttt{sender\_seq =
+my\_head, src\_coord, direction}). \texttt{my\_head} is incremented in
+the same cycle---a local flip-flop bump, not a cross-PE atomic---and
+\texttt{tl.send} returns fire-and-forget. If the backpressure compare
+fails, the controller stalls the CPU in either of two modes selected
+at init: \texttt{poll} (CPU re-reads a status CSR) or \texttt{sleep}
+(controller asserts a wake event when a credit arrives). Both modes
+are benchmarked in the results.
+
+\emph{Transport.} PE\_DMA is the only block that touches the fabric,
+and it has been extended for IPCQ in two ways. First, it now exposes
+two virtual channels---\texttt{vc\_compute} for GEMM/Math
+\texttt{TileToken}s and \texttt{vc\_comm} for \texttt{IpcqDmaToken}s
+---with independent state machines and a chunk-level
+(\SI{256}{\byte}) weighted round-robin arbiter on the shared physical
+link. A large GEMM tile DMA can no longer monopolize the link against
+a pending IPCQ send, enabling compute and communication to make
+progress independently as an architectural property of the design. Second, on the sender side PE\_DMA packs the piggyback
+header into the same flit train as the data, and on the receiver side
+it runs the I6 \emph{atomic terminal handler}: pay the per-flit
+bottleneck-BW drain, then, in one indivisible block, (i) write the
+payload into \texttt{MemoryStore} at \texttt{dst\_addr} (the
+receiver's ring slot) and (ii) forward an \texttt{IpcqMetaArrival}
+$\{\texttt{sender\_seq, dst\_addr}\}$ to the local PE\_IPCQ over a
+PE-internal wire. No yield, no PE\_CPU interaction, no cache-coherence
+round trip---the bytes land in TCM and the metadata reaches the
+controller in the same cycle.
+
+\emph{Receive and credit return.} PE\_IPCQ's Meta Extractor
+range-matches the incoming \texttt{dst\_addr} against each direction's
+$[\texttt{rx\_base}, \texttt{rx\_base} + \texttt{n\_slots} \times
+\texttt{slot\_size})$ window---unambiguous even when two directions
+share a peer, as in a 2-rank bidirectional ring---and updates
+\texttt{peer\_head\_cache[d] := max(prev, sender\_seq + 1)}, releasing
+any \texttt{tl.recv} blocked on direction $d$. When the kernel
+eventually consumes the slot, the controller increments
+\texttt{my\_tail} and emits a \SI{16}{\byte} credit packet on the
+dedicated fast-path channel preloaded at init; the latency is the
+real fabric path (per-node overhead + edge propagation + 16 B /
+bottleneck BW), so an in-cube credit returns faster than a cross-SIP
+credit and the model carries no magic constants. The sender's PE\_IPCQ
+absorbs the credit, advances \texttt{peer\_tail\_cache}, and
+de-asserts backpressure if it was stalled. The pointer-synchronization
+problem the software baseline solved with explicit atomic RMWs and
+polling loops has been split in two and dissolved into existing
+traffic: \emph{head} updates ride on the DMA payload itself, with the
+receiver's PE\_DMA performing the data + metadata write in a single
+atomic step, so the sender never blocks waiting for the receiver to
+see it; \emph{tail} updates ride on a dedicated 16 B credit on a
+side-channel, with no software in the loop. Send becomes a single
+MMIO write, receive becomes a flip-flop read, backpressure becomes
+one 64-bit subtract, and the per-message cost in IPCQ is dominated by
+the link traversal rather than by metadata bookkeeping---which is
+what the H2 2025 measurement showed the pure-software queue could
+not achieve. On top of this substrate the collective runs a
+hierarchical local-reduce / global all-reduce-broadcast schedule
+across whatever inter-device topology the configuration specifies,
+with the IPCQ ring buffer placed in on-PE TCM, PE-local HBM, or
+cube-shared SRAM---the third knob the results section sweeps.
 
 \subsection{Results}
 
-Following the milestone-evaluation convention, the collective sweep builds
-its own six-device (six-SIP, $2\times3$) configurations---distinct from
-the two-SIP default of Table~\ref{tab:hw}---and measures all-reduce
-latency as a function of payload size for three inter-device topologies:
-a 1D ring, a 2D mesh (no wrap), and a 2D torus. Table~\ref{tab:allreduce}
-and Figure~\ref{fig:allreduce-cmp} report the result.
+All measurements in this section run on the PE\_IPCQ substrate
+described above; the topology sweep is intended to characterize how
+effectively the proposed mechanism exposes the underlying interconnect's
+properties, not to compare PE\_IPCQ against an alternative
+communication primitive. Following the milestone-evaluation convention,
+the collective sweep builds its own six-device (six-SIP, $2\times3$)
+configurations---distinct from the two-SIP default of
+Table~\ref{tab:hw}---and measures all-reduce latency as a function of
+payload size for three inter-device topologies: a 1D ring, a 2D mesh
+(no wrap), and a 2D torus. Table~\ref{tab:allreduce} and
+Figure~\ref{fig:allreduce-cmp} report the result.
 
 \begin{table}[t]
 \centering
@@ -64,13 +163,19 @@ per-PE payload. Lower is better; the torus wins at every size.}
 \begin{figure}[t]
 \centering
 \includegraphics[width=\linewidth]{allreduce_comparison.png}
-\caption{All-reduce latency vs.\ per-PE payload for the three topologies,
-against the analytic torus model and an external full-system simulator
-(FSIM) reference. The measured torus tracks the analytic curve within a
-small constant factor; the FSIM single-device point
-(\SI{366}{\micro\second}) sits an order of magnitude above the
-KernBench algorithmic latency, illustrating the difference between an
-achievable-kernel number and a full end-to-end-stack number.}
+\caption{All-reduce latency vs.\ per-PE payload for the three PE\_IPCQ
+topologies, against the analytic torus model. The isolated point in
+the top panel (\SI{366}{\micro\second}) is the only data point
+available from the H2 2025 software-queue measurement campaign---an
+intra-device all-reduce across 16 CUBEs on a single device. A true
+6-device inter-SIP measurement under the same software queue was not
+collected, so this point in fact \emph{understates} the SW-queue cost
+for the multidevice configuration KernBench measures here; even so the
+proposed PE\_IPCQ inter-device curves sit roughly an order of
+magnitude below it, which is the headline SW-vs-HW comparison this
+section makes. The measured torus tracks the analytic curve within a
+small constant factor, the gap reflecting real link serialization that
+the analytic model idealizes away.}
 \label{fig:allreduce-cmp}
 \end{figure}