ywkang
b6315c3c90
Two strands bundled as the 1H-codesign-paper refresh unit: (A) This session — single-op cost-model reflection (depends on2d8271c): - §2 Table 2 (tab:hw): split "FIXED per command" into "FIXED per single-op command" (8 cycles) and "FIXED per composite command" (40 cycles); §2 dispatch-overhead prose updated to the two-class split. - §3.4 (sec:gemm-vs-async): rename paragraph headers + prose to async-full / async-tiled; "atomic" -> "single-op" throughout; reframe mechanism #3 from the old DMA-only fast-path to the single-op fast-path. Headline narrative now: even with EVERY single-op cmd (96 DMA + 48 dot + 47 add) charged the light 8-cycle FIXED, composite still wins ~2.8x at K=3072 purely on command-count structure (1 vs 192 commands) -- down from the pre-D8 ~6.3x, and explicitly NOT a modelling artifact. Numbers refreshed from the regenerated sweep: async-full 3.83->3.91, async-tiled 1.14->~2.53, under-tile corner 1.06->1.21, depth-2 vs depth-inf spread <1%. New figure wired in. - build/main.pdf rebuilt (tectonic); pdftotext-verified (no broken refs; Table 2 split, single-op terms, 2.8x/2.53/192-host-commands all present). (B) Prior-session paper work riding along uncommitted: §4 all-reduce deep-edit, §5 GQA, §6 discussion trims; milestone_1h_ccl.py plot label "FSIM" -> "H2 2025 SW queue baseline"; regenerated diagrams under docs/diagrams/** and gemm output PNGs under 1H_milestone_output/gemm/. (Composite-window gemm plots are unaffected by D8 — D8 only changes single-op dispatch FIXED, which the composite window excludes.) All TODO items for the D8 single-op extension are now complete and pushed across 3 commits (2d8271ccost-model+ADR+tests,821bbf2bench harness, this paper refresh). Full regression green (826 passed, 1 skipped). No remaining work. NOTE for review (carried from2d8271c): ADR-0065's "2x CPU-offload win" headline for GQA decode opt2 may want a refresh to the post-D8 ~1.87x. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
55 lines
3.0 KiB
TeX
55 lines
3.0 KiB
TeX
\section{Discussion: which hardware changes are meaningful}
|
|
\label{sec:discussion}
|
|
|
|
Read together, the three studies point to a consistent ranking of where
|
|
hardware investment pays off for attention-centric decoding.
|
|
|
|
\paragraph{The composite command is foundational, but indirectly.} Its
|
|
direct effect---driving compute-rich GEMMs to
|
|
\textasciitilde\SI{78}{\percent} of peak---matters most for the
|
|
compute-bound parts of a model (the large feed-forward and projection
|
|
matrices). For attention itself, its more important effect is
|
|
\emph{diagnostic}: by making issue and compute nearly free, it removes the
|
|
overhead that would otherwise mask the true bottleneck, and the fused GQA
|
|
results then show unambiguously that the kernel is data-movement bound.
|
|
Without a cheap, self-routing issue mechanism we would not be able to tell
|
|
whether attention is slow because of compute or because of data movement;
|
|
with it, the answer is clear.
|
|
|
|
\paragraph{The communication path is where attention latency actually
|
|
lives.} Every all-reduce and fused-GQA measurement says the same thing:
|
|
the limiting resource is moving and reducing data, not multiplying it. That
|
|
makes the PE\_IPCQ design and the choices around it the highest-value
|
|
hardware levers for this workload:
|
|
\begin{itemize}
|
|
\item \textbf{On-device collectives with a compute/communication
|
|
virtual-channel split.} Performing the reduction on the device, with
|
|
\texttt{vc\_comm} separated from \texttt{vc\_compute} so the reduction
|
|
does not stall the compute DMA, is what lets attention overlap KV
|
|
movement with score computation at all.
|
|
\item \textbf{Fast on-PE staging memory.} Keeping the IPCQ buffer in TCM
|
|
rather than HBM or SRAM is worth \SI{14}{}--\SI{38}{\percent} of
|
|
collective latency at large payloads---a pure placement decision with a
|
|
first-order effect.
|
|
\item \textbf{Wrap-around (torus) inter-device links.} A torus fabric
|
|
buys \SI{20}{}--\SI{25}{\percent} over a mesh or ring at scale by
|
|
shortening the worst-case reduction path.
|
|
\end{itemize}
|
|
|
|
\paragraph{Raw MAC throughput is not the constraint for attention.} The
|
|
GQA panels leave the GEMM and vector-math engines two to three orders of
|
|
magnitude below the DMA engine in busy time. Adding MAC area would not move
|
|
decode latency; the workload cannot use it. This is the single most
|
|
actionable finding for an attention-dominated roadmap. The implication is
|
|
that future hardware investment should prioritize communication and
|
|
memory-system efficiency over additional compute throughput for
|
|
attention-dominated inference workloads.
|
|
|
|
\paragraph{Caveats.} These conclusions are achievable-kernel results from a
|
|
deterministic model, not E2E measurements; absolute numbers carry the
|
|
model's idealizations (\S\ref{sec:latency}), though the relative rankings
|
|
that drive the recommendations are robust to them. The headline GQA panels
|
|
are also at modest scale (up to four users, single SIP), and one designed
|
|
refinement---the two-composite \textsf{softmax\_merge} decode---is not yet
|
|
in the measured path. Larger-scale and multi-SIP headline runs are 2H work.
|