paper(gemm): per-PE HBM BW + TFLOPS evaluation plots; unify peak to 8 TFLOP/s

Two new evaluation plots for §3 GEMM, both using the composite window
as the denominator (the up-front tl.load of A in load_ref is therefore
excluded — only HBM traffic and compute inside the composite count):

- gemm_hbm_bw_util.png: per-PE HBM bandwidth utilization against the
  256 GB/s per-PE ceiling. Answers "is this workload memory-bound on
  this configuration?" — small/single-tile shapes sit at 22–46%
  (pipeline-fill-limited), large or output-write-dominated shapes
  saturate (≥85%).

- gemm_per_pe_tflops.png: per-PE GEMM throughput against the 8 TFLOP/s
  engine peak. The deep-K K=3072 load_ref case lands at
  7.18 TFLOP/s (~90% of peak) — the configuration's clean win.
  ref_ref at the same shape drops to 3.83 TFLOP/s because the second
  operand doubles HBM pressure and the kernel hits the BW ceiling.
  The variant gap on this chart is the operational cost of NOT
  pre-staging the activation.

Both charts compare two operand-staging variants on every shape:
load_ref ("activation pre-staged", weight-only HBM streaming) and
ref_ref (both A and W streamed from HBM). load_load is omitted —
with both operands pre-staged the composite carries no HBM traffic
and the BW metric collapses to 0.

Analytic / measured peak unified to the single hardware spec
(8.0 TFLOP/s = 8000 flops/ns). T_STAGE is now derived as
tile_flops / peak (= 16.384 ns for the 32×64×32 tile) rather than
the previous hardcoded 16 ns, so MAC efficiency and per-PE TFLOPS%
share the same denominator and never disagree. Side effect: max
analytic-vs-measured gap tightens from 2.2 ppt to 1.4 ppt across
the sweep.

§3 Results restructured: lead with the two new evaluation plots
(BW saturation, achieved throughput), then the MAC-utilization
analytic-vs-measured chart serves as the validation step, then the
per-stage engine wall-clock as the supporting diagnostic. §3
Analysis updated to use the load_ref / ref_ref contrast as the
hardware-software boundary line that motivates the GQA kernel of §5.

§2.4 Accuracy claim updated to match: GEMM analytic-vs-measured
agreement is now "within 1.4 ppt across every swept shape", from
~7.7% at single-tile up to ~90% at K=3072. Subtitle/title vertical
stacking glitch in the matplotlib charts fixed in passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/report/1H-codesign-paper/sections/02-platform.tex

@@ -311,9 +311,9 @@ translates into physically reasonable kernel latencies.
 
 First, in the GEMM study (\S\ref{sec:gemm}), simulator-measured MAC
 efficiency tracks an analytic ideal-pipeline model within
-\SI{2.2}{ppt} across every swept shape---from a single-tile
-$M{=}K{=}N{=}32$ at $\sim\SI{7.5}{\percent}$ up to the deep-$K$
-$K{=}3072$ case at $\sim\SI{88}{\percent}$. The residual gap is
+\SI{1.4}{ppt} across every swept shape---from a single-tile
+$M{=}K{=}N{=}32$ at $\sim\SI{7.7}{\percent}$ up to the deep-$K$
+$K{=}3072$ case at $\sim\SI{90}{\percent}$. The residual gap is
 fill/tail overhead the closed-form pipeline model omits.
 
 Second, in the all-reduce study (\S\ref{sec:allreduce},

docs/report/1H-codesign-paper/sections/03-gemm.tex

@@ -43,12 +43,84 @@ kernel to fuse its softmax work into the GEMM pipeline
 \subsection{Results}
 
 We sweep eight GEMM shapes spanning square, tall, wide, and deep-$K$
-geometries, under three operand-staging variants
-(\textsf{ref\_ref}, both operands streamed from HBM; \textsf{load\_ref},
-one operand resident in TCM; \textsf{load\_load}, both resident).
-Figure~\ref{fig:gemm-util} reports MAC utilization and efficiency, and
-Figure~\ref{fig:gemm-stages} breaks the kernel into per-stage engine
-busy time.
+geometries under two operand-staging variants that bracket the
+realistic LLM cases. In \textsf{load\_ref}, the activation $A$ is
+pre-staged on chip and only the weight $W$ streams from HBM during
+the composite (the ``activation in TCM, weights from HBM'' case
+typical of decoding with a small batch). In \textsf{ref\_ref}, both
+operands stream from HBM during the composite (the ``cold both
+sides'' case, where the activation does not fit on-chip or is itself
+the output of a producer composite that wrote back to HBM). Both
+variants run the same composite command; only the up-front staging
+of $A$ differs.
+
+We evaluate the composite GEMM along two axes that together describe
+how well it lands on the hardware: \emph{HBM bandwidth utilization}
+(does the workload saturate the per-PE HBM bandwidth, i.e.\ is it
+memory-bound on this configuration?) and \emph{achieved GEMM
+throughput} (what fraction of the \SI{8}{\tera\flop\per\second}
+per-PE peak does the kernel actually deliver). Both metrics use the
+composite window as the denominator, so the up-front \textsf{tl.load}
+of $A$ in \textsf{load\_ref} is excluded — only HBM traffic and
+compute that happen \emph{inside} the composite count.
+
+\begin{figure}[t]
+\centering
+\includegraphics[width=\linewidth]{gemm_hbm_bw_util.png}
+\caption{Per-PE HBM bandwidth utilization during the composite window,
+for the two staging variants. The ceiling is the per-PE HBM bandwidth
+(\SI{256}{\giga\byte\per\second}; dashed line at \SI{100}{\percent}).
+\textsf{ref\_ref} streams both $A$ and $W$ and so always sits at a
+higher BW-utilization than \textsf{load\_ref} on the same shape;
+both variants saturate ($>\SI{85}{\percent}$) once the kernel
+has enough total bytes to keep the link busy (deep-$K$ $K{=}3072$,
+or low-reuse output-dominated $M{=}128,K{=}8,N{=}128$). Small or
+single-tile shapes do not have enough total traffic to saturate the
+link and are bottlenecked by pipeline fill rather than HBM.}
+\label{fig:gemm-bw}
+\end{figure}
+
+\begin{figure}[t]
+\centering
+\includegraphics[width=\linewidth]{gemm_per_pe_tflops.png}
+\caption{Per-PE GEMM throughput delivered during the composite window
+(reference line at the \SI{8}{\tera\flop\per\second} per-PE engine
+peak). At the deep-$K$ corner $M{=}32,K{=}3072,N{=}32$ the
+activation-pre-staged kernel reaches
+\SI{7.18}{\tera\flop\per\second} ($\sim\SI{90}{\percent}$ of peak);
+the both-from-HBM variant drops to
+\SI{3.83}{\tera\flop\per\second} ($\sim\SI{48}{\percent}$) at the
+same shape because the second operand doubles HBM pressure and the
+kernel hits the BW ceiling shown in Fig.~\ref{fig:gemm-bw}.
+Under-tile shapes ($M{=}K{=}N{=}32$, $M{=}8$, $K{=}8$) are
+hard-capped by tile-fill at \SI{50}{\percent}, \SI{25}{\percent},
+\SI{12.5}{\percent} of peak regardless of staging.}
+\label{fig:gemm-tflops}
+\end{figure}
+
+The composite reaches the hardware roofline in two distinct regimes.
+\emph{Memory-bound}: $K{=}3072$ deep-$K$ saturates HBM (\SI{88}{\percent}
+load\_ref, \SI{94}{\percent} ref\_ref) and the low-reuse
+$M{=}128,K{=}8,N{=}128$ corner saturates even harder
+($>\SI{96}{\percent}$ in both), because back-to-back output
+writes dominate. \emph{Compute-rich}: at the same deep-$K$ shape,
+\textsf{load\_ref} delivers \SI{7.18}{\tera\flop\per\second}
+($\sim\SI{90}{\percent}$ of the per-PE peak) — the configuration's
+clean win. The \emph{distance between the two variants} at the same
+shape is the cost of \emph{not} pre-staging $A$: at $K{=}3072$ the
+throughput halves (\SI{7.18}{\tera\flop\per\second} $\to$
+\SI{3.83}{\tera\flop\per\second}) because the second operand doubles
+HBM pressure
+and the kernel hits the BW ceiling. This is the operational
+take-away — when the activation can live on-chip the composite
+delivers near-peak GEMM; when it cannot the BW ceiling dominates and
+throughput halves.
+
+Figure~\ref{fig:gemm-util} validates that simulator-measured
+efficiency tracks an analytic ideal-pipeline model (within
+\SI{1.4}{ppt} across every swept shape), and
+Figure~\ref{fig:gemm-stages} resolves the per-stage engine wall-clock
+that backs the above interpretation.
 
 \begin{figure}[t]
 \centering
@@ -60,9 +132,9 @@ cannot fill the MAC tile and cap at \SI{50}{\percent},
 \SI{25}{\percent}, \SI{12.5}{\percent}. For tile-filling shapes,
 efficiency climbs with tile count---from
 \textasciitilde\SI{15}{\percent} at one tile to \textasciitilde%
-\SI{88}{\percent} measured at 48 tiles ($K{=}3072$)---and the
+\SI{90}{\percent} measured at 48 tiles ($K{=}3072$)---and the
 measured bars track the analytic ideal-pipeline prediction within
-\SI{2.2}{ppt} across every shape.}
+\SI{1.4}{ppt} across every shape.}
 \label{fig:gemm-util}
 \end{figure}
 
@@ -81,31 +153,24 @@ where the output write becomes the largest stage.}
 \label{fig:gemm-stages}
 \end{figure}
 
-Two regularities stand out. (i) Utilization is governed by how
-completely the problem fills the MAC tile: the three under-tile shapes
-are hard-capped well below \SI{100}{\percent}, independent of how the
-kernel is issued. (ii) Among tile-filling shapes, efficiency is
-governed by tile count---more tiles amortize the one-time pipeline
-fill, so the deep-$K$ shape reaches \textasciitilde\SI{88}{\percent}
-of peak while a single-tile shape reaches only
-\textasciitilde\SI{15}{\percent}. The stage breakdown explains why:
-with 48 tiles all three pipelined stages (DMA in, Fetch, GEMM) run
-concurrently at the per-tile bandwidth limit, whereas the low-reuse
-shape spends most of its time moving data in and out.
-
 \subsection{Analysis and meaning}
 
-The composite command does not manufacture bandwidth or MAC throughput; it
-removes the two software-shaped obstacles between a GEMM and its hardware
-roofline. By paying issue cost once per GEMM and chaining tile-stages
-through token self-routing, it lets the scheduler keep the pipeline full,
-so compute-rich shapes actually reach the efficiency their arithmetic
-intensity allows ($\sim$\SI{88}{\percent} measured at 48 tiles), and
-data-bound shapes actually reach their DMA bound instead of stalling on
-command overhead. The close agreement between measured and theoretical
-efficiency (Figure~\ref{fig:gemm-util}) is also the report's primary
-validation that KernBench's latency model is faithful in the regime that
-matters. The hardware implication is concrete: a single-command,
-self-routing tile pipeline is the issue mechanism that makes the MAC array
-usable, and it is a prerequisite---not a luxury---for the fused attention
-kernel of \S\ref{sec:gqa}.
+The composite command does not manufacture bandwidth or MAC
+throughput; it removes the two software-shaped obstacles between a
+GEMM and its hardware roofline. By paying the issue cost once per
+GEMM rather than once per tile-stage, and by chaining tile-stages
+through token self-routing, it lets the scheduler keep every stage
+busy on a different tile, so compute-rich shapes actually reach the
+efficiency their arithmetic intensity allows
+($\sim$\SI{90}{\percent} of GEMM peak at the deep-$K$
+\textsf{load\_ref} corner) and data-bound shapes actually reach
+their HBM-BW ceiling instead of stalling on command overhead.
+
+The split between \textsf{load\_ref} and \textsf{ref\_ref} also makes
+the hardware-software boundary explicit: when the activation fits in
+on-chip storage the composite is BW-headroom-rich and saturates the
+GEMM engine; when it does not, both operands compete for the same
+HBM port and the kernel is BW-bound. This is the lever the GQA
+kernel of \S\ref{sec:gqa} reaches for next — keeping the right
+working set on-chip so the composite pipeline lands in the
+compute-rich regime rather than the BW-bound one.