paper: §2 figure layout + Accuracy subsection + Table 1 → 2-col; §6 trim historical 4-panel content

§2 platform:
- Accuracy promoted from \paragraph to \subsection (sec:accuracy);
  it sits at the same heading level as Why / Device / Latency / HW
  config and so reads as a first-class component of the platform
  description rather than a tail-end footnote.
- Three architecture diagrams (SIP, CUBE, PE) collapsed into one
  figure* using subcaption: Fig.~\ref{fig:hw-arch}(a) SIP and (b)
  CUBE side-by-side on row 1, (c) PE wide on row 2, total height
  capped at ~40% of the page. Subfigure cross-references rewritten
  to Fig.~ref{fig:hw-arch}\subref{...} in the body text. main.tex
  now pulls in the subcaption package.
- Table 1 (modeled hardware configuration) promoted to table*
  (two-column / full page width) and the row layout rewritten as a
  4-column tabular so Hierarchy + PE + Command-issue sit on the
  left and Memory + Interconnect sit on the right. The previous
  single-column rendering was getting cut at the right edge of the
  printed column.

§6 GQA:
- Removed the historical four-panel headline table (tab:gqa), the
  two figures (fig:gqa-lat, fig:gqa-break), and the prose paragraph
  that cited their per-panel numbers. Reason: the underlying
  milestone-gqa-headline bench has been simplified in collaborator
  commit 65c365f ("drop misleading single_user_/multi_user_
  panels") and no longer reproduces that data, so the section was
  left referencing a dataset the current bench cannot regenerate.
- The previously-added 4-cases long-context decode comparison is
  now §6's only Results subsection, retitled to
  "Results: long-context decode and parallelism strategies".

The headline GQA result that survives into the paper is therefore
the parallelism trade-off study (Case 4 / Cube-SP × PE-SP at 34 us
buying an 8x KV-memory reduction over the latency leaders).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-16 11:16:16 -07:00
parent 22f3968b51
commit b715002b5a
4 changed files with 85 additions and 156 deletions
@@ -41,70 +41,7 @@ that restructures the decode step into two stateful composites (a named
\textsf{softmax\_merge} recipe) is designed but not yet wired into the
measured path; results below reflect the implemented kernel only.
\subsection{Results}
We measure four headline panels that vary the user count $C$ and the phase:
single- and multi-user prefill ($T_q=4$, $S_{kv}=16$), and single- and
multi-user decode ($P=8$ PEs, $S_{kv}=64$ and $128$), all at $d_{\text{head}}=64$
and $G=8$. For each panel we harvest end-to-end latency
(max event end minus min event start, the same window convention as the
GEMM study) together with the per-engine busy time and the operation mix.
Figure~\ref{fig:gqa-lat} and Figure~\ref{fig:gqa-break} report the result;
the underlying numbers are in Table~\ref{tab:gqa}.
\begin{table}[t]
\centering
\caption{Fused GQA per-panel latency and operation mix. Compute (GEMM,
MATH) is a tiny fraction of DMA occupancy; IPCQ copies grow with users and
PEs.}
\label{tab:gqa}
\small
\begin{tabular}{@{}lrrrr@{}}
\toprule
\textbf{Panel} & \textbf{Lat.\ (ns)} & \textbf{GEMM} & \textbf{IPCQ} & \textbf{DMA rd} \\
\midrule
prefill C=1 & 445 & 2 & 0 & 3 \\
prefill C=4 (Ring) & 4630 & 32 & 24 & 12 \\
decode C=1, P=8 & 3632 & 16 & 21 & 24 \\
decode C=4, P=8 & 6693 & 64 & 93 & 96 \\
\bottomrule
\end{tabular}
\end{table}
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{gqa_latency_by_panel.png}
\caption{Fused GQA end-to-end latency. Latency grows from
\SI{445}{\nano\second} (single-user prefill) to \SI{6693}{\nano\second}
(four-user decode) as the KV history and the number of participating
devices grow.}
\label{fig:gqa-lat}
\end{figure}
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{gqa_op_engine_breakdown.png}
\caption{Where the work goes. Left: operation counts---GEMM and IPCQ-copy
volume both scale with users and PEs. Right: summed engine occupancy on a
log scale---the DMA engine dominates by two to three orders of magnitude
over the GEMM and MATH engines in every panel.}
\label{fig:gqa-break}
\end{figure}
The dominant observation is in Figure~\ref{fig:gqa-break}: the compute
engines are almost idle. The GEMM engine accumulates only
\SIrange{2}{33}{\nano\second} of busy time across the panels and the
vector-math engine \SIrange{5}{688}{\nano\second}, while the DMA engine
accumulates \SIrange{72}{15920}{\nano\second}. Fused GQA, as modeled here,
is overwhelmingly data-movement bound. The operation mix shows why the
collective machinery matters: IPCQ-copy count rises from zero (single-user
prefill) to 93 (four-user decode) as the kernel reduces partial outputs
across more PEs and CUBEs, and DMA-read count rises in step as more KV
shards are streamed. The PE control-processor dispatch cost registered as
zero in this configuration---command issue is simply not on the critical
path when data movement is this dominant.
\subsection{Long-context decode: parallelism strategies}
\subsection{Results: long-context decode and parallelism strategies}
The four headline panels above stress the kernel at moderate context
lengths. Long-context decode---the regime where KV cache size, not