paper: §2 figure layout + Accuracy subsection + Table 1 → 2-col; §6 trim historical 4-panel content
§2 platform:
- Accuracy promoted from \paragraph to \subsection (sec:accuracy);
it sits at the same heading level as Why / Device / Latency / HW
config and so reads as a first-class component of the platform
description rather than a tail-end footnote.
- Three architecture diagrams (SIP, CUBE, PE) collapsed into one
figure* using subcaption: Fig.~\ref{fig:hw-arch}(a) SIP and (b)
CUBE side-by-side on row 1, (c) PE wide on row 2, total height
capped at ~40% of the page. Subfigure cross-references rewritten
to Fig.~ref{fig:hw-arch}\subref{...} in the body text. main.tex
now pulls in the subcaption package.
- Table 1 (modeled hardware configuration) promoted to table*
(two-column / full page width) and the row layout rewritten as a
4-column tabular so Hierarchy + PE + Command-issue sit on the
left and Memory + Interconnect sit on the right. The previous
single-column rendering was getting cut at the right edge of the
printed column.
§6 GQA:
- Removed the historical four-panel headline table (tab:gqa), the
two figures (fig:gqa-lat, fig:gqa-break), and the prose paragraph
that cited their per-panel numbers. Reason: the underlying
milestone-gqa-headline bench has been simplified in collaborator
commit 65c365f ("drop misleading single_user_/multi_user_
panels") and no longer reproduces that data, so the section was
left referencing a dataset the current bench cannot regenerate.
- The previously-added 4-cases long-context decode comparison is
now §6's only Results subsection, retitled to
"Results: long-context decode and parallelism strategies".
The headline GQA result that survives into the paper is therefore
the parallelism trade-off study (Case 4 / Cube-SP × PE-SP at 34 us
buying an 8x KV-memory reduction over the latency leaders).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -41,70 +41,7 @@ that restructures the decode step into two stateful composites (a named
|
||||
\textsf{softmax\_merge} recipe) is designed but not yet wired into the
|
||||
measured path; results below reflect the implemented kernel only.
|
||||
|
||||
\subsection{Results}
|
||||
|
||||
We measure four headline panels that vary the user count $C$ and the phase:
|
||||
single- and multi-user prefill ($T_q=4$, $S_{kv}=16$), and single- and
|
||||
multi-user decode ($P=8$ PEs, $S_{kv}=64$ and $128$), all at $d_{\text{head}}=64$
|
||||
and $G=8$. For each panel we harvest end-to-end latency
|
||||
(max event end minus min event start, the same window convention as the
|
||||
GEMM study) together with the per-engine busy time and the operation mix.
|
||||
Figure~\ref{fig:gqa-lat} and Figure~\ref{fig:gqa-break} report the result;
|
||||
the underlying numbers are in Table~\ref{tab:gqa}.
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Fused GQA per-panel latency and operation mix. Compute (GEMM,
|
||||
MATH) is a tiny fraction of DMA occupancy; IPCQ copies grow with users and
|
||||
PEs.}
|
||||
\label{tab:gqa}
|
||||
\small
|
||||
\begin{tabular}{@{}lrrrr@{}}
|
||||
\toprule
|
||||
\textbf{Panel} & \textbf{Lat.\ (ns)} & \textbf{GEMM} & \textbf{IPCQ} & \textbf{DMA rd} \\
|
||||
\midrule
|
||||
prefill C=1 & 445 & 2 & 0 & 3 \\
|
||||
prefill C=4 (Ring) & 4630 & 32 & 24 & 12 \\
|
||||
decode C=1, P=8 & 3632 & 16 & 21 & 24 \\
|
||||
decode C=4, P=8 & 6693 & 64 & 93 & 96 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\includegraphics[width=\linewidth]{gqa_latency_by_panel.png}
|
||||
\caption{Fused GQA end-to-end latency. Latency grows from
|
||||
\SI{445}{\nano\second} (single-user prefill) to \SI{6693}{\nano\second}
|
||||
(four-user decode) as the KV history and the number of participating
|
||||
devices grow.}
|
||||
\label{fig:gqa-lat}
|
||||
\end{figure}
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\includegraphics[width=\linewidth]{gqa_op_engine_breakdown.png}
|
||||
\caption{Where the work goes. Left: operation counts---GEMM and IPCQ-copy
|
||||
volume both scale with users and PEs. Right: summed engine occupancy on a
|
||||
log scale---the DMA engine dominates by two to three orders of magnitude
|
||||
over the GEMM and MATH engines in every panel.}
|
||||
\label{fig:gqa-break}
|
||||
\end{figure}
|
||||
|
||||
The dominant observation is in Figure~\ref{fig:gqa-break}: the compute
|
||||
engines are almost idle. The GEMM engine accumulates only
|
||||
\SIrange{2}{33}{\nano\second} of busy time across the panels and the
|
||||
vector-math engine \SIrange{5}{688}{\nano\second}, while the DMA engine
|
||||
accumulates \SIrange{72}{15920}{\nano\second}. Fused GQA, as modeled here,
|
||||
is overwhelmingly data-movement bound. The operation mix shows why the
|
||||
collective machinery matters: IPCQ-copy count rises from zero (single-user
|
||||
prefill) to 93 (four-user decode) as the kernel reduces partial outputs
|
||||
across more PEs and CUBEs, and DMA-read count rises in step as more KV
|
||||
shards are streamed. The PE control-processor dispatch cost registered as
|
||||
zero in this configuration---command issue is simply not on the critical
|
||||
path when data movement is this dominant.
|
||||
|
||||
\subsection{Long-context decode: parallelism strategies}
|
||||
\subsection{Results: long-context decode and parallelism strategies}
|
||||
|
||||
The four headline panels above stress the kernel at moderate context
|
||||
lengths. Long-context decode---the regime where KV cache size, not
|
||||
|
||||
Reference in New Issue
Block a user