paper: §2 figure layout + Accuracy subsection + Table 1 → 2-col; §6 trim historical 4-panel content
§2 platform:
- Accuracy promoted from \paragraph to \subsection (sec:accuracy);
it sits at the same heading level as Why / Device / Latency / HW
config and so reads as a first-class component of the platform
description rather than a tail-end footnote.
- Three architecture diagrams (SIP, CUBE, PE) collapsed into one
figure* using subcaption: Fig.~\ref{fig:hw-arch}(a) SIP and (b)
CUBE side-by-side on row 1, (c) PE wide on row 2, total height
capped at ~40% of the page. Subfigure cross-references rewritten
to Fig.~ref{fig:hw-arch}\subref{...} in the body text. main.tex
now pulls in the subcaption package.
- Table 1 (modeled hardware configuration) promoted to table*
(two-column / full page width) and the row layout rewritten as a
4-column tabular so Hierarchy + PE + Command-issue sit on the
left and Memory + Interconnect sit on the right. The previous
single-column rendering was getting cut at the right edge of the
printed column.
§6 GQA:
- Removed the historical four-panel headline table (tab:gqa), the
two figures (fig:gqa-lat, fig:gqa-break), and the prose paragraph
that cited their per-panel numbers. Reason: the underlying
milestone-gqa-headline bench has been simplified in collaborator
commit 65c365f ("drop misleading single_user_/multi_user_
panels") and no longer reproduces that data, so the section was
left referencing a dataset the current bench cannot regenerate.
- The previously-added 4-cases long-context decode comparison is
now §6's only Results subsection, retitled to
"Results: long-context decode and parallelism strategies".
The headline GQA result that survives into the paper is therefore
the parallelism trade-off study (Case 4 / Cube-SP × PE-SP at 34 us
buying an 8x KV-memory reduction over the latency leaders).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -39,73 +39,73 @@ KernBench models AHBM as a hierarchy of SIPs, CUBEs, and processing
|
||||
elements (PEs). At the system level, multiple SIPs are connected
|
||||
through inter-package links, while each SIP contains a collection of
|
||||
CUBEs joined by an on-package interconnect
|
||||
(Fig.~\ref{fig:sip-arch}).
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\includegraphics[width=\linewidth]{sip_architecture.pdf}
|
||||
\caption{Modeled hardware graph at the \emph{SIP} level (one
|
||||
example configuration; specific parameters in \S\ref{sec:hw} /
|
||||
Table~\ref{tab:hw}). The SIP holds a $4{\times}4$ mesh of CUBEs and
|
||||
an IO chiplet, with each line a directed link labelled by its
|
||||
physical distance and bandwidth.}
|
||||
\label{fig:sip-arch}
|
||||
\end{figure}
|
||||
(Fig.~\ref{fig:hw-arch}\subref{fig:sip-arch}).
|
||||
|
||||
Each CUBE contains eight PEs, shared SRAM, HBM controllers, an
|
||||
\textsf{M\_CPU} control processor, and an intra-CUBE router mesh
|
||||
(Fig.~\ref{fig:cube-arch}). Together these components form the
|
||||
execution substrate for all kernels evaluated in this report. This
|
||||
organization reflects the memory-centric nature of AHBM: each PE is
|
||||
paired with a dedicated slice of HBM bandwidth and local on-PE
|
||||
storage (TCM), so compute lives next to the data it consumes rather
|
||||
than fetching it through a far-away memory controller. The platform's
|
||||
performance question is therefore not ``how many FLOPs can the chip
|
||||
do'' but ``how well can a kernel keep each compute engine fed from
|
||||
its locally-attached memory while moving the unavoidable traffic
|
||||
between PEs efficiently.''
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\includegraphics[width=\linewidth]{cube_architecture.pdf}
|
||||
\caption{Modeled hardware graph at the \emph{CUBE} level---a
|
||||
zoom-in on one CUBE node from Fig.~\ref{fig:sip-arch}. The CUBE
|
||||
holds 8 PEs (each with its own HBM channels), the \textsf{M\_CPU}
|
||||
control processor, a shared SRAM, an intra-CUBE NoC router mesh, and
|
||||
UCIe links to neighbouring CUBEs.}
|
||||
\label{fig:cube-arch}
|
||||
\end{figure}
|
||||
(Fig.~\ref{fig:hw-arch}\subref{fig:cube-arch}). Together these
|
||||
components form the execution substrate for all kernels evaluated in
|
||||
this report. This organization reflects the memory-centric nature of
|
||||
AHBM: each PE is paired with a dedicated slice of HBM bandwidth and
|
||||
local on-PE storage (TCM), so compute lives next to the data it
|
||||
consumes rather than fetching it through a far-away memory
|
||||
controller. The platform's performance question is therefore not
|
||||
``how many FLOPs can the chip do'' but ``how well can a kernel keep
|
||||
each compute engine fed from its locally-attached memory while
|
||||
moving the unavoidable traffic between PEs efficiently.''
|
||||
|
||||
Within a PE, commands are dispatched by \textsf{PE\_CPU} to
|
||||
\textsf{PE\_SCHED}, which routes work to specialized execution
|
||||
engines---DMA, FETCH/STORE, GEMM, vector-math, and IPCQ
|
||||
(Fig.~\ref{fig:pe-arch}). Commands come in two flavours. \emph{Atomic}
|
||||
commands target a single engine---a plain DMA read, a GEMM tile, a
|
||||
vector-math op, an IPCQ send/receive---and are the natural unit for
|
||||
short or special-purpose work; the \textsf{PE\_CPU} itself also runs
|
||||
control-plane work directly. \emph{Composite} commands, by contrast,
|
||||
carry an ordered pipeline of operations across multiple engines
|
||||
(\textsf{DMA\_READ} $\rightarrow$ \textsf{FETCH} $\rightarrow$
|
||||
\textsf{GEMM}/\textsf{MATH} $\rightarrow$ \textsf{STORE} $\rightarrow$
|
||||
\textsf{DMA\_WRITE}) that the \textsf{PE\_SCHED} tiles and streams
|
||||
without per-tile redispatch. The composite form is the substrate for
|
||||
the GEMM optimization (\S\ref{sec:gemm}) and, combined with on-PE
|
||||
collectives, for fused attention (\S\ref{sec:gqa}).
|
||||
(Fig.~\ref{fig:hw-arch}\subref{fig:pe-arch}). Commands come in two
|
||||
flavours. \emph{Atomic} commands target a single engine---a plain
|
||||
DMA read, a GEMM tile, a vector-math op, an IPCQ send/receive---and
|
||||
are the natural unit for short or special-purpose work; the
|
||||
\textsf{PE\_CPU} itself also runs control-plane work directly.
|
||||
\emph{Composite} commands, by contrast, carry an ordered pipeline of
|
||||
operations across multiple engines (\textsf{DMA\_READ} $\rightarrow$
|
||||
\textsf{FETCH} $\rightarrow$ \textsf{GEMM}/\textsf{MATH} $\rightarrow$
|
||||
\textsf{STORE} $\rightarrow$ \textsf{DMA\_WRITE}) that the
|
||||
\textsf{PE\_SCHED} tiles and streams without per-tile redispatch. The
|
||||
composite form is the substrate for the GEMM optimization
|
||||
(\S\ref{sec:gemm}) and, combined with on-PE collectives, for fused
|
||||
attention (\S\ref{sec:gqa}).
|
||||
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\includegraphics[width=\linewidth]{pe_architecture.png}
|
||||
\caption{Modeled PE architecture used in this report---one example
|
||||
configuration whose specific parameters are listed in
|
||||
\S\ref{sec:hw} (Table~\ref{tab:hw}); KernBench is not tied to this
|
||||
particular decomposition and supports arbitrary PE block layouts
|
||||
provided each component is given a port and bandwidth model.
|
||||
\textsf{PE\_CPU} dispatches commands to the \textsf{PE\_SCHED}, which
|
||||
routes tile-token streams through the \textsf{PE\_DMA},
|
||||
\textsf{PE\_FETCH\_STORE}, and \textsf{GEMM}/\textsf{MATH} engines
|
||||
along on-chip links; the \textsf{PE\_IPCQ} provides the control plane
|
||||
for on-device collective communication.}
|
||||
\label{fig:pe-arch}
|
||||
\begin{subfigure}[t]{0.46\textwidth}
|
||||
\centering
|
||||
\includegraphics[width=\linewidth,height=0.85\linewidth,keepaspectratio]{sip_architecture.pdf}
|
||||
\caption{SIP level: $4{\times}4$ CUBE mesh + IO chiplet.}
|
||||
\label{fig:sip-arch}
|
||||
\end{subfigure}\hfill
|
||||
\begin{subfigure}[t]{0.46\textwidth}
|
||||
\centering
|
||||
\includegraphics[width=\linewidth,height=0.85\linewidth,keepaspectratio]{cube_architecture.pdf}
|
||||
\caption{CUBE level (zoom-in of one CUBE in (a)): 8 PEs, HBM
|
||||
channels, M\_CPU, SRAM, NoC router mesh, UCIe links.}
|
||||
\label{fig:cube-arch}
|
||||
\end{subfigure}
|
||||
|
||||
\vspace{0.6em}
|
||||
|
||||
\begin{subfigure}[t]{\textwidth}
|
||||
\centering
|
||||
\includegraphics[width=\linewidth]{pe_architecture.png}
|
||||
\caption{PE level (zoom-in of one PE in (b)): \textsf{PE\_CPU}
|
||||
dispatches commands to \textsf{PE\_SCHED}, which routes tile-token
|
||||
streams through \textsf{PE\_DMA}, \textsf{PE\_FETCH\_STORE}, and
|
||||
\textsf{GEMM}/\textsf{MATH} engines; \textsf{PE\_IPCQ} is the
|
||||
on-device collective control plane.}
|
||||
\label{fig:pe-arch}
|
||||
\end{subfigure}
|
||||
\caption{Modeled hardware graph at the SIP, CUBE, and PE levels
|
||||
(one example configuration; specific parameters in \S\ref{sec:hw} /
|
||||
Table~\ref{tab:hw}). KernBench is not tied to this particular
|
||||
arrangement: each box is a modeled component node, each line a
|
||||
directed link with bandwidth and propagation attributes, and any
|
||||
topology that respects those attributes is supported.}
|
||||
\label{fig:hw-arch}
|
||||
\end{figure*}
|
||||
|
||||
KernBench is layered along the flow of a request:
|
||||
@@ -171,9 +171,8 @@ There are no hidden shortcuts, implicit bypasses, or magic paths: if a
|
||||
request reaches its destination, the path it took is explicit in the
|
||||
graph, and the latency it incurred is the sum of the per-node and
|
||||
per-edge costs paid along that path. The same graph representation
|
||||
applies recursively at every hierarchy level---system, SIP
|
||||
(Fig.~\ref{fig:sip-arch}), CUBE (Fig.~\ref{fig:cube-arch}), and PE
|
||||
(Fig.~\ref{fig:pe-arch}).
|
||||
applies recursively at every hierarchy level---system, SIP, CUBE, and
|
||||
PE (Fig.~\ref{fig:hw-arch}).
|
||||
|
||||
\paragraph{From graph to discrete-event simulation.} The graph is
|
||||
driven by a discrete-event engine. Two kinds of events advance
|
||||
@@ -247,7 +246,10 @@ queues accept. In the configurations measured here, command issue is
|
||||
not the bottleneck---data movement is---so this term stays small
|
||||
relative to DMA and collective time.
|
||||
|
||||
\paragraph{Accuracy.} The model is precise about the effects that
|
||||
\subsection{Accuracy}
|
||||
\label{sec:accuracy}
|
||||
|
||||
The model is precise about the effects that
|
||||
dominate kernel latency on this class of hardware: per-edge bandwidth
|
||||
occupancy and flit-level serialization, HBM pseudo-channel parallelism,
|
||||
and per-component switching overhead. Two independent cross-checks
|
||||
@@ -367,44 +369,33 @@ configuration. Workload-specific parameters such as
|
||||
matrix dimensions, sequence lengths, and collective
|
||||
payload sizes are introduced in their respective sections.
|
||||
|
||||
\begin{table}[t]
|
||||
\begin{table*}[t]
|
||||
\centering
|
||||
\caption{Modeled hardware configuration (shared by all experiments).}
|
||||
\label{tab:hw}
|
||||
\small
|
||||
\begin{tabular}{@{}ll@{}}
|
||||
\begin{tabular}{@{}ll@{\hspace{3em}}ll@{}}
|
||||
\toprule
|
||||
\textbf{Parameter} & \textbf{Value} \\
|
||||
\textbf{Parameter} & \textbf{Value} & \textbf{Parameter} & \textbf{Value} \\
|
||||
\midrule
|
||||
\multicolumn{2}{@{}l}{\emph{Hierarchy}} \\
|
||||
SIPs & 2 (1D ring) \\
|
||||
CUBEs per SIP & 16 ($4\times4$ mesh) \\
|
||||
PEs per CUBE & 8 (4 corners $\times$ 2) \\
|
||||
PEs total & 256 \\
|
||||
\multicolumn{2}{@{}l}{\emph{Hierarchy}} & \multicolumn{2}{@{}l}{\emph{Memory (per CUBE)}} \\
|
||||
SIPs & 2 (1D ring) & HBM capacity & \SI{48}{\giga\byte} (8 slices) \\
|
||||
CUBEs per SIP & 16 ($4\times4$ mesh) & HBM aggregate BW & \SI{1024}{\giga\byte\per\second} \\
|
||||
PEs per CUBE & 8 (4 corners $\times$ 2) & HBM pseudo-channels & 64 (8 per PE), \SI{32}{\giga\byte\per\second} each \\
|
||||
PEs total & 256 & SRAM (shared) & \SI{32}{\mega\byte}, \SI{128}{\giga\byte\per\second} link \\
|
||||
& & HBM burst & \SI{256}{\byte} \\
|
||||
\midrule
|
||||
\multicolumn{2}{@{}l}{\emph{Processing element (PE)}} \\
|
||||
GEMM engine peak & \SI{8}{\tera\flop\per\second} (f16) \\
|
||||
TCM (on-PE) & \SI{16}{\mega\byte}, \SI{512}{\giga\byte\per\second} R/W \\
|
||||
\quad kernel scratch & \SI{1}{\mega\byte} \\
|
||||
DMA engines & 1 read + 1 write \\
|
||||
CPU / scheduler overhead & \SI{2}{\nano\second} / \SI{1}{\nano\second} \\
|
||||
\multicolumn{2}{@{}l}{\emph{Processing element (PE)}} & \multicolumn{2}{@{}l}{\emph{Interconnect}} \\
|
||||
GEMM engine peak & \SI{8}{\tera\flop\per\second} (f16) & Intra-CUBE NoC link & \SI{256}{\giga\byte\per\second}, \SI{2}{\nano\second}/router \\
|
||||
TCM (on-PE) & \SI{16}{\mega\byte}, \SI{512}{\giga\byte\per\second} R/W & Inter-CUBE (UCIe PHY) & \SI{512}{\giga\byte\per\second}, \SI{8}{\nano\second}, XY routing \\
|
||||
\quad kernel scratch & \SI{1}{\mega\byte} & Inter-SIP (PCIe) & \SI{768}{\giga\byte\per\second} per endpoint \\
|
||||
DMA engines & 1 read + 1 write & & \\
|
||||
CPU / scheduler overhead & \SI{2}{\nano\second} / \SI{1}{\nano\second} & & \\
|
||||
\midrule
|
||||
\multicolumn{2}{@{}l}{\emph{Memory (per CUBE)}} \\
|
||||
HBM capacity & \SI{48}{\giga\byte} (8 slices) \\
|
||||
HBM aggregate BW & \SI{1024}{\giga\byte\per\second} \\
|
||||
HBM pseudo-channels & 64 (8 per PE), \SI{32}{\giga\byte\per\second} each \\
|
||||
SRAM (shared) & \SI{32}{\mega\byte}, \SI{128}{\giga\byte\per\second} link \\
|
||||
HBM burst & \SI{256}{\byte} \\
|
||||
\midrule
|
||||
\multicolumn{2}{@{}l}{\emph{Interconnect}} \\
|
||||
Intra-CUBE NoC link & \SI{256}{\giga\byte\per\second}, \SI{2}{\nano\second}/router \\
|
||||
Inter-CUBE (UCIe PHY) & \SI{512}{\giga\byte\per\second}, \SI{8}{\nano\second}, XY routing \\
|
||||
Inter-SIP (PCIe) & \SI{768}{\giga\byte\per\second} per endpoint \\
|
||||
\midrule
|
||||
\multicolumn{2}{@{}l}{\emph{Command-issue cost model (defaults)}} \\
|
||||
FIXED per command & 40 cycles \\
|
||||
per-byte rate $R$ & 0.0625 cycles/byte (\SI{16}{\byte\per\cycle}) \\
|
||||
composite size cap & \SI{1024}{\byte} \\
|
||||
\multicolumn{2}{@{}l}{\emph{Command-issue cost model (defaults)}} & & \\
|
||||
FIXED per command & 40 cycles & & \\
|
||||
per-byte rate $R$ & 0.0625 cycles/byte (\SI{16}{\byte\per\cycle}) & & \\
|
||||
composite size cap & \SI{1024}{\byte} & & \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\end{table*}
|
||||
|
||||
Reference in New Issue
Block a user