paper: §2 platform deep-edit + §1 GQA framing + figure regen
§2 (KernBench Platform): - Fix Table 2 HBM aggregate BW (1024 → 2048 GB/s); drop stale hbm_total_bw_gbs from topology.yaml (never read by sim_engine) - Split PE_CPU / PE_SCHED fixed-cost row; disambiguate from the 40-cycle command-dispatch FIXED term - §2.2 two-pass: expand to describe Pass 1 timing and Pass 2 data data-correctness path - §2.3 dispatch model: add motivation sentence for the descriptor-size linear form - §2.4 Accuracy: reorder GEMM → All-reduce → Probe → Simplifications; drop FSIM aside (already covered by §4 Fig 5 caption); soften 'every ns' → 'every modeled latency contribution' - §2.5 HW config: add 64 TFLOP/s vs 2048 GB/s (~31 FLOP/byte) balance-point intuition and forward pointer to §5 GQA decode - Fig 1 caption: separate illustrative topology from experimental configuration §1: tighten GQA-as-primary-bandwidth-bottleneck framing. Figures: regenerate SIP / CUBE architecture (SVG sources + PDF + generator scripts). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,16 +1,15 @@
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\begin{subfigure}[b]{0.42\textwidth}
|
||||
\begin{subfigure}[b]{0.495\textwidth}
|
||||
\centering
|
||||
\includegraphics[width=\linewidth,height=0.6\linewidth,keepaspectratio]{sip_architecture.pdf}
|
||||
\caption{SIP level: $4{\times}4$ CUBE mesh + IO chiplet.}
|
||||
\includegraphics[width=\linewidth,height=0.7\linewidth,keepaspectratio]{sip_architecture.pdf}
|
||||
\caption{SIP Architecture}
|
||||
\label{fig:sip-arch}
|
||||
\end{subfigure}\hfill
|
||||
\begin{subfigure}[b]{0.42\textwidth}
|
||||
\centering
|
||||
\includegraphics[width=\linewidth,height=0.6\linewidth,keepaspectratio]{cube_architecture.pdf}
|
||||
\caption{CUBE level (zoom-in of one CUBE in (a)): 8 PEs, HBM
|
||||
channels, M\_CPU, SRAM, NoC router mesh, UCIe links.}
|
||||
\includegraphics[width=\linewidth,height=0.84\linewidth]{cube_architecture.pdf}
|
||||
\caption{CUBE Architecture}
|
||||
\label{fig:cube-arch}
|
||||
\end{subfigure}
|
||||
|
||||
@@ -26,9 +25,12 @@
|
||||
on-device collective control plane.}
|
||||
\label{fig:pe-arch}
|
||||
\end{subfigure}
|
||||
\caption{Modeled hardware graph at the SIP, CUBE, and PE levels
|
||||
(one example configuration; specific parameters in \S\ref{sec:hw} /
|
||||
Table~\ref{tab:hw}). KernBench is not tied to this particular
|
||||
\caption{Modeled hardware graph at the SIP, CUBE, and PE levels.
|
||||
This figure is an \emph{illustrative topology} chosen for
|
||||
readability; the \emph{experimental configuration} used throughout
|
||||
this report (port counts, slice counts, and link parameters) is
|
||||
specified in \S\ref{sec:hw} / Table~\ref{tab:hw}, and numbers in the
|
||||
two may differ. KernBench is not tied to this particular
|
||||
arrangement: each box is a modeled component node, each line a
|
||||
directed link with bandwidth and propagation attributes, and any
|
||||
topology that respects those attributes is supported.}
|
||||
@@ -144,8 +146,20 @@ KernBench is layered along the flow of a request:
|
||||
controllers, and the inter-chiplet links.
|
||||
\end{itemize}
|
||||
|
||||
Data and timing are handled in two passes, so that a kernel's numeric
|
||||
results and its latency are computed consistently but independently.
|
||||
Data and timing are handled in two passes, so that a kernel's
|
||||
numeric results and its latency are computed consistently but
|
||||
independently. \textbf{Pass~1 (timing)} runs the kernel under
|
||||
the discrete-event engine: memory ops (\textsf{tl.load},
|
||||
\textsf{tl.store}) execute against a host-side \textsf{MemoryStore}
|
||||
and return real tensor data, while compute ops (GEMM, vector-math)
|
||||
only emit records into an op-log carrying their operands, shapes,
|
||||
dtypes, and scheduled time. \textbf{Pass~2 (data)} replays that
|
||||
op-log offline in numpy, producing the actual numeric outputs and
|
||||
comparing them against a reference computation under per-dtype
|
||||
tolerances (e.g.\ \texttt{rtol}/\texttt{atol} $=10^{-3}$ for f16).
|
||||
Pass~2 is optional---runs that need only latency skip it---but when
|
||||
enabled it guarantees that every reported timing number corresponds
|
||||
to a kernel whose numeric output has been verified end-to-end.
|
||||
|
||||
\subsection{Latency model: graph traversal and contention}
|
||||
\label{sec:latency}
|
||||
@@ -185,7 +199,7 @@ fires them one at a time, with ties broken under a deterministic
|
||||
policy so that the same kernel on the same topology always yields the
|
||||
same trace. Per-request correlation IDs are stamped at injection and
|
||||
carried through every hop, so the path from injection to completion
|
||||
is fully traceable. Every nanosecond in a reported latency
|
||||
is fully traceable. Every modeled latency contribution
|
||||
corresponds to exactly one of these events on exactly one node or
|
||||
edge---there is no slack in the budget.
|
||||
|
||||
@@ -202,6 +216,65 @@ collective engines hold the request for their service time before
|
||||
releasing it downstream. Each of these is attached to a specific node
|
||||
or edge in the graph; together they make up the entire latency budget.
|
||||
|
||||
Beyond data movement and execution latency, KernBench
|
||||
also models the control-plane cost required to issue work
|
||||
to accelerator engines. Since every DMA, GEMM, vector-
|
||||
math, and IPCQ operation is initiated through the
|
||||
\textsf{PE\_CPU} $\rightarrow$ \textsf{PE\_SCHED} path,
|
||||
command dispatch latency contributes to the end-to-end
|
||||
execution time of all kernels.
|
||||
|
||||
\paragraph{Command dispatch overhead model.} The cost incurred by
|
||||
the \textsf{PE\_CPU} when it dispatches a command to one of the
|
||||
accelerator engines is modelled structurally rather than with a
|
||||
per-operation calibration table. The \textsf{PE\_CPU} charges, per
|
||||
command,
|
||||
\[
|
||||
d_{\text{cmd}} = \textsf{FIXED} + b_{\text{logical}} \cdot R,
|
||||
\]
|
||||
This linear-in-size form reflects the serialization cost of writing a
|
||||
command descriptor into the scheduler queue: a fixed per-command
|
||||
bookkeeping cost plus a byte-wise descriptor-transfer cost.
|
||||
Concretely, $b_{\text{logical}}$ is the command's hardware-logical byte
|
||||
size, \textsf{FIXED} captures the fixed per-command cost (queue-tail
|
||||
update, completion registration) and $R$ captures the per-byte cost of
|
||||
serializing the command descriptor into the scheduler queue. This
|
||||
\textsf{FIXED} is distinct from Table~\ref{tab:hw}'s
|
||||
\SI{2}{\nano\second} / \SI{1}{\nano\second} \textsf{PE\_CPU} /
|
||||
\textsf{PE\_SCHED} fixed costs: the latter are per-component
|
||||
traversal overheads each command pays when it transits those nodes,
|
||||
whereas the 40-cycle \textsf{FIXED} term is the command-descriptor
|
||||
issue cost. The
|
||||
default anchoring (\textsf{FIXED} $= 40$ cycles, $R = 0.0625$
|
||||
cycles/byte, i.e.\ \SI{16}{\byte\per\cycle}, at \SI{1}{\giga\hertz})
|
||||
places a typical composite at roughly \SI{43}{\nano\second}, and a
|
||||
hard cap on a composite's descriptor size prevents the model from
|
||||
rewarding arbitrarily large fused commands beyond what real descriptor
|
||||
queues accept. In the configurations measured here, command issue is
|
||||
not the bottleneck---data movement is---so this term stays small
|
||||
relative to DMA and collective time.
|
||||
|
||||
\begin{table*}[t]
|
||||
\centering
|
||||
\caption{PE\,$\to$\,HBM DMA latency probe at varying hop distances
|
||||
(\SI{32}{\kibi\byte} transfer; output captured from \texttt{kernbench
|
||||
probe}). \emph{Util\%} is the achieved bandwidth as a fraction of the
|
||||
path's bottleneck-edge bandwidth.}
|
||||
\label{tab:probe-pe-dma}
|
||||
\small
|
||||
\begin{tabular}{@{}lrrr@{}}
|
||||
\toprule
|
||||
Case & Latency~(\si{\nano\second}) & Util\% (\,32\,KiB) & Util\% (\,1\,MiB) \\
|
||||
\midrule
|
||||
PE\,$\to$\,local HBM & 141.0 & 90.8 & 100.0 \\
|
||||
PE\,$\to$\,same-half HBM & 147.9 & 86.6 & ~99.9 \\
|
||||
PE\,$\to$\,cross-half HBM & 161.2 & 79.4 & ~99.8 \\
|
||||
PE\,$\to$\,cross-CUBE (best) & 330.5 & 77.5 & ~99.6 \\
|
||||
PE\,$\to$\,cross-CUBE (worst) & 677.1 & 37.8 & ~97.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
|
||||
\subsubsection{Congestion and contention modeling}
|
||||
\label{sec:congestion}
|
||||
|
||||
@@ -225,26 +298,6 @@ it reveals where the real bottlenecks form and which hardware levers
|
||||
actually relieve them---which is exactly the question the codesign
|
||||
work in this report turns on.
|
||||
|
||||
\paragraph{Command dispatch overhead model.} The cost incurred by
|
||||
the \textsf{PE\_CPU} when it dispatches a command to one of the
|
||||
accelerator engines is modelled structurally rather than with a
|
||||
per-operation calibration table. The \textsf{PE\_CPU} charges, per
|
||||
command,
|
||||
\[
|
||||
d_{\text{cmd}} = \textsf{FIXED} + b_{\text{logical}} \cdot R,
|
||||
\]
|
||||
where $b_{\text{logical}}$ is the command's hardware-logical byte size,
|
||||
\textsf{FIXED} captures the fixed per-command cost (queue-tail update,
|
||||
completion registration) and $R$ captures the per-byte cost of
|
||||
serializing the command descriptor into the scheduler queue. The
|
||||
default anchoring (\textsf{FIXED} $= 40$ cycles, $R = 0.0625$
|
||||
cycles/byte, i.e.\ \SI{16}{\byte\per\cycle}, at \SI{1}{\giga\hertz})
|
||||
places a typical composite at roughly \SI{43}{\nano\second}, and a
|
||||
hard cap on a composite's descriptor size prevents the model from
|
||||
rewarding arbitrarily large fused commands beyond what real descriptor
|
||||
queues accept. In the configurations measured here, command issue is
|
||||
not the bottleneck---data movement is---so this term stays small
|
||||
relative to DMA and collective time.
|
||||
|
||||
\subsection{Accuracy}
|
||||
\label{sec:accuracy}
|
||||
@@ -254,41 +307,29 @@ dominate kernel latency on this class of hardware: per-edge bandwidth
|
||||
occupancy and flit-level serialization, HBM pseudo-channel parallelism,
|
||||
and per-component switching overhead. Two independent cross-checks
|
||||
drawn from the experiments in this report confirm that this precision
|
||||
translates into physically reasonable kernel latencies. First, in the
|
||||
GEMM study (\S\ref{sec:gemm}), simulator-measured MAC efficiency
|
||||
tracks an analytic ideal-pipeline model within roughly
|
||||
translates into physically reasonable kernel latencies.
|
||||
|
||||
First, in the GEMM study (\S\ref{sec:gemm}), simulator-measured MAC
|
||||
efficiency tracks an analytic ideal-pipeline model within roughly
|
||||
\SIrange{10}{20}{\percent} across a wide range of tile counts; the
|
||||
residual gap is attributable to pipeline-fill and DMA effects the
|
||||
analytic model omits. Second, in the all-reduce study
|
||||
(\S\ref{sec:allreduce}, Fig.~\ref{fig:allreduce-cmp}), simulator
|
||||
latency for a 2D-torus over six devices follows the expected
|
||||
startup-plus-per-packet shape across the entire payload sweep---tight
|
||||
at small payloads where startup dominates, and within a single-digit
|
||||
multiplicative factor at the largest payloads, where the residual gap
|
||||
is explained by per-router switching the analytic shape elides. A
|
||||
single-device point from an external full-system simulator (FSIM) at
|
||||
the largest payload sits an order of magnitude above the KernBench
|
||||
multi-device torus, illustrating the well-known gap between an
|
||||
achievable-kernel number and a full end-to-end-stack number rather
|
||||
than a model error. The known simplifications---FIFO router
|
||||
arbitration (instead of round-robin), HBM scheduler without
|
||||
write-buffer reordering, no bank conflict, no refresh or thermal
|
||||
effects, and no upstream backpressure---are the price of a
|
||||
deterministic, inspectable model. These simplifications bound the
|
||||
absolute accuracy, but the agreement with both analytic models and the
|
||||
external full-system simulator data above indicates that KernBench is
|
||||
sufficiently accurate for evaluating the \emph{relative}
|
||||
hardware--software design trade-offs (tiling A vs.\ B, topology X
|
||||
vs.\ Y, with vs.\ without composite command, mesh vs.\ torus) that
|
||||
are the primary objective of this work.
|
||||
analytic model omits.
|
||||
|
||||
A third source of confidence comes from directly probing the
|
||||
simulator's per-traversal behaviour. Running \texttt{kernbench probe}
|
||||
on the modelled topology issues a sequence of PE-to-HBM DMA reads at
|
||||
progressively greater hop distances and reports the per-component
|
||||
overhead, per-edge serialization, and per-PC drain that the model
|
||||
charges (Table~\ref{tab:probe-pe-dma}). Three properties stand out.
|
||||
First, the reported latency increases \emph{monotonically} with hop
|
||||
Second, in the all-reduce study (\S\ref{sec:allreduce},
|
||||
Fig.~\ref{fig:allreduce-cmp}), simulator latency for a 2D-torus over
|
||||
six devices follows the expected startup-plus-per-packet shape across
|
||||
the entire payload sweep---tight at small payloads where startup
|
||||
dominates, and within a single-digit multiplicative factor at the
|
||||
largest payloads, where the residual gap is explained by per-router
|
||||
switching the analytic shape elides.
|
||||
|
||||
Third, the simulator's per-traversal behaviour can be probed
|
||||
directly. Running \texttt{kernbench probe} on the modelled topology
|
||||
issues a sequence of PE-to-HBM DMA reads at progressively greater
|
||||
hop distances and reports the per-component overhead, per-edge
|
||||
serialization, and per-PC drain that the model charges
|
||||
(Table~\ref{tab:probe-pe-dma}). Three properties stand out. First,
|
||||
the reported latency increases \emph{monotonically} with hop
|
||||
count---from \SI{141}{\nano\second} at the local HBM slice to
|
||||
\SI{677}{\nano\second} at the worst-case remote-CUBE slice---with the
|
||||
increment per added hop matching the per-router overhead and the
|
||||
@@ -308,26 +349,19 @@ self-consistent---a model error in any of them would surface as a
|
||||
non-monotonic or under-utilising curve here long before it polluted a
|
||||
kernel-level measurement.
|
||||
|
||||
\begin{table*}[t]
|
||||
\centering
|
||||
\caption{PE\,$\to$\,HBM DMA latency probe at varying hop distances
|
||||
(\SI{32}{\kibi\byte} transfer; output captured from \texttt{kernbench
|
||||
probe}). \emph{Util\%} is the achieved bandwidth as a fraction of the
|
||||
path's bottleneck-edge bandwidth.}
|
||||
\label{tab:probe-pe-dma}
|
||||
\small
|
||||
\begin{tabular}{@{}lrrr@{}}
|
||||
\toprule
|
||||
Case & Latency~(\si{\nano\second}) & Util\% (\,32\,KiB) & Util\% (\,1\,MiB) \\
|
||||
\midrule
|
||||
PE\,$\to$\,local HBM & 141.0 & 90.8 & 100.0 \\
|
||||
PE\,$\to$\,same-half HBM & 147.9 & 86.6 & ~99.9 \\
|
||||
PE\,$\to$\,cross-half HBM & 161.2 & 79.4 & ~99.8 \\
|
||||
PE\,$\to$\,cross-CUBE (best) & 330.5 & 77.5 & ~99.6 \\
|
||||
PE\,$\to$\,cross-CUBE (worst) & 677.1 & 37.8 & ~97.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
The known simplifications---FIFO router arbitration (instead of
|
||||
round-robin), HBM scheduler without write-buffer reordering, no bank
|
||||
conflict, no refresh or thermal effects, and no upstream
|
||||
backpressure---are the price of a deterministic, inspectable model.
|
||||
These simplifications bound the absolute accuracy, but the agreement
|
||||
with both analytic models and the probe's internal-consistency
|
||||
checks above indicates that KernBench is sufficiently accurate for
|
||||
evaluating the \emph{relative} hardware--software design trade-offs
|
||||
(tiling A vs.\ B, topology X vs.\ Y, with vs.\ without composite
|
||||
command, mesh vs.\ torus) that are the primary objective of this
|
||||
work.
|
||||
|
||||
|
||||
|
||||
\subsection{Modeled hardware configuration}
|
||||
\label{sec:hw}
|
||||
@@ -344,10 +378,22 @@ such that inference workloads can effectively saturate
|
||||
the available HBM bandwidth. The aggregate compute
|
||||
throughput is therefore balanced against memory-system
|
||||
bandwidth rather than being intentionally over- or
|
||||
under-provisioned. HBM bandwidth is distributed evenly
|
||||
across the PEs within a CUBE, with each PE responsible
|
||||
for servicing approximately one-eighth of the CUBE's
|
||||
memory bandwidth through its dedicated HBM channels.
|
||||
under-provisioned. Concretely, 8 PEs $\times$
|
||||
\SI{8}{\tera\flop\per\second} give roughly
|
||||
\SI{64}{\tera\flop\per\second} of f16 compute per CUBE
|
||||
against \SI{2048}{\giga\byte\per\second} of HBM bandwidth,
|
||||
which places the balanced point at about
|
||||
$\sim$31~FLOP/byte; inference decode kernels typically
|
||||
operate well below that, so HBM bandwidth---not compute---is
|
||||
the natural ceiling on this configuration. For reference, the
|
||||
GQA decode kernels evaluated in \S\ref{sec:gqa} operate at
|
||||
arithmetic intensity well below this balance point, so their
|
||||
ceiling is set by HBM and inter-PE traffic rather than by GEMM
|
||||
throughput. HBM bandwidth
|
||||
is distributed evenly across the PEs within a CUBE, with
|
||||
each PE responsible for servicing approximately one-eighth
|
||||
of the CUBE's memory bandwidth through its dedicated HBM
|
||||
channels.
|
||||
|
||||
The on-chip interconnect is configured using bandwidth
|
||||
and latency parameters representative of commercially
|
||||
@@ -371,7 +417,7 @@ payload sizes are introduced in their respective sections.
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\caption{Modeled hardware configuration (shared by all experiments).}
|
||||
\caption{Modeled hardware configuration}
|
||||
\label{tab:hw}
|
||||
\small
|
||||
\begin{tabular}{@{}ll@{}}
|
||||
@@ -389,11 +435,12 @@ GEMM engine peak & \SI{8}{\tera\flop\per\second} (f16) \\
|
||||
TCM (on-PE) & \SI{16}{\mega\byte}, \SI{512}{\giga\byte\per\second} R/W \\
|
||||
\quad kernel scratch & \SI{1}{\mega\byte} \\
|
||||
DMA engines & 1 read + 1 write \\
|
||||
CPU / scheduler overhead & \SI{2}{\nano\second} / \SI{1}{\nano\second} \\
|
||||
\textsf{PE\_CPU} fixed cost & \SI{2}{\nano\second} \\
|
||||
\textsf{PE\_SCHED} fixed cost & \SI{1}{\nano\second} \\
|
||||
\midrule
|
||||
\multicolumn{2}{@{}l}{\emph{Memory (per CUBE)}} \\
|
||||
HBM capacity & \SI{48}{\giga\byte} (8 slices) \\
|
||||
HBM aggregate BW & \SI{1024}{\giga\byte\per\second} \\
|
||||
HBM aggregate BW & \SI{2048}{\giga\byte\per\second} \\
|
||||
HBM pseudo-channels & 64 (8 per PE), \SI{32}{\giga\byte\per\second} each \\
|
||||
SRAM (shared) & \SI{32}{\mega\byte}, \SI{128}{\giga\byte\per\second} link \\
|
||||
HBM burst & \SI{256}{\byte} \\
|
||||
|
||||
Reference in New Issue
Block a user