ywkang
dd525bfcb7
New `/paper` slash-command skill that synthesizes ADR/SPEC content and live KernBench benchmark results into a sectioned LaTeX technical paper compiled to PDF with Tectonic (auto-installed). The skill negotiates a TOC, grounds every number in committed artifacts or fresh bench runs, and keeps report-only benches isolated. This commit also includes the first generated report: - docs/report/1H-codesign-paper/ — main.tex + per-section .tex, figures, toc.md contract, and the built 8-page main.pdf. Covers the platform (source-level kernels, latency model + accuracy, HW config from topology.yaml), GEMM via composite command, All-Reduce via PE_IPCQ, and fused GQA combining both, plus discussion/conclusion/2H future work. - scripts/paper/ — isolated report harnesses (not registered benches): paper_gqa_latency.py harvests per-panel GQA end-to-end latency + engine occupancy (the milestone only emitted op-counts); paper_plot_gqa.py renders the GQA figures. GEMM/All-Reduce reuse committed milestone figures/CSVs; GQA results are generated fresh. Honest flags retained: PE_CPU dispatch cost is 0 in this config, and the proposed two-composite softmax_merge decode is marked designed-not-measured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
172 lines
8.3 KiB
TeX
172 lines
8.3 KiB
TeX
\section{The KernBench Platform}
|
|
\label{sec:platform}
|
|
|
|
All results in this report are produced on \emph{KernBench}, a
|
|
system-level, discrete-event simulator for LLM kernels running on
|
|
SIP-based AI accelerators. This section explains why the platform exists,
|
|
how it executes a kernel, how it computes latency and how accurate that
|
|
number is, and the concrete hardware configuration used for every
|
|
experiment that follows.
|
|
|
|
\subsection{Why KernBench: source-level kernels without a software stack}
|
|
\label{sec:why}
|
|
|
|
In a production end-to-end (E2E) stack, kernel performance is entangled
|
|
with every layer above the hardware: the compiler's tiling and
|
|
scheduling choices, the framework's operator dispatch, the collective
|
|
library, and the runtime. Good E2E numbers require \emph{all} of those
|
|
layers to be co-optimized, which makes it hard to answer a narrower but
|
|
more fundamental question: \emph{given the hardware, how fast can a
|
|
well-written kernel be, and which hardware features actually make it
|
|
faster?}
|
|
|
|
KernBench is built to answer exactly that question. Kernels are written
|
|
and executed at the \emph{source level}---as algorithmic descriptions in a
|
|
small tile-oriented kernel API---with no dependency on a compiler or any
|
|
other software-stack layer. The simulator takes the kernel and a hardware
|
|
topology and reports the latency that the modeled hardware would deliver.
|
|
This isolation is deliberate: it lets us study algorithm-level
|
|
optimizations (how to tile a GEMM, how to schedule a collective, how to
|
|
fuse an attention kernel) and the hardware features that support them,
|
|
without the confound of compiler maturity or framework overhead. The cost
|
|
is that KernBench numbers are \emph{not} E2E latencies; they are the
|
|
achievable-kernel latencies that an ideal software stack would expose.
|
|
|
|
\subsection{Execution model}
|
|
\label{sec:exec}
|
|
|
|
KernBench is layered along the flow of a request:
|
|
|
|
\begin{itemize}
|
|
\item The \textbf{runtime API} is host-facing and
|
|
topology-agnostic---it deploys tensors and launches kernels but knows
|
|
nothing about routing or interconnect.
|
|
\item The \textbf{simulation engine} schedules discrete events, routes
|
|
every request through the modeled graph, and tracks completion via
|
|
correlation IDs.
|
|
\item The \textbf{components} are device-side nodes that model hardware
|
|
behavior: the per-PE blocks (scheduler, DMA, GEMM and vector-math
|
|
engines, TCM, IPCQ), the NoC routers, the HBM controllers, and the
|
|
inter-chiplet links.
|
|
\end{itemize}
|
|
|
|
The topology is compiled once at configuration time into an authoritative
|
|
graph of components and links; it is never mutated during a run. Within a
|
|
PE, work is expressed as \emph{composite commands}: a single command
|
|
carries an ordered pipeline of operations
|
|
(\textsf{DMA\_READ} $\rightarrow$ \textsf{FETCH} $\rightarrow$
|
|
\textsf{GEMM}/\textsf{MATH} $\rightarrow$ \textsf{STORE} $\rightarrow$
|
|
\textsf{DMA\_WRITE}) that the PE scheduler tiles and streams. This
|
|
composite mechanism is the substrate for the GEMM optimization
|
|
(\S\ref{sec:gemm}) and, combined with on-PE collectives, for fused
|
|
attention (\S\ref{sec:gqa}). Data and timing are handled in two passes, so
|
|
that a kernel's numeric results and its latency are computed
|
|
consistently but independently.
|
|
|
|
\subsection{Latency model and its accuracy}
|
|
\label{sec:latency}
|
|
|
|
KernBench obeys a set of golden invariants that keep its latency numbers
|
|
physically meaningful. End-to-end latency is computed \emph{strictly by
|
|
explicit traversal} over modeled components and links: every routed
|
|
request incurs latency greater than zero, routing is deterministic, and
|
|
every valid request flow has explicit connectivity. There are no hidden
|
|
shortcuts, implicit waits, or magic delays---if a delay exists in the
|
|
result, it came from a scheduled event on a modeled component or link.
|
|
|
|
Latency accumulates from three kinds of contributions: per-node fixed
|
|
overheads (each component carries an \texttt{overhead\_ns}), per-link
|
|
transfer time, and per-service occupancy. The interconnect is modeled at
|
|
fine granularity. Each directed link serializes traffic through a
|
|
bandwidth-limited FIFO, so a busy link delays later flits. Payloads are
|
|
decomposed into fixed-size flits (default \SI{256}{\byte} bursts) that
|
|
arrive at $\text{prop} + \text{flit\_bytes}/\text{bw}$ intervals, so link
|
|
bandwidth throttles arrival rate rather than being applied as a lump sum.
|
|
HBM is modeled with per-pseudo-channel parallelism: a stateless array of
|
|
channel-availability timestamps with address-based channel selection
|
|
captures bank-level concurrency.
|
|
|
|
The cost of \emph{issuing} a command is modeled structurally rather than
|
|
with a per-operation calibration table. The PE control processor charges, per command,
|
|
\[
|
|
d_{\text{cmd}} = \textsf{FIXED} + b_{\text{logical}} \cdot R,
|
|
\]
|
|
where $b_{\text{logical}}$ is the command's hardware-logical byte size,
|
|
\textsf{FIXED} captures the fixed per-command cost (queue-tail
|
|
update, completion registration) and $R$ captures the per-byte cost of
|
|
serializing the command descriptor into the scheduler queue. The default
|
|
anchoring (\textsf{FIXED} $=40$ cycles, $R = 0.0625$ cycles/byte, i.e.\
|
|
\SI{16}{\byte\per\cycle}, at \SI{1}{\giga\hertz}) places a typical
|
|
composite at roughly \SI{43}{\nano\second}, and a hard cap on a
|
|
composite's descriptor size prevents the model from rewarding
|
|
arbitrarily large fused commands beyond what real descriptor queues
|
|
accept. The dispatch cost is enabled and tuned per topology; in the
|
|
configurations measured here, command issue is not the bottleneck---data
|
|
movement is---so this term stays small relative to DMA and collective
|
|
time.
|
|
|
|
How accurate is all this? The model is precise about the things that
|
|
dominate kernel latency on this class of hardware: link bandwidth
|
|
occupancy and serialization, HBM channel parallelism, flit-level
|
|
streaming, and per-component switching overhead. The GEMM study
|
|
(\S\ref{sec:gemm}) provides a direct check: the measured MAC efficiency
|
|
tracks the analytic (theoretical) efficiency within roughly
|
|
\SIrange{10}{20}{\percent} across a wide range of tile counts, with the
|
|
gap attributable to pipeline fill and DMA effects that the analytic model
|
|
omits. The known simplifications---idealized arbitration, no thermal or
|
|
refresh effects, fixed burst granularity---are the price of a
|
|
deterministic, inspectable model; they bound the absolute accuracy but do
|
|
not distort the \emph{relative} comparisons (tiling A vs.\ B, topology X
|
|
vs.\ Y) that this report is built on.
|
|
|
|
\subsection{Modeled hardware configuration}
|
|
\label{sec:hw}
|
|
|
|
Table~\ref{tab:hw} summarizes the hardware configuration used for every
|
|
experiment in this report. It is read directly from the simulator's
|
|
topology description; per-experiment workload parameters (matrix shapes,
|
|
collective sizes, sequence lengths) are stated in their respective
|
|
sections rather than here.
|
|
|
|
\begin{table}[t]
|
|
\centering
|
|
\caption{Modeled hardware configuration (shared by all experiments).}
|
|
\label{tab:hw}
|
|
\small
|
|
\begin{tabular}{@{}ll@{}}
|
|
\toprule
|
|
\textbf{Parameter} & \textbf{Value} \\
|
|
\midrule
|
|
\multicolumn{2}{@{}l}{\emph{Hierarchy}} \\
|
|
SIPs & 2 (1D ring) \\
|
|
CUBEs per SIP & 16 ($4\times4$ mesh) \\
|
|
PEs per CUBE & 8 (4 corners $\times$ 2) \\
|
|
PEs total & 256 \\
|
|
\midrule
|
|
\multicolumn{2}{@{}l}{\emph{Processing element (PE)}} \\
|
|
GEMM engine peak & \SI{8}{\tera\flop\per\second} (f16) \\
|
|
TCM (on-PE) & \SI{16}{\mega\byte}, \SI{512}{\giga\byte\per\second} R/W \\
|
|
\quad kernel scratch & \SI{1}{\mega\byte} \\
|
|
DMA engines & 1 read + 1 write \\
|
|
CPU / scheduler overhead & \SI{2}{\nano\second} / \SI{1}{\nano\second} \\
|
|
\midrule
|
|
\multicolumn{2}{@{}l}{\emph{Memory (per CUBE)}} \\
|
|
HBM capacity & \SI{48}{\giga\byte} (8 slices) \\
|
|
HBM aggregate BW & \SI{1024}{\giga\byte\per\second} \\
|
|
HBM pseudo-channels & 64 (8 per PE), \SI{32}{\giga\byte\per\second} each \\
|
|
SRAM (shared) & \SI{32}{\mega\byte}, \SI{128}{\giga\byte\per\second} link \\
|
|
HBM burst & \SI{256}{\byte} \\
|
|
\midrule
|
|
\multicolumn{2}{@{}l}{\emph{Interconnect}} \\
|
|
Intra-CUBE NoC link & \SI{256}{\giga\byte\per\second}, \SI{2}{\nano\second}/router \\
|
|
Inter-CUBE (UCIe PHY) & \SI{512}{\giga\byte\per\second}, \SI{8}{\nano\second}, XY routing \\
|
|
Inter-SIP (PCIe) & \SI{768}{\giga\byte\per\second} per endpoint \\
|
|
\midrule
|
|
\multicolumn{2}{@{}l}{\emph{Command-issue cost model (defaults)}} \\
|
|
FIXED per command & 40 cycles \\
|
|
per-byte rate $R$ & 0.0625 cycles/byte (\SI{16}{\byte\per\cycle}) \\
|
|
composite size cap & \SI{1024}{\byte} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|