ywkang/kernbench2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

paper(platform): WIP — §2 KernBench Platform mid-edit

Browse Source 
Mid-pass through §2. Captured progress so far:

- §2 intro previews the three new threads (graph view, DES engine,
  congestion) the rewritten latency section weaves together.
- §2.3 retitled to "Latency model: a graph traversed by events" and
  restructured into bold-led paragraphs:
  - The hardware as a graph (nodes = components, edges = links).
  - From graph to discrete-event simulation (node/edge events,
    deterministic ordering, correlation-ID trace).
  - Latency contributions (per-node fixed, per-edge size-aware,
    per-service occupancy).
  - Congestion: per-edge FIFO BW occupancy, HBM per-PC parallelism,
    component serial workers — the mechanisms that surface real
    bottlenecks instead of peak-BW roofline.
  - Control-plane cost model (FIXED + b·R) — unchanged.
  - Accuracy: extended with a second cross-check from the all-reduce
    study (torus vs. analytic startup-plus-per-packet model + FSIM
    external single-device reference). Existing GEMM analytic-vs-
    measured 10-20% check retained.
- §2.4 Hardware-configuration table unchanged.

Still TODO: cross-check accuracy numbers against a fresh
milestone-1h-gemm/ccl re-run; the current artifacts predate the
ADR-0064 Rev2 cost-model and IPCQ Phase-2 race fix and may need a
refresh before §2.3 final lock-in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yangwook Kang
2026-06-11 16:31:42 -07:00
parent d5267ec718
commit 9270d3435a
2 changed files with 132 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/report/1H-codesign-paper/build/main.pdf
BIN
View File
Binary file not shown.
docs/report/1H-codesign-paper/sections/02-platform.tex
+132 -75
View File
@@ -2,11 +2,12 @@
\label{sec:platform} \label{sec:platform}
All results in this report are produced on \emph{KernBench}, a All results in this report are produced on \emph{KernBench}, a
system-level, discrete-event simulator for LLM kernels running on system-level discrete-event simulator for LLM kernels running on AHBM.
SIP-based AI accelerators. This section explains why the platform exists, This section explains why the platform exists, how it executes a
how it executes a kernel, how it computes latency and how accurate that kernel, how its latency model works---specifically how the hardware is
number is, and the concrete hardware configuration used for every viewed as a graph, how that graph is driven by a discrete-event engine,
experiment that follows. and how congestion is captured---and the concrete hardware configuration
used for every experiment that follows.
\subsection{Why KernBench: source-level kernels without a software stack} \subsection{Why KernBench: source-level kernels without a software stack}
\label{sec:why} \label{sec:why}
@@ -21,16 +22,16 @@ well-written kernel be, and which hardware features actually make it
faster?} faster?}
KernBench is built to answer exactly that question. Kernels are written KernBench is built to answer exactly that question. Kernels are written
and executed at the \emph{source level}---as algorithmic descriptions in a and executed at the \emph{source level}---as algorithmic descriptions
small tile-oriented kernel API---with no dependency on a compiler or any in a small tile-oriented kernel API---with no dependency on a compiler
other software-stack layer. The simulator takes the kernel and a hardware or any other software-stack layer. The simulator takes the kernel and a
topology and reports the latency that the modeled hardware would deliver. hardware topology and reports the latency the modeled hardware would
This isolation is deliberate: it lets us study algorithm-level deliver. This isolation is deliberate: it lets us study algorithm-level
optimizations (how to tile a GEMM, how to schedule a collective, how to optimizations (how to tile a GEMM, how to schedule a collective, how to
fuse an attention kernel) and the hardware features that support them, fuse an attention kernel) and the hardware features that support them,
without the confound of compiler maturity or framework overhead. The cost without the confound of compiler maturity or framework overhead. The
is that KernBench numbers are \emph{not} E2E latencies; they are the cost is that KernBench numbers are \emph{not} E2E latencies; they are
achievable-kernel latencies that an ideal software stack would expose. the achievable-kernel latencies an ideal software stack would expose.
\subsection{Execution model} \subsection{Execution model}
\label{sec:exec} \label{sec:exec}
@@ -43,90 +44,146 @@ KernBench is layered along the flow of a request:
nothing about routing or interconnect. nothing about routing or interconnect.
\item The \textbf{simulation engine} schedules discrete events, routes \item The \textbf{simulation engine} schedules discrete events, routes
every request through the modeled graph, and tracks completion via every request through the modeled graph, and tracks completion via
correlation IDs. per-request correlation IDs.
\item The \textbf{components} are device-side nodes that model hardware \item The \textbf{components} are device-side nodes that model
behavior: the per-PE blocks (scheduler, DMA, GEMM and vector-math hardware behavior: the per-PE blocks (scheduler, DMA, GEMM and
engines, TCM, IPCQ), the NoC routers, the HBM controllers, and the vector-math engines, TCM, IPCQ), the NoC routers, the HBM
inter-chiplet links. controllers, and the inter-chiplet links.
\end{itemize} \end{itemize}
The topology is compiled once at configuration time into an authoritative Within a PE, work is expressed as \emph{composite commands}: a single
graph of components and links; it is never mutated during a run. Within a command carries an ordered pipeline of operations
PE, work is expressed as \emph{composite commands}: a single command
carries an ordered pipeline of operations
(\textsf{DMA\_READ} $\rightarrow$ \textsf{FETCH} $\rightarrow$ (\textsf{DMA\_READ} $\rightarrow$ \textsf{FETCH} $\rightarrow$
\textsf{GEMM}/\textsf{MATH} $\rightarrow$ \textsf{STORE} $\rightarrow$ \textsf{GEMM}/\textsf{MATH} $\rightarrow$ \textsf{STORE} $\rightarrow$
\textsf{DMA\_WRITE}) that the PE scheduler tiles and streams. This \textsf{DMA\_WRITE}) that the PE scheduler tiles and streams. This
composite mechanism is the substrate for the GEMM optimization composite mechanism is the substrate for the GEMM optimization
(\S\ref{sec:gemm}) and, combined with on-PE collectives, for fused (\S\ref{sec:gemm}) and, combined with on-PE collectives, for fused
attention (\S\ref{sec:gqa}). Data and timing are handled in two passes, so attention (\S\ref{sec:gqa}). Data and timing are handled in two passes,
that a kernel's numeric results and its latency are computed so that a kernel's numeric results and its latency are computed
consistently but independently. consistently but independently.
\subsection{Latency model and its accuracy} \subsection{Latency model: a graph traversed by events}
\label{sec:latency} \label{sec:latency}
KernBench obeys a set of golden invariants that keep its latency numbers \paragraph{The hardware as a graph.} KernBench views the modeled
physically meaningful. End-to-end latency is computed \emph{strictly by hardware as a directed graph. \emph{Nodes} are the components listed
explicit traversal} over modeled components and links: every routed above; \emph{edges} are the interconnect links between them, each
request incurs latency greater than zero, routing is deterministic, and carrying bandwidth (\si{\giga\byte\per\second}) and propagation
every valid request flow has explicit connectivity. There are no hidden (\si{\nano\second}) attributes. The topology is compiled once at
shortcuts, implicit waits, or magic delays---if a delay exists in the configuration time into this graph and is never mutated during a run.
result, it came from a scheduled event on a modeled component or link. Every routed request---a DMA, a remote read, an IPCQ message, a
kernel-launch command---is a \emph{traversal} of this graph from a
source node to a destination service, hopping through routers and
links along the way. There are no hidden shortcuts, implicit bypasses,
or magic paths: if a request reaches its destination, the path it took
is explicit in the graph, and the latency it incurred is the sum of
the per-node and per-edge costs paid along that path.
Latency accumulates from three kinds of contributions: per-node fixed \paragraph{From graph to discrete-event simulation.} The graph is
overheads (each component carries an \texttt{overhead\_ns}), per-link driven by a discrete-event engine. Two kinds of events advance
transfer time, and per-service occupancy. The interconnect is modeled at simulation time: \emph{node events} (component switching overhead,
fine granularity. Each directed link serializes traffic through a service completions such as an HBM channel commit or a GEMM tile
bandwidth-limited FIFO, so a busy link delays later flits. Payloads are finish) and \emph{edge events} (the flit-by-flit serialization of a
decomposed into fixed-size flits (default \SI{256}{\byte} bursts) that payload across a bandwidth-limited link). The engine maintains a
arrive at $\text{prop} + \text{flit\_bytes}/\text{bw}$ intervals, so link priority queue of pending events ordered by their scheduled time,
bandwidth throttles arrival rate rather than being applied as a lump sum. fires them one at a time, and treats ties under a deterministic
HBM is modeled with per-pseudo-channel parallelism: a stateless array of ordering policy so that the same kernel on the same topology always
channel-availability timestamps with address-based channel selection yields the same trace. Per-request correlation IDs are stamped at
captures bank-level concurrency. injection and carried through every hop, so the trace is recoverable
from injection to completion. Every nanosecond in a reported latency
traces back to exactly one of these events on exactly one node or
edge.
The cost of \emph{issuing} a command is modeled structurally rather than \paragraph{Latency contributions.} Three kinds of latency accumulate
with a per-operation calibration table. The PE control processor charges, per command, along a traversal: (i) \emph{per-node fixed overhead}---each component
carries a small switching cost (router decode, controller pickup,
scheduler handoff); (ii) \emph{per-edge transfer time}---each link's
payload is decomposed into fixed-size flits (default
\SI{256}{\byte}), and each flit arrives at
$\text{prop}+\text{flit\_bytes}/\text{bw}$ after the previous one,
giving wormhole semantics across multi-hop paths; and (iii)
\emph{per-service occupancy}---memory controllers, GEMM stages, and
collective engines hold the request for their service time before
releasing it downstream. Each of these is attached to a specific node
or edge in the graph; together they make up the entire latency budget.
\paragraph{Congestion: where bottlenecks emerge.} The simulator's
sharpness comes from how it models contention for those nodes and
edges. \emph{Every directed edge has a FIFO}: an arriving flit takes
its bandwidth-limited transfer time on top of whatever earlier flits
are still being served, so a busy link queues later traffic behind
earlier traffic rather than transferring everything at peak BW.
\emph{HBM is modeled with per-pseudo-channel parallelism}: a stateless
array of channel-availability timestamps with address-based channel
selection captures the bank-level concurrency that real HBM exposes,
so 64 channels per CUBE deliver real parallelism on uniform addresses
but a hot channel surfaces as the bottleneck on skewed ones.
\emph{Every component has a serial worker}: a router carrying two
heavy streams interleaves them at flit granularity in arrival order
rather than fanning out for free, so two concurrent collectives sharing
a link share its bandwidth, not double it. Without these mechanisms
the simulator would simply re-confirm the peak-BW roofline; with them,
it reveals where the real bottlenecks form and which hardware levers
actually relieve them---which is exactly the question the codesign
work in this report turns on.
\paragraph{Control-plane (issue) cost model.} The cost of \emph{issuing}
a command is modeled structurally rather than with a per-operation
calibration table. The PE control processor charges, per command,
\[ \[
d_{\text{cmd}} = \textsf{FIXED} + b_{\text{logical}} \cdot R, d_{\text{cmd}} = \textsf{FIXED} + b_{\text{logical}} \cdot R,
\] \]
where $b_{\text{logical}}$ is the command's hardware-logical byte size, where $b_{\text{logical}}$ is the command's hardware-logical byte size,
\textsf{FIXED} captures the fixed per-command cost (queue-tail \textsf{FIXED} captures the fixed per-command cost (queue-tail update,
update, completion registration) and $R$ captures the per-byte cost of completion registration) and $R$ captures the per-byte cost of
serializing the command descriptor into the scheduler queue. The default serializing the command descriptor into the scheduler queue. The
anchoring (\textsf{FIXED} $=40$ cycles, $R = 0.0625$ cycles/byte, i.e.\ default anchoring (\textsf{FIXED} $= 40$ cycles, $R = 0.0625$
\SI{16}{\byte\per\cycle}, at \SI{1}{\giga\hertz}) places a typical cycles/byte, i.e.\ \SI{16}{\byte\per\cycle}, at \SI{1}{\giga\hertz})
composite at roughly \SI{43}{\nano\second}, and a hard cap on a places a typical composite at roughly \SI{43}{\nano\second}, and a
composite's descriptor size prevents the model from rewarding hard cap on a composite's descriptor size prevents the model from
arbitrarily large fused commands beyond what real descriptor queues rewarding arbitrarily large fused commands beyond what real descriptor
accept. The dispatch cost is enabled and tuned per topology; in the queues accept. In the configurations measured here, command issue is
configurations measured here, command issue is not the bottleneck---data not the bottleneck---data movement is---so this term stays small
movement is---so this term stays small relative to DMA and collective relative to DMA and collective time.
time.
How accurate is all this? The model is precise about the things that \paragraph{Accuracy.} The model is precise about the effects that
dominate kernel latency on this class of hardware: link bandwidth dominate kernel latency on this class of hardware: per-edge bandwidth
occupancy and serialization, HBM channel parallelism, flit-level occupancy and flit-level serialization, HBM pseudo-channel parallelism,
streaming, and per-component switching overhead. The GEMM study and per-component switching overhead. Two independent cross-checks
(\S\ref{sec:gemm}) provides a direct check: the measured MAC efficiency drawn from the experiments in this report confirm that this precision
tracks the analytic (theoretical) efficiency within roughly translates into physically reasonable kernel latencies. First, in the
\SIrange{10}{20}{\percent} across a wide range of tile counts, with the GEMM study (\S\ref{sec:gemm}), simulator-measured MAC efficiency
gap attributable to pipeline fill and DMA effects that the analytic model tracks an analytic ideal-pipeline model within roughly
omits. The known simplifications---idealized arbitration, no thermal or \SIrange{10}{20}{\percent} across a wide range of tile counts; the
refresh effects, fixed burst granularity---are the price of a residual gap is attributable to pipeline-fill and DMA effects the
deterministic, inspectable model; they bound the absolute accuracy but do analytic model omits. Second, in the all-reduce study
not distort the \emph{relative} comparisons (tiling A vs.\ B, topology X (\S\ref{sec:allreduce}, Fig.~\ref{fig:allreduce-cmp}), simulator
vs.\ Y) that this report is built on. latency for a 2D-torus over six devices follows the expected
startup-plus-per-packet shape across the entire payload sweep---tight
at small payloads where startup dominates, and within a single-digit
multiplicative factor at the largest payloads, where the residual gap
is explained by per-router switching the analytic shape elides. A
single-device point from an external full-system simulator (FSIM) at
the largest payload sits an order of magnitude above the KernBench
multi-device torus, illustrating the well-known gap between an
achievable-kernel number and a full end-to-end-stack number rather
than a model error. The known simplifications---FIFO router
arbitration (instead of round-robin), HBM scheduler without
write-buffer reordering, no bank conflict, no refresh or thermal
effects, and no upstream backpressure---are the price of a
deterministic, inspectable model. They bound the absolute accuracy but
do not distort the \emph{relative} comparisons (tiling A vs.\ B,
topology X vs.\ Y, with vs.\ without composite command, mesh vs.\
torus) that this report is built on.
\subsection{Modeled hardware configuration} \subsection{Modeled hardware configuration}
\label{sec:hw} \label{sec:hw}
Table~\ref{tab:hw} summarizes the hardware configuration used for every Table~\ref{tab:hw} summarizes the hardware configuration used for
experiment in this report. It is read directly from the simulator's every experiment in this report. It is read directly from the
topology description; per-experiment workload parameters (matrix shapes, simulator's topology description; per-experiment workload parameters
collective sizes, sequence lengths) are stated in their respective (matrix shapes, collective sizes, sequence lengths) are stated in
sections rather than here. their respective sections rather than here.
\begin{table}[t] \begin{table}[t]
\centering \centering