ywkang/kernbench2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

paper(platform): edit-pass through §2 KernBench Platform done

Browse Source 
- Reordered §2 subsections to follow the SIP → CUBE → PE → graph
  flow: Why KernBench → Device and execution model → Latency model
  → Modeled hardware configuration. Readers now meet the device
  hierarchy before the graph abstraction that re-uses it.
- §2.2 Device and execution model: starts with the SIP/CUBE/PE
  hierarchy paragraphs (each anchoring fig:sip-arch, fig:cube-arch,
  fig:pe-arch); then the runtime-API/sim-engine/components bullet
  list; then the atomic-vs-composite command distinction (corrects
  the prior over-narrow framing that read every PE command as
  composite -- atomic single-engine commands exist too, and PE_CPU
  itself runs control-plane work directly).
- §2.3 Latency model: opens with the four-contribution decomposition
  (per-node overhead, per-edge transmission, drain, queuing delay)
  and the latency_model schematic; retains existing The hardware as
  a graph / From graph to DES / Latency contributions / Congestion
  / Control-plane cost model / Accuracy paragraphs. Accuracy
  paragraph now closes on KernBench's sufficiency for *relative*
  HW/SW design trade-offs given analytic + external-simulator
  agreement.
- New figures and assets:
  - figures/sip_architecture.pdf  (SIP-level graph view)
  - figures/cube_architecture.pdf (CUBE-level zoom-in)
  - figures/latency_model.png      (conceptual latency-model
                                   schematic with per-node /
                                   per-edge / drain / queuing-delay
                                   colour coding)
  - figures/pe_architecture.png    (carried over)
- Source-of-truth generator for the latency schematic:
  scripts/paper/paper_latency_model_diagram.py (a report-only
  harness under scripts/paper/ per the /paper isolation rule).
- main.tex preamble: \usepackage{tikz} added (kept from prior
  sequence-diagram draft -- harmless now that the latency model is
  a PNG; left in to keep paragraph numbering stable).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yangwook Kang
2026-06-15 17:17:05 -07:00
parent 756680f4e6
commit 7f437a20bd
8 changed files with 679 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/report/1H-codesign-paper/sections/02-platform.tex
+142 -51
View File
@@ -3,11 +3,10 @@
All results in this report are produced on \emph{KernBench}, a
system-level discrete-event simulator for LLM kernels running on AHBM.
This section explains why the platform exists, how it executes a
kernel, how its latency model works---specifically how the hardware is
viewed as a graph, how that graph is driven by a discrete-event engine,
and how congestion is captured---and the concrete hardware configuration
used for every experiment that follows.
This section explains why the platform exists, the device and
execution model it presents to a kernel writer, how its latency model
turns that execution into a number, and the concrete hardware
configuration used for every experiment that follows.
\subsection{Why KernBench: source-level kernels without a software stack}
\label{sec:why}
@@ -33,51 +32,140 @@ without the confound of compiler maturity or framework overhead. The
cost is that KernBench numbers are \emph{not} E2E latencies; they are
the achievable-kernel latencies an ideal software stack would expose.
\subsection{Execution model}
\subsection{Device and execution model}
\label{sec:exec}
KernBench models AHBM as a hierarchy of SIPs, CUBEs, and processing
elements (PEs). At the system level, multiple SIPs are connected
through inter-package links, while each SIP contains a collection of
CUBEs joined by an on-package interconnect
(Fig.~\ref{fig:sip-arch}).
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{sip_architecture.pdf}
\caption{Modeled hardware graph at the \emph{SIP} level (one
example configuration; specific parameters in \S\ref{sec:hw} /
Table~\ref{tab:hw}). The SIP holds a $4{\times}4$ mesh of CUBEs and
an IO chiplet, with each line a directed link labelled by its
physical distance and bandwidth.}
\label{fig:sip-arch}
\end{figure}
Each CUBE contains eight PEs, shared SRAM, HBM controllers, an
\textsf{M\_CPU} control processor, and an intra-CUBE router mesh
(Fig.~\ref{fig:cube-arch}). Together these components form the
execution substrate for all kernels evaluated in this report.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{cube_architecture.pdf}
\caption{Modeled hardware graph at the \emph{CUBE} level---a
zoom-in on one CUBE node from Fig.~\ref{fig:sip-arch}. The CUBE
holds 8 PEs (each with its own HBM channels), the \textsf{M\_CPU}
control processor, a shared SRAM, an intra-CUBE NoC router mesh, and
UCIe links to neighbouring CUBEs.}
\label{fig:cube-arch}
\end{figure}
Within a PE, commands are dispatched by \textsf{PE\_CPU} to
\textsf{PE\_SCHED}, which routes work to specialized execution
engines---DMA, FETCH/STORE, GEMM, vector-math, and IPCQ
(Fig.~\ref{fig:pe-arch}). Commands come in two flavours. \emph{Atomic}
commands target a single engine---a plain DMA read, a GEMM tile, a
vector-math op, an IPCQ send/receive---and are the natural unit for
short or special-purpose work; the \textsf{PE\_CPU} itself also runs
control-plane work directly. \emph{Composite} commands, by contrast,
carry an ordered pipeline of operations across multiple engines
(\textsf{DMA\_READ} $\rightarrow$ \textsf{FETCH} $\rightarrow$
\textsf{GEMM}/\textsf{MATH} $\rightarrow$ \textsf{STORE} $\rightarrow$
\textsf{DMA\_WRITE}) that the \textsf{PE\_SCHED} tiles and streams
without per-tile redispatch. The composite form is the substrate for
the GEMM optimization (\S\ref{sec:gemm}) and, combined with on-PE
collectives, for fused attention (\S\ref{sec:gqa}).
\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{pe_architecture.png}
\caption{Modeled PE architecture used in this report---one example
configuration whose specific parameters are listed in
\S\ref{sec:hw} (Table~\ref{tab:hw}); KernBench is not tied to this
particular decomposition and supports arbitrary PE block layouts
provided each component is given a port and bandwidth model.
\textsf{PE\_CPU} dispatches commands to the \textsf{PE\_SCHED}, which
routes tile-token streams through the \textsf{PE\_DMA},
\textsf{PE\_FETCH\_STORE}, and \textsf{GEMM}/\textsf{MATH} engines
along on-chip links; the \textsf{PE\_IPCQ} provides the control plane
for on-device collective communication.}
\label{fig:pe-arch}
\end{figure*}
KernBench is layered along the flow of a request:
\begin{itemize}
\item The \textbf{runtime API} is host-facing and
topology-agnostic---it deploys tensors and launches kernels but knows
nothing about routing or interconnect.
\item The \textbf{simulation engine} schedules discrete events, routes
every request through the modeled graph, and tracks completion via
per-request correlation IDs.
\item The \textbf{simulation engine} schedules discrete events and
routes every request through the modeled graph.
\item The \textbf{components} are device-side nodes that model
hardware behavior: the per-PE blocks (scheduler, DMA, GEMM and
hardware behaviour: the per-PE blocks (scheduler, DMA, GEMM and
vector-math engines, TCM, IPCQ), the NoC routers, the HBM
controllers, and the inter-chiplet links.
\end{itemize}
Within a PE, work is expressed as \emph{composite commands}: a single
command carries an ordered pipeline of operations
(\textsf{DMA\_READ} $\rightarrow$ \textsf{FETCH} $\rightarrow$
\textsf{GEMM}/\textsf{MATH} $\rightarrow$ \textsf{STORE} $\rightarrow$
\textsf{DMA\_WRITE}) that the PE scheduler tiles and streams. This
composite mechanism is the substrate for the GEMM optimization
(\S\ref{sec:gemm}) and, combined with on-PE collectives, for fused
attention (\S\ref{sec:gqa}). Data and timing are handled in two passes,
so that a kernel's numeric results and its latency are computed
consistently but independently.
Data and timing are handled in two passes, so that a kernel's numeric
results and its latency are computed consistently but independently.
\subsection{Latency model: a graph traversed by events}
\subsection{Latency model: graph traversal and contention}
\label{sec:latency}
\paragraph{The hardware as a graph.} KernBench views the modeled
hardware as a directed graph. \emph{Nodes} are the components listed
above; \emph{edges} are the interconnect links between them, each
carrying bandwidth (\si{\giga\byte\per\second}) and propagation
(\si{\nano\second}) attributes. The topology is compiled once at
The modeled hardware hierarchy described above is represented
internally as a directed graph. Nodes correspond to hardware
components---PEs, routers, memory controllers, SRAM blocks, IO
chiplets---while edges represent communication links with associated
bandwidth (\si{\giga\byte\per\second}) and propagation
(\si{\nano\second}) attributes. Every operation in KernBench---DMA
transfers, remote memory accesses, collective communication, command
dispatch---is modelled as a traversal through this graph. End-to-end
latency is decomposed into four contributions accumulated along the
traversal path (Fig.~\ref{fig:latency-model}): \textbf{per-node
overhead} at each component, \textbf{per-edge transmission} on each
wire, \textbf{drain} (per-flit service occupancy) at the destination,
and \textbf{queuing delay} at the shared FIFOs that the wire and the
destination share between concurrent transactions.
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{latency_model.png}
\caption{Conceptual schematic of the latency model. Two source nodes
(Requester A/B) inject flits through a chain of routers into a
destination node; on the shared edge between routers, flits from the
two transactions are interleaved flit-by-flit by the wire's FIFO
arrival order. End-to-end latency is the sum of four contributions:
\textbf{per-node overhead} (the switch's fixed processing cost,
shown in light yellow---the same colour as the destination's
processing-logic block), \textbf{per-edge transmission}
($\textit{flit\_size}/\textit{BW}$ on each wire),
\textbf{drain} (per-flit service occupancy at the destination's
channel), and \textbf{queuing delay} (waiting in a FIFO when a
shared resource is busy). The places where queuing actually
accumulates are highlighted in green: the router output queue and
the destination's input queue. This is the model, not a
measurement; specific bandwidths and overheads are listed in
\S\ref{sec:hw} (Table~\ref{tab:hw}).}
\label{fig:latency-model}
\end{figure*}
\paragraph{The hardware as a graph.} The topology is compiled once at
configuration time into this graph and is never mutated during a run.
Every routed request---a DMA, a remote read, an IPCQ message, a
kernel-launch command---is a \emph{traversal} of this graph from a
source node to a destination service, hopping through routers and
links along the way. There are no hidden shortcuts, implicit bypasses,
or magic paths: if a request reaches its destination, the path it took
is explicit in the graph, and the latency it incurred is the sum of
the per-node and per-edge costs paid along that path.
There are no hidden shortcuts, implicit bypasses, or magic paths: if a
request reaches its destination, the path it took is explicit in the
graph, and the latency it incurred is the sum of the per-node and
per-edge costs paid along that path. The same graph representation
applies recursively at every hierarchy level---system, SIP
(Fig.~\ref{fig:sip-arch}), CUBE (Fig.~\ref{fig:cube-arch}), and PE
(Fig.~\ref{fig:pe-arch}).
\paragraph{From graph to discrete-event simulation.} The graph is
driven by a discrete-event engine. Two kinds of events advance
@@ -85,14 +173,14 @@ simulation time: \emph{node events} (component switching overhead,
service completions such as an HBM channel commit or a GEMM tile
finish) and \emph{edge events} (the flit-by-flit serialization of a
payload across a bandwidth-limited link). The engine maintains a
priority queue of pending events ordered by their scheduled time,
fires them one at a time, and treats ties under a deterministic
ordering policy so that the same kernel on the same topology always
yields the same trace. Per-request correlation IDs are stamped at
injection and carried through every hop, so the trace is recoverable
from injection to completion. Every nanosecond in a reported latency
traces back to exactly one of these events on exactly one node or
edge.
priority queue of pending events ordered by their scheduled time and
fires them one at a time, with ties broken under a deterministic
policy so that the same kernel on the same topology always yields the
same trace. Per-request correlation IDs are stamped at injection and
carried through every hop, so the path from injection to completion
is fully traceable. Every nanosecond in a reported latency
corresponds to exactly one of these events on exactly one node or
edge---there is no slack in the budget.
\paragraph{Latency contributions.} Three kinds of latency accumulate
along a traversal: (i) \emph{per-node fixed overhead}---each component
@@ -113,11 +201,11 @@ edges. \emph{Every directed edge has a FIFO}: an arriving flit takes
its bandwidth-limited transfer time on top of whatever earlier flits
are still being served, so a busy link queues later traffic behind
earlier traffic rather than transferring everything at peak BW.
\emph{HBM is modeled with per-pseudo-channel parallelism}: a stateless
array of channel-availability timestamps with address-based channel
selection captures the bank-level concurrency that real HBM exposes,
so 64 channels per CUBE deliver real parallelism on uniform addresses
but a hot channel surfaces as the bottleneck on skewed ones.
\emph{HBM is modelled with per-pseudo-channel parallelism}: a
stateless array of channel-availability timestamps with address-based
channel selection captures the bank-level concurrency that real HBM
exposes, so 64 channels per CUBE deliver real parallelism on uniform
addresses but a hot channel surfaces as the bottleneck on skewed ones.
\emph{Every component has a serial worker}: a router carrying two
heavy streams interleaves them at flit granularity in arrival order
rather than fanning out for free, so two concurrent collectives sharing
@@ -128,7 +216,7 @@ actually relieve them---which is exactly the question the codesign
work in this report turns on.
\paragraph{Control-plane (issue) cost model.} The cost of \emph{issuing}
a command is modeled structurally rather than with a per-operation
a command is modelled structurally rather than with a per-operation
calibration table. The PE control processor charges, per command,
\[
d_{\text{cmd}} = \textsf{FIXED} + b_{\text{logical}} \cdot R,
@@ -171,10 +259,13 @@ than a model error. The known simplifications---FIFO router
arbitration (instead of round-robin), HBM scheduler without
write-buffer reordering, no bank conflict, no refresh or thermal
effects, and no upstream backpressure---are the price of a
deterministic, inspectable model. They bound the absolute accuracy but
do not distort the \emph{relative} comparisons (tiling A vs.\ B,
topology X vs.\ Y, with vs.\ without composite command, mesh vs.\
torus) that this report is built on.
deterministic, inspectable model. These simplifications bound the
absolute accuracy, but the agreement with both analytic models and the
external full-system simulator data above indicates that KernBench is
sufficiently accurate for evaluating the \emph{relative}
hardware--software design trade-offs (tiling A vs.\ B, topology X
vs.\ Y, with vs.\ without composite command, mesh vs.\ torus) that
are the primary objective of this work.
\subsection{Modeled hardware configuration}
\label{sec:hw}