diff --git a/docs/report/1H-codesign-paper/build/main.pdf b/docs/report/1H-codesign-paper/build/main.pdf index cae60c5..bab0298 100644 Binary files a/docs/report/1H-codesign-paper/build/main.pdf and b/docs/report/1H-codesign-paper/build/main.pdf differ diff --git a/docs/report/1H-codesign-paper/main.tex b/docs/report/1H-codesign-paper/main.tex index 8219248..2d77f29 100644 --- a/docs/report/1H-codesign-paper/main.tex +++ b/docs/report/1H-codesign-paper/main.tex @@ -13,6 +13,7 @@ \usepackage{hyperref} \hypersetup{colorlinks=true,linkcolor=blue!50!black,citecolor=blue!50!black,urlcolor=blue!50!black} \usepackage{caption} +\usepackage{subcaption} \captionsetup{font=small,labelfont=bf} \usepackage{microtype} \usepackage{tikz} diff --git a/docs/report/1H-codesign-paper/sections/02-platform.tex b/docs/report/1H-codesign-paper/sections/02-platform.tex index cee8d71..786df55 100644 --- a/docs/report/1H-codesign-paper/sections/02-platform.tex +++ b/docs/report/1H-codesign-paper/sections/02-platform.tex @@ -39,73 +39,73 @@ KernBench models AHBM as a hierarchy of SIPs, CUBEs, and processing elements (PEs). At the system level, multiple SIPs are connected through inter-package links, while each SIP contains a collection of CUBEs joined by an on-package interconnect -(Fig.~\ref{fig:sip-arch}). - -\begin{figure}[t] -\centering -\includegraphics[width=\linewidth]{sip_architecture.pdf} -\caption{Modeled hardware graph at the \emph{SIP} level (one -example configuration; specific parameters in \S\ref{sec:hw} / -Table~\ref{tab:hw}). The SIP holds a $4{\times}4$ mesh of CUBEs and -an IO chiplet, with each line a directed link labelled by its -physical distance and bandwidth.} -\label{fig:sip-arch} -\end{figure} +(Fig.~\ref{fig:hw-arch}\subref{fig:sip-arch}). Each CUBE contains eight PEs, shared SRAM, HBM controllers, an \textsf{M\_CPU} control processor, and an intra-CUBE router mesh -(Fig.~\ref{fig:cube-arch}). Together these components form the -execution substrate for all kernels evaluated in this report. This -organization reflects the memory-centric nature of AHBM: each PE is -paired with a dedicated slice of HBM bandwidth and local on-PE -storage (TCM), so compute lives next to the data it consumes rather -than fetching it through a far-away memory controller. The platform's -performance question is therefore not ``how many FLOPs can the chip -do'' but ``how well can a kernel keep each compute engine fed from -its locally-attached memory while moving the unavoidable traffic -between PEs efficiently.'' - -\begin{figure}[t] -\centering -\includegraphics[width=\linewidth]{cube_architecture.pdf} -\caption{Modeled hardware graph at the \emph{CUBE} level---a -zoom-in on one CUBE node from Fig.~\ref{fig:sip-arch}. The CUBE -holds 8 PEs (each with its own HBM channels), the \textsf{M\_CPU} -control processor, a shared SRAM, an intra-CUBE NoC router mesh, and -UCIe links to neighbouring CUBEs.} -\label{fig:cube-arch} -\end{figure} +(Fig.~\ref{fig:hw-arch}\subref{fig:cube-arch}). Together these +components form the execution substrate for all kernels evaluated in +this report. This organization reflects the memory-centric nature of +AHBM: each PE is paired with a dedicated slice of HBM bandwidth and +local on-PE storage (TCM), so compute lives next to the data it +consumes rather than fetching it through a far-away memory +controller. The platform's performance question is therefore not +``how many FLOPs can the chip do'' but ``how well can a kernel keep +each compute engine fed from its locally-attached memory while +moving the unavoidable traffic between PEs efficiently.'' Within a PE, commands are dispatched by \textsf{PE\_CPU} to \textsf{PE\_SCHED}, which routes work to specialized execution engines---DMA, FETCH/STORE, GEMM, vector-math, and IPCQ -(Fig.~\ref{fig:pe-arch}). Commands come in two flavours. \emph{Atomic} -commands target a single engine---a plain DMA read, a GEMM tile, a -vector-math op, an IPCQ send/receive---and are the natural unit for -short or special-purpose work; the \textsf{PE\_CPU} itself also runs -control-plane work directly. \emph{Composite} commands, by contrast, -carry an ordered pipeline of operations across multiple engines -(\textsf{DMA\_READ} $\rightarrow$ \textsf{FETCH} $\rightarrow$ -\textsf{GEMM}/\textsf{MATH} $\rightarrow$ \textsf{STORE} $\rightarrow$ -\textsf{DMA\_WRITE}) that the \textsf{PE\_SCHED} tiles and streams -without per-tile redispatch. The composite form is the substrate for -the GEMM optimization (\S\ref{sec:gemm}) and, combined with on-PE -collectives, for fused attention (\S\ref{sec:gqa}). +(Fig.~\ref{fig:hw-arch}\subref{fig:pe-arch}). Commands come in two +flavours. \emph{Atomic} commands target a single engine---a plain +DMA read, a GEMM tile, a vector-math op, an IPCQ send/receive---and +are the natural unit for short or special-purpose work; the +\textsf{PE\_CPU} itself also runs control-plane work directly. +\emph{Composite} commands, by contrast, carry an ordered pipeline of +operations across multiple engines (\textsf{DMA\_READ} $\rightarrow$ +\textsf{FETCH} $\rightarrow$ \textsf{GEMM}/\textsf{MATH} $\rightarrow$ +\textsf{STORE} $\rightarrow$ \textsf{DMA\_WRITE}) that the +\textsf{PE\_SCHED} tiles and streams without per-tile redispatch. The +composite form is the substrate for the GEMM optimization +(\S\ref{sec:gemm}) and, combined with on-PE collectives, for fused +attention (\S\ref{sec:gqa}). \begin{figure*}[t] \centering -\includegraphics[width=\linewidth]{pe_architecture.png} -\caption{Modeled PE architecture used in this report---one example -configuration whose specific parameters are listed in -\S\ref{sec:hw} (Table~\ref{tab:hw}); KernBench is not tied to this -particular decomposition and supports arbitrary PE block layouts -provided each component is given a port and bandwidth model. -\textsf{PE\_CPU} dispatches commands to the \textsf{PE\_SCHED}, which -routes tile-token streams through the \textsf{PE\_DMA}, -\textsf{PE\_FETCH\_STORE}, and \textsf{GEMM}/\textsf{MATH} engines -along on-chip links; the \textsf{PE\_IPCQ} provides the control plane -for on-device collective communication.} -\label{fig:pe-arch} +\begin{subfigure}[t]{0.46\textwidth} + \centering + \includegraphics[width=\linewidth,height=0.85\linewidth,keepaspectratio]{sip_architecture.pdf} + \caption{SIP level: $4{\times}4$ CUBE mesh + IO chiplet.} + \label{fig:sip-arch} +\end{subfigure}\hfill +\begin{subfigure}[t]{0.46\textwidth} + \centering + \includegraphics[width=\linewidth,height=0.85\linewidth,keepaspectratio]{cube_architecture.pdf} + \caption{CUBE level (zoom-in of one CUBE in (a)): 8 PEs, HBM + channels, M\_CPU, SRAM, NoC router mesh, UCIe links.} + \label{fig:cube-arch} +\end{subfigure} + +\vspace{0.6em} + +\begin{subfigure}[t]{\textwidth} + \centering + \includegraphics[width=\linewidth]{pe_architecture.png} + \caption{PE level (zoom-in of one PE in (b)): \textsf{PE\_CPU} + dispatches commands to \textsf{PE\_SCHED}, which routes tile-token + streams through \textsf{PE\_DMA}, \textsf{PE\_FETCH\_STORE}, and + \textsf{GEMM}/\textsf{MATH} engines; \textsf{PE\_IPCQ} is the + on-device collective control plane.} + \label{fig:pe-arch} +\end{subfigure} +\caption{Modeled hardware graph at the SIP, CUBE, and PE levels +(one example configuration; specific parameters in \S\ref{sec:hw} / +Table~\ref{tab:hw}). KernBench is not tied to this particular +arrangement: each box is a modeled component node, each line a +directed link with bandwidth and propagation attributes, and any +topology that respects those attributes is supported.} +\label{fig:hw-arch} \end{figure*} KernBench is layered along the flow of a request: @@ -171,9 +171,8 @@ There are no hidden shortcuts, implicit bypasses, or magic paths: if a request reaches its destination, the path it took is explicit in the graph, and the latency it incurred is the sum of the per-node and per-edge costs paid along that path. The same graph representation -applies recursively at every hierarchy level---system, SIP -(Fig.~\ref{fig:sip-arch}), CUBE (Fig.~\ref{fig:cube-arch}), and PE -(Fig.~\ref{fig:pe-arch}). +applies recursively at every hierarchy level---system, SIP, CUBE, and +PE (Fig.~\ref{fig:hw-arch}). \paragraph{From graph to discrete-event simulation.} The graph is driven by a discrete-event engine. Two kinds of events advance @@ -247,7 +246,10 @@ queues accept. In the configurations measured here, command issue is not the bottleneck---data movement is---so this term stays small relative to DMA and collective time. -\paragraph{Accuracy.} The model is precise about the effects that +\subsection{Accuracy} +\label{sec:accuracy} + +The model is precise about the effects that dominate kernel latency on this class of hardware: per-edge bandwidth occupancy and flit-level serialization, HBM pseudo-channel parallelism, and per-component switching overhead. Two independent cross-checks @@ -367,44 +369,33 @@ configuration. Workload-specific parameters such as matrix dimensions, sequence lengths, and collective payload sizes are introduced in their respective sections. -\begin{table}[t] +\begin{table*}[t] \centering \caption{Modeled hardware configuration (shared by all experiments).} \label{tab:hw} \small -\begin{tabular}{@{}ll@{}} +\begin{tabular}{@{}ll@{\hspace{3em}}ll@{}} \toprule -\textbf{Parameter} & \textbf{Value} \\ +\textbf{Parameter} & \textbf{Value} & \textbf{Parameter} & \textbf{Value} \\ \midrule -\multicolumn{2}{@{}l}{\emph{Hierarchy}} \\ -SIPs & 2 (1D ring) \\ -CUBEs per SIP & 16 ($4\times4$ mesh) \\ -PEs per CUBE & 8 (4 corners $\times$ 2) \\ -PEs total & 256 \\ +\multicolumn{2}{@{}l}{\emph{Hierarchy}} & \multicolumn{2}{@{}l}{\emph{Memory (per CUBE)}} \\ +SIPs & 2 (1D ring) & HBM capacity & \SI{48}{\giga\byte} (8 slices) \\ +CUBEs per SIP & 16 ($4\times4$ mesh) & HBM aggregate BW & \SI{1024}{\giga\byte\per\second} \\ +PEs per CUBE & 8 (4 corners $\times$ 2) & HBM pseudo-channels & 64 (8 per PE), \SI{32}{\giga\byte\per\second} each \\ +PEs total & 256 & SRAM (shared) & \SI{32}{\mega\byte}, \SI{128}{\giga\byte\per\second} link \\ + & & HBM burst & \SI{256}{\byte} \\ \midrule -\multicolumn{2}{@{}l}{\emph{Processing element (PE)}} \\ -GEMM engine peak & \SI{8}{\tera\flop\per\second} (f16) \\ -TCM (on-PE) & \SI{16}{\mega\byte}, \SI{512}{\giga\byte\per\second} R/W \\ -\quad kernel scratch & \SI{1}{\mega\byte} \\ -DMA engines & 1 read + 1 write \\ -CPU / scheduler overhead & \SI{2}{\nano\second} / \SI{1}{\nano\second} \\ +\multicolumn{2}{@{}l}{\emph{Processing element (PE)}} & \multicolumn{2}{@{}l}{\emph{Interconnect}} \\ +GEMM engine peak & \SI{8}{\tera\flop\per\second} (f16) & Intra-CUBE NoC link & \SI{256}{\giga\byte\per\second}, \SI{2}{\nano\second}/router \\ +TCM (on-PE) & \SI{16}{\mega\byte}, \SI{512}{\giga\byte\per\second} R/W & Inter-CUBE (UCIe PHY) & \SI{512}{\giga\byte\per\second}, \SI{8}{\nano\second}, XY routing \\ +\quad kernel scratch & \SI{1}{\mega\byte} & Inter-SIP (PCIe) & \SI{768}{\giga\byte\per\second} per endpoint \\ +DMA engines & 1 read + 1 write & & \\ +CPU / scheduler overhead & \SI{2}{\nano\second} / \SI{1}{\nano\second} & & \\ \midrule -\multicolumn{2}{@{}l}{\emph{Memory (per CUBE)}} \\ -HBM capacity & \SI{48}{\giga\byte} (8 slices) \\ -HBM aggregate BW & \SI{1024}{\giga\byte\per\second} \\ -HBM pseudo-channels & 64 (8 per PE), \SI{32}{\giga\byte\per\second} each \\ -SRAM (shared) & \SI{32}{\mega\byte}, \SI{128}{\giga\byte\per\second} link \\ -HBM burst & \SI{256}{\byte} \\ -\midrule -\multicolumn{2}{@{}l}{\emph{Interconnect}} \\ -Intra-CUBE NoC link & \SI{256}{\giga\byte\per\second}, \SI{2}{\nano\second}/router \\ -Inter-CUBE (UCIe PHY) & \SI{512}{\giga\byte\per\second}, \SI{8}{\nano\second}, XY routing \\ -Inter-SIP (PCIe) & \SI{768}{\giga\byte\per\second} per endpoint \\ -\midrule -\multicolumn{2}{@{}l}{\emph{Command-issue cost model (defaults)}} \\ -FIXED per command & 40 cycles \\ -per-byte rate $R$ & 0.0625 cycles/byte (\SI{16}{\byte\per\cycle}) \\ -composite size cap & \SI{1024}{\byte} \\ +\multicolumn{2}{@{}l}{\emph{Command-issue cost model (defaults)}} & & \\ +FIXED per command & 40 cycles & & \\ +per-byte rate $R$ & 0.0625 cycles/byte (\SI{16}{\byte\per\cycle}) & & \\ +composite size cap & \SI{1024}{\byte} & & \\ \bottomrule \end{tabular} -\end{table} +\end{table*} diff --git a/docs/report/1H-codesign-paper/sections/05-gqa.tex b/docs/report/1H-codesign-paper/sections/05-gqa.tex index 2a0710f..6181eed 100644 --- a/docs/report/1H-codesign-paper/sections/05-gqa.tex +++ b/docs/report/1H-codesign-paper/sections/05-gqa.tex @@ -41,70 +41,7 @@ that restructures the decode step into two stateful composites (a named \textsf{softmax\_merge} recipe) is designed but not yet wired into the measured path; results below reflect the implemented kernel only. -\subsection{Results} - -We measure four headline panels that vary the user count $C$ and the phase: -single- and multi-user prefill ($T_q=4$, $S_{kv}=16$), and single- and -multi-user decode ($P=8$ PEs, $S_{kv}=64$ and $128$), all at $d_{\text{head}}=64$ -and $G=8$. For each panel we harvest end-to-end latency -(max event end minus min event start, the same window convention as the -GEMM study) together with the per-engine busy time and the operation mix. -Figure~\ref{fig:gqa-lat} and Figure~\ref{fig:gqa-break} report the result; -the underlying numbers are in Table~\ref{tab:gqa}. - -\begin{table}[t] -\centering -\caption{Fused GQA per-panel latency and operation mix. Compute (GEMM, -MATH) is a tiny fraction of DMA occupancy; IPCQ copies grow with users and -PEs.} -\label{tab:gqa} -\small -\begin{tabular}{@{}lrrrr@{}} -\toprule -\textbf{Panel} & \textbf{Lat.\ (ns)} & \textbf{GEMM} & \textbf{IPCQ} & \textbf{DMA rd} \\ -\midrule -prefill C=1 & 445 & 2 & 0 & 3 \\ -prefill C=4 (Ring) & 4630 & 32 & 24 & 12 \\ -decode C=1, P=8 & 3632 & 16 & 21 & 24 \\ -decode C=4, P=8 & 6693 & 64 & 93 & 96 \\ -\bottomrule -\end{tabular} -\end{table} - -\begin{figure}[t] -\centering -\includegraphics[width=\linewidth]{gqa_latency_by_panel.png} -\caption{Fused GQA end-to-end latency. Latency grows from -\SI{445}{\nano\second} (single-user prefill) to \SI{6693}{\nano\second} -(four-user decode) as the KV history and the number of participating -devices grow.} -\label{fig:gqa-lat} -\end{figure} - -\begin{figure}[t] -\centering -\includegraphics[width=\linewidth]{gqa_op_engine_breakdown.png} -\caption{Where the work goes. Left: operation counts---GEMM and IPCQ-copy -volume both scale with users and PEs. Right: summed engine occupancy on a -log scale---the DMA engine dominates by two to three orders of magnitude -over the GEMM and MATH engines in every panel.} -\label{fig:gqa-break} -\end{figure} - -The dominant observation is in Figure~\ref{fig:gqa-break}: the compute -engines are almost idle. The GEMM engine accumulates only -\SIrange{2}{33}{\nano\second} of busy time across the panels and the -vector-math engine \SIrange{5}{688}{\nano\second}, while the DMA engine -accumulates \SIrange{72}{15920}{\nano\second}. Fused GQA, as modeled here, -is overwhelmingly data-movement bound. The operation mix shows why the -collective machinery matters: IPCQ-copy count rises from zero (single-user -prefill) to 93 (four-user decode) as the kernel reduces partial outputs -across more PEs and CUBEs, and DMA-read count rises in step as more KV -shards are streamed. The PE control-processor dispatch cost registered as -zero in this configuration---command issue is simply not on the critical -path when data movement is this dominant. - -\subsection{Long-context decode: parallelism strategies} +\subsection{Results: long-context decode and parallelism strategies} The four headline panels above stress the kernel at moderate context lengths. Long-context decode---the regime where KV cache size, not