kernbench2/docs/report/1H-codesign-paper/sections/04-allreduce.tex

\section{PE\_IPCQ and Collective Communication}
\label{sec:allreduce}

\subsection{Why it is needed}

Distributing a transformer across devices turns every tensor-parallel
layer into a collective: partial results computed on different PEs, CUBEs,
and SIPs must be summed and redistributed with an all-reduce. If that
collective is handled by the host or by a generic DMA path, three problems
appear. The reduction traffic competes with the kernel's own compute DMA
on the same links, causing head-of-line blocking; there is no efficient
peer-to-peer ring primitive, so data takes extra hops; and the ordering is
hard to make deterministic. The hardware question is how to perform
collectives \emph{on the device}, overlapped with compute and reproducible
run-to-run.

\subsection{Design}

KernBench models a dedicated per-PE collective engine, \textbf{PE\_IPCQ}
(inter-PE communication queue). It is a control-plane block: it owns the
ring-buffer address arithmetic, head/tail pointers, peer-pointer caches,
backpressure, and the four-direction (N/S/E/W) neighbor map, with eight
ring buffers per PE (four directions $\times$ \{tx, rx\}). Crucially,
PE\_IPCQ does \emph{not} move data itself---it delegates the actual
transfer to PE\_DMA, keeping a clean control/data split. To stop
collective traffic from blocking compute, PE\_DMA is extended into a
two-channel virtual-channel model: \texttt{vc\_compute} carries tile
load/store for GEMM and vector math, \texttt{vc\_comm} carries IPCQ sends,
each with an independent state machine. The same physical link is shared
but progresses in chunks (\SI{256}{\byte}), so a large GEMM DMA does not
lock the link end-to-end against a pending reduction. On top of this
substrate the collective runs a hierarchical local-reduce / global
all-reduce-broadcast schedule across whatever inter-device topology the
configuration specifies.

\subsection{Results}

Following the milestone-evaluation convention, the collective sweep builds
its own six-device (six-SIP, $2\times3$) configurations---distinct from
the two-SIP default of Table~\ref{tab:hw}---and measures all-reduce
latency as a function of payload size for three inter-device topologies:
a 1D ring, a 2D mesh (no wrap), and a 2D torus. Table~\ref{tab:allreduce}
and Figure~\ref{fig:allreduce-cmp} report the result.

\begin{table}[t]
\centering
\caption{All-reduce latency (ns) across six devices, by topology and
per-PE payload. Lower is better; the torus wins at every size.}
\label{tab:allreduce}
\small
\begin{tabular}{@{}rrrr@{}}
\toprule
\textbf{Bytes/PE} & \textbf{2D mesh} & \textbf{Ring 1D} & \textbf{2D torus} \\
\midrule
256     & 2667  & 2365  & 1701 \\
4{,}096  & 4450  & 4082  & 3038 \\
16{,}384 & 8900  & 8217  & 6403 \\
65{,}536 & 26705 & 24766 & 19865 \\
98{,}304 & 38574 & 35798 & 28840 \\
\bottomrule
\end{tabular}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{allreduce_comparison.png}
\caption{All-reduce latency vs.\ per-PE payload for the three topologies,
against the analytic torus model and an external full-system simulator
(FSIM) reference. The measured torus tracks the analytic curve within a
small constant factor; the FSIM single-device point
(\SI{366}{\micro\second}) sits an order of magnitude above the
KernBench algorithmic latency, illustrating the difference between an
achievable-kernel number and a full end-to-end-stack number.}
\label{fig:allreduce-cmp}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{allreduce_buffer_kind.png}
\caption{Effect of IPCQ staging-buffer placement (2D torus). At
\SI{64}{\kibi\byte}/PE, TCM staging (\SI{19865}{\nano\second}) beats HBM
(\SI{23081}{\nano\second}) by \textasciitilde\SI{14}{\percent} and SRAM
(\SI{32201}{\nano\second}) by \textasciitilde\SI{38}{\percent}; at small
payloads the three are indistinguishable.}
\label{fig:allreduce-buf}
\end{figure}

Three findings follow. First, topology matters and the effect grows with
size: at \SI{96}{\kilo\byte}/PE the 2D torus is \SI{25}{\percent} faster
than the mesh and \SI{19}{\percent} faster than the ring, because its
wrap-around links shorten the worst-case reduction path. Second, the
measured curves grow smoothly with payload and stay within a small
constant factor of the analytic torus model, with the gap reflecting real
link serialization that the analytic model idealizes away. Third, where
the IPCQ staging buffer lives is a first-order knob: keeping it in on-PE
TCM is materially faster than HBM or shared SRAM at large payloads
(Figure~\ref{fig:allreduce-buf}).

\subsection{Analysis and meaning}

PE\_IPCQ turns the collective from an off-device, contention-prone
operation into an on-device primitive whose latency tracks the
interconnect's physical limits. The control/data split keeps the engine
small while reusing PE\_DMA for movement, and the virtual-channel split is
what lets a reduction proceed without stalling the compute DMA that feeds
the very GEMMs producing the partials---the same property the fused
attention kernel relies on when it interleaves KV reduction with score
computation (\S\ref{sec:gqa}). For the hardware roadmap the results argue
two things: provisioning wrap-around (torus) inter-device links is worth
roughly a \SI{20}{}--\SI{25}{\percent} collective-latency reduction at
scale, and giving the IPCQ a fast on-PE staging buffer (TCM) rather than
forcing it through HBM or SRAM is worth another \SI{14}{}--%
\SI{38}{\percent} at large payloads.