paper(1H): 6-case GQA long-ctx + IPCQ design alternatives + GEMM terminology
- §6 GQA: rewrite long-context decode from 4-case to 6-case. Data Placement Policy now presents the six placement options and their intrinsic per-PE memory / per-token comm costs (no winner predicted); the long-context subsection selects the best placement for that regime (Case 6 ★, both-axes S_kv shard) from the measured 6-case sweep. Add KV-sharding diagram + analytical budget/summary figures. - Regenerate the 6-row decode sweep (milestone-1h-gqa) and the sweep-dependent decode panels (latency/traffic/parallelism/memory) so figures, prose, and sweep_decode.json are mutually consistent. - §5 All-Reduce: add IPCQ design alternatives (architecture + decision matrix) as design-rationale schematics (illustrative step-counts, not measured) and the bench-generated topology diagram. - §4 GEMM: rename "user-orchestrated/user-level" -> "kernel-orchestrated/ kernel-level" (orchestration runs in the kernel program vs the scheduler-orchestrated composite); minor accuracy fixes (7.18 TFLOP/s ~10% below peak; ~781 ns DMA). - Recompile build/main.pdf. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -172,16 +172,16 @@ kernel of \S\ref{sec:gqa} reaches for next — keeping the right
|
||||
working set on-chip so the composite pipeline lands in the
|
||||
compute-rich regime rather than the BW-bound one.
|
||||
|
||||
\subsection{Why composite, and not user-orchestrated async loading?}
|
||||
\subsection{Why composite, and not kernel-orchestrated async loading?}
|
||||
\label{sec:gemm-vs-async}
|
||||
|
||||
A reader familiar with double-buffered GEMM kernels on conventional
|
||||
hardware may ask: why a hardware-side composite command at all? Why
|
||||
isn't the obvious user-level pattern --- async-load each operand,
|
||||
isn't the obvious kernel-level pattern --- async-load each operand,
|
||||
overlap with compute, accumulate --- sufficient?
|
||||
|
||||
To answer this concretely we contrast composite against two
|
||||
user-orchestrated baselines that have access to the same single-op
|
||||
kernel-orchestrated baselines that have access to the same single-op
|
||||
primitives the platform exposes (\textsf{tl.load} for async DMA into
|
||||
TCM, \textsf{tl.dot} for a single-op GEMM command on TCM-resident
|
||||
operands, \textsf{tl.store} for a DMA write-back). Both baselines
|
||||
@@ -199,7 +199,7 @@ way at the runtime API surface to express ``start computing on
|
||||
tile 0 of $B$ while tile 1 is still in flight.'' Load-of-$B$ and
|
||||
GEMM therefore serialize.
|
||||
|
||||
\paragraph{Async-tiled (chunked prefetch).} The user-level workaround is to
|
||||
\paragraph{Async-tiled (chunked prefetch).} The kernel-level workaround is to
|
||||
split $B$ along $K$ into \textsf{TILE\_K}-sized chunks, issue async
|
||||
\textsf{tl.load}s for those chunks, issue one \textsf{tl.dot} per
|
||||
chunk (each blocking only on its own $b_i$), and accumulate via
|
||||
@@ -227,7 +227,7 @@ $K_{\text{KV}}=4096, d_{\text{head}}=128$ already needs
|
||||
--- past the cap. async-full and queue-all async-tiled are therefore
|
||||
not just slower than composite but \emph{architecturally infeasible}
|
||||
at LLM context length. The depth-2 async-tiled kernel is the only
|
||||
user-level
|
||||
kernel-level
|
||||
variant whose peak TCM footprint stays
|
||||
$O(2 \cdot \textsf{TILE\_K} \cdot N)$ regardless of $K$, the same
|
||||
order as composite's per-tile streaming buffer. It is the apples-to-apples
|
||||
@@ -249,7 +249,7 @@ onto $K / \textsf{TILE\_K} = 48$ hardware tiles, so per-tile costs
|
||||
amplify into the largest measurable gap. The work content is
|
||||
identical for all four kernels: $\sim$6.3 M f16 MACs and
|
||||
$\sim$386 KiB of $B$ traffic from HBM, which together require
|
||||
$\sim$786 ns of GEMM-engine compute and $\sim$750 ns of DMA on a
|
||||
$\sim$786 ns of GEMM-engine compute and $\sim$781 ns of DMA on a
|
||||
saturated per-PE link. What differs is the number of host commands
|
||||
the same work is decomposed into --- 2 for composite (one
|
||||
\textsf{tl.load(A)} plus one composite), 4 for async-full, and 192
|
||||
@@ -257,7 +257,7 @@ for either async-tiled variant (48 $A$-loads + 48 $B$-loads + 48
|
||||
\textsf{tl.dot}s + 47 elementwise adds + 1 store). The
|
||||
engine-pipeline-window throughput tracks that decomposition closely:
|
||||
composite reaches \SI{7.18}{\tera\flop\per\second} (post-overlap
|
||||
limit, only \SI{12}{\percent} below the \SI{8}{\tera\flop\per\second}
|
||||
limit, only \SI{10}{\percent} below the \SI{8}{\tera\flop\per\second}
|
||||
per-PE GEMM peak), async-full \SI{3.91}{\tera\flop\per\second}
|
||||
(DMA and compute serialize on a single big dot), and both async-tiled
|
||||
variants $\sim$\SI{2.53}{\tera\flop\per\second} (192 commands' worth of
|
||||
@@ -272,7 +272,7 @@ specific simulator mechanisms.
|
||||
four issuance patterns: composite (one command, scheduler streams
|
||||
per-tile internally), async-full (one \textsf{tl.dot} on
|
||||
fully-loaded $B$), async-tiled with depth-2 double-buffer
|
||||
(TCM-bounded; the only user-level variant that scales to LLM
|
||||
(TCM-bounded; the only kernel-level variant that scales to LLM
|
||||
context length), and async-tiled with depth-$\infty$
|
||||
(all B-tiles queued up front; included as a sanity check that the
|
||||
prefetch depth is \emph{not} what separates the async-tiled kernel from
|
||||
@@ -293,7 +293,7 @@ and all three async kernels beat it.}
|
||||
\end{figure*}
|
||||
|
||||
\paragraph{Decomposing the gap.} Three structural mechanisms separate
|
||||
composite from the user-level baselines, and they layer.
|
||||
composite from the kernel-level baselines, and they layer.
|
||||
|
||||
\emph{1. Inter-engine token routing happens below the host-side
|
||||
dispatch path.} The composite encodes the full
|
||||
@@ -309,7 +309,7 @@ hand-offs total --- behind a single command from the host's point of
|
||||
view.
|
||||
|
||||
\emph{2. \textsf{tl.dot} cannot replicate that per-tile pipeline at the
|
||||
user level.} A single-op GEMM command is handled on the GEMM engine as
|
||||
kernel level.} A single-op GEMM command is handled on the GEMM engine as
|
||||
a single monolithic compute timeout for the supplied $M{\times}K{\times}N$;
|
||||
there is no internal token loop that would let a streaming DMA of
|
||||
$B[i{+}1]$ overlap with the GEMM of $B[i]$ inside one
|
||||
@@ -388,7 +388,7 @@ the four kernels to combine (a) macro-command dispatch at the host
|
||||
boundary (amortizing the structural CPU cost across all the work a
|
||||
single GEMM does), (b) scheduler-internal per-HW-tile streaming of
|
||||
DMA$\rightleftarrows$compute, and (c) TCM-bounded streaming buffer.
|
||||
User-orchestrated async kernels can have any two of those, not all
|
||||
Kernel-orchestrated async kernels can have any two of those, not all
|
||||
three: async-full pays one host dispatch (a) but forfeits per-tile
|
||||
overlap (b) and pins all of $B$ in TCM (c); depth-$\infty$ async-tiled
|
||||
achieves inter-chunk overlap but at $N_K$ host dispatches and
|
||||
|
||||
Reference in New Issue
Block a user