ywkang
edb30326ce
- §6 Supporting Agentic Workloads: how the fused GQA design extends to agentic fan-out/fan-in; three-layer split (framework/runtime/kernel); Stationary-KV vs Distributed-Q execution policies. - §7 Hardware Performance-Spec Search for GQA: WIP stub (sweep intent over GEMM TFLOPS, MATH-engine ALUs, CUBE↔CUBE and SIP↔SIP BW). - Renumber Discussion/Conclusion/Future-Work to 08/09/10; update main.tex input order and toc.md. - Add Agentic_Runtime_Architecture.md design note; rebuild main.pdf. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
28 lines
1.6 KiB
TeX
28 lines
1.6 KiB
TeX
\section{Conclusion}
|
|
\label{sec:conclusion}
|
|
|
|
This 1H work set out to make attention-centric LLM kernels fast through
|
|
hardware--software codesign, and to do so on a platform that isolates
|
|
algorithm-level behavior from the rest of the software stack. The result is
|
|
a coherent picture rather than three separate optimizations. A composite
|
|
command that issues a tiled GEMM as one self-routing pipeline makes the MAC
|
|
array usable---reaching \textasciitilde\SI{78}{\percent} of peak on
|
|
compute-rich shapes with measured efficiency tracking theory---and, just as
|
|
importantly, makes compute cheap enough that the real bottleneck becomes
|
|
visible. A per-PE on-device collective engine, PE\_IPCQ, turns all-reduce
|
|
into a primitive whose latency follows the interconnect's physical limits,
|
|
with topology and staging-memory choices each worth tens of percent. Fused
|
|
Grouped-Query Attention then combines the two and shows the payoff and the
|
|
lesson at once: the kernel is data-movement bound, so the optimizations
|
|
that move and reduce data---not those that add arithmetic---are what
|
|
determine its speed.
|
|
|
|
The practical conclusion for the hardware roadmap is therefore specific.
|
|
The changes worth keeping are the single-command self-routing GEMM
|
|
pipeline, the on-device PE\_IPCQ collective with its compute/communication
|
|
virtual-channel split, fast on-PE staging memory, and wrap-around
|
|
inter-device links. Additional MAC throughput is not, for this workload, a
|
|
meaningful investment. KernBench made these conclusions measurable by
|
|
holding everything except the algorithm and the hardware fixed; the next
|
|
half extends the same method beyond attention to the rest of the decoder.
|