ywkang
dd525bfcb7
New `/paper` slash-command skill that synthesizes ADR/SPEC content and live KernBench benchmark results into a sectioned LaTeX technical paper compiled to PDF with Tectonic (auto-installed). The skill negotiates a TOC, grounds every number in committed artifacts or fresh bench runs, and keeps report-only benches isolated. This commit also includes the first generated report: - docs/report/1H-codesign-paper/ — main.tex + per-section .tex, figures, toc.md contract, and the built 8-page main.pdf. Covers the platform (source-level kernels, latency model + accuracy, HW config from topology.yaml), GEMM via composite command, All-Reduce via PE_IPCQ, and fused GQA combining both, plus discussion/conclusion/2H future work. - scripts/paper/ — isolated report harnesses (not registered benches): paper_gqa_latency.py harvests per-panel GQA end-to-end latency + engine occupancy (the milestone only emitted op-counts); paper_plot_gqa.py renders the GQA figures. GEMM/All-Reduce reuse committed milestone figures/CSVs; GQA results are generated fresh. Honest flags retained: PE_CPU dispatch cost is 0 in this config, and the proposed two-composite softmax_merge decode is marked designed-not-measured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
50 lines
2.7 KiB
TeX
50 lines
2.7 KiB
TeX
\section{Introduction}
|
|
\label{sec:intro}
|
|
|
|
The performance of a large language model on an accelerator is decided less
|
|
by peak FLOPS than by how well the kernel exploits the memory system and
|
|
the interconnect. This is especially true of attention, which in the
|
|
decode phase reads a large KV cache to do a small amount of arithmetic.
|
|
Optimizing such kernels is fundamentally a \emph{codesign} problem: the
|
|
algorithm (how to tile, fuse, and reduce) and the hardware (what issue
|
|
mechanism, what collective engine, what memory hierarchy) have to be
|
|
designed against each other. Optimizing only the software leaves the
|
|
hardware idle; adding hardware that the software cannot reach is wasted
|
|
area.
|
|
|
|
\paragraph{The 1H focus: attention.} This half's work targets attention.
|
|
Multi-head attention (MHA) was studied previously and is taken here as the
|
|
established baseline. The new work is FlashAttention-style tiling and
|
|
Grouped-Query Attention (GQA)---the form used by modern long-context
|
|
decoders, in which several query heads share a single KV head to shrink the
|
|
KV cache. Realizing a fast \emph{fused} GQA kernel, however, is not a
|
|
single optimization. It decomposes into two enabling optimizations that we
|
|
studied separately and then brought together inside the fused kernel:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{GEMM optimization} via a composite command
|
|
(\S\ref{sec:gemm}): a way to issue a tiled matrix multiply as one
|
|
self-routing pipeline so the MAC array stays fed without per-tile command
|
|
overhead. Attention's $Q\!\cdot\!K^{\top}$ and $P\!\cdot\!V$ products are
|
|
exactly such GEMMs.
|
|
\item \textbf{Communication optimization} via PE\_IPCQ
|
|
(\S\ref{sec:allreduce}): a per-PE on-device collective engine that
|
|
performs all-reduce overlapped with compute. Attention's multi-user and
|
|
sequence-parallel KV reductions are exactly such collectives.
|
|
\end{enumerate}
|
|
|
|
The report therefore reads as \emph{two enablers building to one capstone}:
|
|
GEMM (\S\ref{sec:gemm}) and all-reduce (\S\ref{sec:allreduce}) are
|
|
developed and measured on their own, and then fused GQA
|
|
(\S\ref{sec:gqa}) shows them operating together. Before the results,
|
|
\S\ref{sec:platform} describes the KernBench platform---why a
|
|
software-stack-independent, source-level simulator is the right tool for
|
|
this question, how it executes a kernel, how it computes latency and how
|
|
accurate that is, and the exact hardware configuration used throughout.
|
|
We close with a cross-cutting discussion of which hardware changes are
|
|
worth their cost (\S\ref{sec:discussion}), a conclusion
|
|
(\S\ref{sec:conclusion}), and the 2H agenda
|
|
(\S\ref{sec:future}): extending from attention to the feed-forward and
|
|
mixture-of-experts layers, and to the compute/data distribution questions
|
|
that full LLM decoding and agentic workloads raise.
|