\section{Introduction}
\label{sec:intro}

The performance of a large language model on an accelerator is decided less
by peak FLOPS than by how well the kernel exploits the memory system and
the interconnect. This is especially true of attention, which in the
decode phase reads a large KV cache to do a small amount of arithmetic.
Optimizing such kernels is fundamentally a \emph{codesign} problem: the
algorithm (how to tile, fuse, and reduce) and the hardware (what issue
mechanism, what collective engine, what memory hierarchy) have to be
designed against each other. Optimizing only the software leaves the
hardware idle; adding hardware that the software cannot reach is wasted
area.

\paragraph{The 1H focus: attention.} This half's work targets attention.
Multi-head attention (MHA) was studied previously and is taken here as the
established baseline. The new work is FlashAttention-style tiling and
Grouped-Query Attention (GQA)---the form used by modern long-context
decoders, in which several query heads share a single KV head to shrink the
KV cache. Realizing a fast \emph{fused} GQA kernel, however, is not a
single optimization. It decomposes into two enabling optimizations that we
studied separately and then brought together inside the fused kernel:

\begin{enumerate}
  \item \textbf{GEMM optimization} via a composite command
  (\S\ref{sec:gemm}): a way to issue a tiled matrix multiply as one
  self-routing pipeline so the MAC array stays fed without per-tile command
  overhead. Attention's $Q\!\cdot\!K^{\top}$ and $P\!\cdot\!V$ products are
  exactly such GEMMs.
  \item \textbf{Communication optimization} via PE\_IPCQ
  (\S\ref{sec:allreduce}): a per-PE on-device collective engine that
  performs all-reduce overlapped with compute. Attention's multi-user and
  sequence-parallel KV reductions are exactly such collectives.
\end{enumerate}

The report therefore reads as \emph{two enablers building to one capstone}:
GEMM (\S\ref{sec:gemm}) and all-reduce (\S\ref{sec:allreduce}) are
developed and measured on their own, and then fused GQA
(\S\ref{sec:gqa}) shows them operating together. Before the results,
\S\ref{sec:platform} describes the KernBench platform---why a
software-stack-independent, source-level simulator is the right tool for
this question, how it executes a kernel, how it computes latency and how
accurate that is, and the exact hardware configuration used throughout.
We close with a cross-cutting discussion of which hardware changes are
worth their cost (\S\ref{sec:discussion}), a conclusion
(\S\ref{sec:conclusion}), and the 2H agenda
(\S\ref{sec:future}): extending from attention to the feed-forward and
mixture-of-experts layers, and to the compute/data distribution questions
that full LLM decoding and agentic workloads raise.