\section*{Executive Summary}
\addcontentsline{toc}{section}{Executive Summary}

This report studies how to make attention-centric LLM kernels fast through
hardware--software codesign, using KernBench, a discrete-event simulator
that runs kernels at the source level---without a compiler or other
software-stack dependency---so that algorithm-level optimizations and the
hardware features that support them can be isolated and measured.

The 1H work centers on attention. Building on prior multi-head-attention
work, we target FlashAttention-style Grouped-Query Attention (GQA) and find
that a fast fused attention kernel rests on two enabling optimizations,
each studied in its own right and then combined inside the fused kernel:
a \textbf{composite command} that issues a tiled GEMM as a single,
self-routing pipeline, and \textbf{PE\_IPCQ}, a per-PE on-device collective
engine for all-reduce.

The measured results form a single, consistent story. For GEMM, the
composite command lets compute-rich shapes reach
\textasciitilde\SI{78}{\percent} of peak MAC efficiency, with the simulator's
measured efficiency tracking the analytic prediction within
\SIrange{10}{20}{\percent}---both a performance result and a validation of
the latency model. For all-reduce, PE\_IPCQ delivers on-device collectives
whose latency tracks the interconnect's physical limits; a 2D-torus
inter-device fabric is \SI{20}{}--\SI{25}{\percent} faster than a mesh or
ring at scale, and staging the collective in on-PE TCM rather than HBM or
SRAM saves a further \SI{14}{}--\SI{38}{\percent}. For fused GQA, the two
enablers combine: because the composite command keeps the MAC array nearly
idle, the kernel is exposed as overwhelmingly data-movement bound---the DMA
engine outweighs the compute engines by two to three orders of magnitude,
and on-device IPCQ reduction traffic scales with users and devices.

\textbf{Bottom line.} For an attention-dominated decoder, the hardware
changes worth keeping are the ones that move and reduce data faster: the
single-command self-routing GEMM pipeline (which makes the MAC array
usable and exposes the real bottleneck), the on-device PE\_IPCQ collective
with its compute/communication virtual-channel split, fast on-PE staging
memory, and wrap-around (torus) inter-device links. Additional raw MAC
throughput is not the limiting factor for this workload.