report(S5.2): explain why dense mapping wins throughput (per-PE efficiency)

Both 1-kv (2 users x 64 PE) and 8-kv (16 users x 8 PE) fill all 128 PEs of
the SIP; the throughput gap is per-PE HBM efficiency. Splitting a head over
8 PEs leaves each a single tile, so pipeline-fill and the 8-PE softmax
reduce dominate -> 46% HBM util; one head per PE streams contiguously with
no reduce -> 76%. Throughput ratio (1.6x) tracks the utilization ratio
(1.7x): decode is bandwidth-bound, so throughput follows total HBM
efficiency, not PE count. 1-kv spends hardware on latency (8 PE/head for a
4.8x speedup = 60% strong-scaling efficiency).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/report/1H-codesign-paper/sections/05-gqa.tex

@@ -256,8 +256,27 @@ Aggregate throughput at $8$K therefore rises almost linearly with $B$, from
 \SI{0.86}{}~requests\,/\,\si{\micro\second} (\textsf{1-kv-per-cube}, capped
 at two users) to \SI{1.41}{} (\textsf{8-kv-per-cube}, sixteen users): the
 \emph{dense} mapping wins on throughput even though it is the \emph{slowest}
-per request, because it packs $8\times$ more concurrent users onto the SIP.
-The design conclusion is regime-dependent: \textsf{1-kv-per-cube} is the
+per request.
+
+The reason is worth spelling out, because both mappings fill the same
+hardware: at capacity \textsf{1-kv-per-cube} runs two users
+$\times$ 64 PEs and \textsf{8-kv-per-cube} sixteen users $\times$ 8 PEs, so
+each puts all 128 PEs of the SIP to work. What differs is per-PE
+\emph{efficiency}. Splitting one head across eight PEs
+(\textsf{1-kv-per-cube}) leaves each PE only $S_{kv}/8$ tokens---a single
+tile---so its fixed overheads (pipeline fill on the lazy KV load, and the
+eight-PE online-softmax chain-reduce) dominate the little streaming work it
+does, and it sustains just \SI{46}{\percent} of the per-PE HBM ceiling.
+Giving each head a whole PE (\textsf{8-kv-per-cube}) streams the full
+$S_{kv}$-token cache contiguously with no reduce, amortizing those overheads
+over $8\times$ more work and reaching \SI{76}{\percent}. The measured
+throughput ratio ($1.41/0.86 = 1.6\times$) matches the utilization ratio
+($0.76/0.46 = 1.7\times$) almost exactly: because decode is
+bandwidth-bound, aggregate throughput is set by total HBM efficiency, not
+by PE count. \textsf{1-kv-per-cube} is really spending hardware on
+\emph{latency}---eight PEs per head buy only a $4.8\times$ speedup, a $60\%$
+strong-scaling efficiency---and that same $40\%$ overhead is what caps its
+throughput once the SIP is full. The design conclusion is regime-dependent: \textsf{1-kv-per-cube} is the
 latency-optimal choice for single-stream or low-batch decode, while denser
 mappings are the throughput-optimal choice for high-batch short-context
 serving. At long context the per-request latencies already scale as $1/C$,