diff --git a/docs/report/1H-codesign-paper/build/main.pdf b/docs/report/1H-codesign-paper/build/main.pdf index 35a5490..9d341a0 100644 Binary files a/docs/report/1H-codesign-paper/build/main.pdf and b/docs/report/1H-codesign-paper/build/main.pdf differ diff --git a/docs/report/1H-codesign-paper/sections/05-gqa.tex b/docs/report/1H-codesign-paper/sections/05-gqa.tex index a7de6ba..13fd776 100644 --- a/docs/report/1H-codesign-paper/sections/05-gqa.tex +++ b/docs/report/1H-codesign-paper/sections/05-gqa.tex @@ -256,8 +256,27 @@ Aggregate throughput at $8$K therefore rises almost linearly with $B$, from \SI{0.86}{}~requests\,/\,\si{\micro\second} (\textsf{1-kv-per-cube}, capped at two users) to \SI{1.41}{} (\textsf{8-kv-per-cube}, sixteen users): the \emph{dense} mapping wins on throughput even though it is the \emph{slowest} -per request, because it packs $8\times$ more concurrent users onto the SIP. -The design conclusion is regime-dependent: \textsf{1-kv-per-cube} is the +per request. + +The reason is worth spelling out, because both mappings fill the same +hardware: at capacity \textsf{1-kv-per-cube} runs two users +$\times$ 64 PEs and \textsf{8-kv-per-cube} sixteen users $\times$ 8 PEs, so +each puts all 128 PEs of the SIP to work. What differs is per-PE +\emph{efficiency}. Splitting one head across eight PEs +(\textsf{1-kv-per-cube}) leaves each PE only $S_{kv}/8$ tokens---a single +tile---so its fixed overheads (pipeline fill on the lazy KV load, and the +eight-PE online-softmax chain-reduce) dominate the little streaming work it +does, and it sustains just \SI{46}{\percent} of the per-PE HBM ceiling. +Giving each head a whole PE (\textsf{8-kv-per-cube}) streams the full +$S_{kv}$-token cache contiguously with no reduce, amortizing those overheads +over $8\times$ more work and reaching \SI{76}{\percent}. The measured +throughput ratio ($1.41/0.86 = 1.6\times$) matches the utilization ratio +($0.76/0.46 = 1.7\times$) almost exactly: because decode is +bandwidth-bound, aggregate throughput is set by total HBM efficiency, not +by PE count. \textsf{1-kv-per-cube} is really spending hardware on +\emph{latency}---eight PEs per head buy only a $4.8\times$ speedup, a $60\%$ +strong-scaling efficiency---and that same $40\%$ overhead is what caps its +throughput once the SIP is full. The design conclusion is regime-dependent: \textsf{1-kv-per-cube} is the latency-optimal choice for single-stream or low-batch decode, while denser mappings are the throughput-optimal choice for high-batch short-context serving. At long context the per-request latencies already scale as $1/C$,