ywkang
edb30326ce
- §6 Supporting Agentic Workloads: how the fused GQA design extends to agentic fan-out/fan-in; three-layer split (framework/runtime/kernel); Stationary-KV vs Distributed-Q execution policies. - §7 Hardware Performance-Spec Search for GQA: WIP stub (sweep intent over GEMM TFLOPS, MATH-engine ALUs, CUBE↔CUBE and SIP↔SIP BW). - Renumber Discussion/Conclusion/Future-Work to 08/09/10; update main.tex input order and toc.md. - Add Agentic_Runtime_Architecture.md design note; rebuild main.pdf. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
52 lines
3.0 KiB
TeX
52 lines
3.0 KiB
TeX
\section{Future Work}
|
|
\label{sec:future}
|
|
|
|
The 1H work covered attention end to end. The natural next step is to
|
|
complete the decoder and then to ask how compute and data should be
|
|
distributed for realistic, agentic workloads.
|
|
|
|
\paragraph{From attention to the full decoder: FFN and MoE.} A decoder
|
|
block is attention followed by a feed-forward network (FFN), and in modern
|
|
models that FFN is increasingly a mixture-of-experts (MoE) layer. The FFN
|
|
is the compute-rich counterpart to attention---it is where the composite
|
|
command's MAC-efficiency gains (\S\ref{sec:gemm}) should matter most---so
|
|
adding it gives a balanced view of a full block instead of its
|
|
memory-bound half alone. MoE adds a new dimension: a routing step selects a
|
|
few experts per token, turning the dense FFN GEMM into a sparse, data-
|
|
dependent dispatch. The open questions are how to issue expert GEMMs as
|
|
composites under data-dependent token counts, and how to move tokens to
|
|
experts efficiently---an all-to-all-shaped communication pattern distinct
|
|
from the all-reduce studied here, and a natural extension of the PE\_IPCQ
|
|
work.
|
|
|
|
\paragraph{Compute and data distribution for full LLM decoding.} With both
|
|
attention and FFN/MoE in hand, the question becomes where each layer's
|
|
compute and state should live. Attention is KV-bound and favors keeping the
|
|
KV cache close to the PEs that consume it; FFN/MoE is compute-bound and
|
|
favors spreading GEMM work across PEs; MoE routing makes the optimal
|
|
placement token- and time-dependent. A 2H goal is to use KernBench to
|
|
explore these placement and parallelization trade-offs---tensor vs.\ expert
|
|
vs.\ sequence parallelism---under a single, software-stack-independent
|
|
model, so the interconnect and memory implications of each choice are
|
|
measured rather than assumed.
|
|
|
|
\paragraph{Agentic workloads: from design to implementation.}
|
|
Section~\ref{sec:agentic} established \emph{how} the fused GQA design
|
|
extends to agentic fan-out/fan-in and \emph{which} responsibilities fall to
|
|
the framework, the runtime, and the kernel. The 2H step is no longer to
|
|
analyse the workload but to \emph{build} that support: the detailed design
|
|
and implementation of all three levels. This is necessary work rather than
|
|
optional, because today's agentic frameworks are effectively all
|
|
closed-source---there is no open substrate that exposes shared-prefix KV
|
|
reuse, cross-agent query batching, and schema/pointer-based fan-in down to
|
|
the hardware. Concretely, 2H targets: a \textbf{framework} layer that
|
|
manages the agent tree, identifies co-schedulable branches, and enforces
|
|
compact fan-in; a \textbf{runtime} layer that forks logical contexts by
|
|
page table without copying shared KV, batches sub-agent query rows, and
|
|
selects the execution policy; and a \textbf{kernel} layer that runs the
|
|
batched replicated-Q attention and hierarchical online-softmax merge over
|
|
PE\_IPCQ. Bringing these up on KernBench turns the
|
|
Section~\ref{sec:agentic} design into a measured, end-to-end agentic path
|
|
and lets us confirm which of the 1H hardware levers still dominate once the
|
|
full framework/runtime/kernel stack is in place.
|