ywkang
dd525bfcb7
New `/paper` slash-command skill that synthesizes ADR/SPEC content and live KernBench benchmark results into a sectioned LaTeX technical paper compiled to PDF with Tectonic (auto-installed). The skill negotiates a TOC, grounds every number in committed artifacts or fresh bench runs, and keeps report-only benches isolated. This commit also includes the first generated report: - docs/report/1H-codesign-paper/ — main.tex + per-section .tex, figures, toc.md contract, and the built 8-page main.pdf. Covers the platform (source-level kernels, latency model + accuracy, HW config from topology.yaml), GEMM via composite command, All-Reduce via PE_IPCQ, and fused GQA combining both, plus discussion/conclusion/2H future work. - scripts/paper/ — isolated report harnesses (not registered benches): paper_gqa_latency.py harvests per-panel GQA end-to-end latency + engine occupancy (the milestone only emitted op-counts); paper_plot_gqa.py renders the GQA figures. GEMM/All-Reduce reuse committed milestone figures/CSVs; GQA results are generated fresh. Honest flags retained: PE_CPU dispatch cost is 0 in this config, and the proposed two-composite softmax_merge decode is marked designed-not-measured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
41 lines
2.3 KiB
TeX
41 lines
2.3 KiB
TeX
\section{Future Work --- 2H}
|
|
\label{sec:future}
|
|
|
|
The 1H work covered attention end to end. The natural next step is to
|
|
complete the decoder and then to ask how compute and data should be
|
|
distributed for realistic, agentic workloads.
|
|
|
|
\paragraph{From attention to the full decoder: FFN and MoE.} A decoder
|
|
block is attention followed by a feed-forward network (FFN), and in modern
|
|
models that FFN is increasingly a mixture-of-experts (MoE) layer. The FFN
|
|
is the compute-rich counterpart to attention---it is where the composite
|
|
command's MAC-efficiency gains (\S\ref{sec:gemm}) should matter most---so
|
|
adding it gives a balanced view of a full block instead of its
|
|
memory-bound half alone. MoE adds a new dimension: a routing step selects a
|
|
few experts per token, turning the dense FFN GEMM into a sparse, data-
|
|
dependent dispatch. The open questions are how to issue expert GEMMs as
|
|
composites under data-dependent token counts, and how to move tokens to
|
|
experts efficiently---an all-to-all-shaped communication pattern distinct
|
|
from the all-reduce studied here, and a natural extension of the PE\_IPCQ
|
|
work.
|
|
|
|
\paragraph{Compute and data distribution for full LLM decoding.} With both
|
|
attention and FFN/MoE in hand, the question becomes where each layer's
|
|
compute and state should live. Attention is KV-bound and favors keeping the
|
|
KV cache close to the PEs that consume it; FFN/MoE is compute-bound and
|
|
favors spreading GEMM work across PEs; MoE routing makes the optimal
|
|
placement token- and time-dependent. A 2H goal is to use KernBench to
|
|
explore these placement and parallelization trade-offs---tensor vs.\ expert
|
|
vs.\ sequence parallelism---under a single, software-stack-independent
|
|
model, so the interconnect and memory implications of each choice are
|
|
measured rather than assumed.
|
|
|
|
\paragraph{Agentic workloads.} Agentic inference interleaves many
|
|
short, bursty decode requests with tool use and long shared contexts,
|
|
which stresses the system differently from a single long generation:
|
|
context reuse across requests, dynamic batching, and uneven expert load all
|
|
change how compute and data should be dispersed. Characterizing how total
|
|
compute and data movement distribute across the SIP/CUBE/PE hierarchy under
|
|
such workloads---and which of the 1H hardware levers still dominate when the
|
|
workload is this irregular---is the broader 2H agenda.
|