ywkang
edb30326ce
- §6 Supporting Agentic Workloads: how the fused GQA design extends to agentic fan-out/fan-in; three-layer split (framework/runtime/kernel); Stationary-KV vs Distributed-Q execution policies. - §7 Hardware Performance-Spec Search for GQA: WIP stub (sweep intent over GEMM TFLOPS, MATH-engine ALUs, CUBE↔CUBE and SIP↔SIP BW). - Renumber Discussion/Conclusion/Future-Work to 08/09/10; update main.tex input order and toc.md. - Add Agentic_Runtime_Architecture.md design note; rebuild main.pdf. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
85 lines
4.4 KiB
Markdown
85 lines
4.4 KiB
Markdown
# KernBench — 2026 1H HW-SW Codesign Report — Table of Contents
|
|
|
|
> Agreed outline (the `/paper` contract). `/paper build` follows this.
|
|
> Period: 2026 1H. Language: English. References: external literature only
|
|
> (no ADR/SPEC named anywhere in the paper). PDF via Tectonic.
|
|
|
|
## Section order → files
|
|
|
|
1. **Executive Summary** — `sections/00-exec-summary.tex`
|
|
The attention-optimization goal, the two enabling optimizations (GEMM
|
|
composite command + PE_IPCQ communication), the fused GQA capstone, the
|
|
headline results, and the bottom-line recommendation on which HW changes
|
|
are worth keeping. Front-loads conclusions; may run slightly longer than
|
|
a terse abstract.
|
|
|
|
2. **Introduction** — `sections/01-introduction.tex`
|
|
1H focus is **attention-kernel optimization**: FlashAttention-style
|
|
tiling and Grouped Query Attention (GQA), building on prior work on
|
|
Multi-Head Attention (MHA, studied earlier — referenced as the
|
|
established baseline, not re-derived). Optimizing a fused attention
|
|
kernel requires two enabling optimizations, each studied in its own
|
|
right and then **combined inside the fused kernel**:
|
|
(a) **GEMM optimization** (the composite command), and
|
|
(b) **communication optimization** (PE_IPCQ).
|
|
Narrative arc: *two enablers → the fused GQA capstone that uses both.*
|
|
Motivate why HW-SW codesign (not software alone) is required.
|
|
|
|
3. **The KernBench Platform** — `sections/02-platform.tex`
|
|
- 3.1 *Why KernBench* — source-level kernel execution, no compiler /
|
|
SW-stack dependency; isolate algorithm-level optimization.
|
|
- 3.2 *Execution model* — discrete-event graph; runtime API → sim_engine
|
|
→ components; PE pipeline + composite commands; 2-pass data/timing.
|
|
- 3.3 *Latency model & accuracy* — traversal-based golden invariants;
|
|
structural CPU-dispatch cost model; known simplifications + calibration.
|
|
- 3.4 *Modeled Hardware Configuration* — shared platform-wide config from
|
|
`topology.yaml` (hierarchy counts, clocks, capacities/BW, link params,
|
|
cost-model constants). Per-experiment workload params live in §4–§6.
|
|
|
|
4. **GEMM Acceleration via Composite Command** — `sections/03-gemm.tex`
|
|
necessity · design · results · analysis. (+ GEMM workload params)
|
|
|
|
5. **All-Reduce Acceleration via PE_IPCQ** — `sections/04-allreduce.tex`
|
|
necessity · design · results · analysis. (+ All-Reduce topologies/sizes)
|
|
|
|
6. **Fused Grouped Query Attention** (composite + PE_IPCQ) — `sections/05-gqa.tex`
|
|
necessity · design · results · analysis. (+ GQA seq/head/user configs)
|
|
|
|
7. **Supporting Agentic Workloads** — `sections/06-agentic.tex`
|
|
How the §6 GQA design extends to agentic fan-out/fan-in: shared-prefix KV
|
|
reuse, batched replicated-Q attention, schema/pointer-based fan-in.
|
|
Implementation split across three levels (agentic framework / runtime /
|
|
kernel) with each component's responsibilities. Design, not yet measured.
|
|
|
|
8. **Hardware Performance-Spec Search for GQA** — `sections/07-hw-spec-search.tex`
|
|
*Work in progress.* Joint sweep of GEMM TFLOPS, MATH-engine ALU count,
|
|
CUBE↔CUBE (die-to-die) BW, SIP↔SIP (card-to-card) BW to find the balanced
|
|
spec for GQA. Intent + method only; data/figures/conclusions deferred.
|
|
|
|
9. **Discussion** — `sections/08-discussion.tex`
|
|
Which HW changes are meaningful, and under what regimes (cross-cutting).
|
|
|
|
10. **Conclusion** — `sections/09-conclusion.tex`
|
|
The codesign thesis, stated plainly, supported by the measured results.
|
|
|
|
11. **Future Work — 2H** — `sections/10-future-work.tex`
|
|
Add the FFN/MoE layer toward full LLM decoding; how compute & data should
|
|
be distributed for agentic / MoE workloads.
|
|
|
|
12. **References** *(optional)* — external literature only (FlashAttention,
|
|
Megatron-LM, GPT-3, Llama 3). No ADR/SPEC entries.
|
|
|
|
## Per-section structure (§4/§5/§6)
|
|
|
|
Each of the three optimization sections follows the same four beats:
|
|
**(a) necessity → (b) design → (c) experimental results → (d) analysis & meaning.**
|
|
|
|
## Build notes
|
|
|
|
- GEMM (§4) and All-Reduce (§5): reuse committed figures/CSVs under
|
|
`src/kernbench/benches/1H_milestone_output/{gemm,ccl}/`.
|
|
- GQA (§6): committed `sweep.json` holds op-counts only — latency + figures
|
|
are generated by an isolated harness under `scripts/paper/` at build time.
|
|
Measure only the implemented path; mark proposed designs (e.g. the
|
|
flat-ops `softmax_merge` decode opt2) as designed-not-measured.
|