kernbench2/docs/report/1H-codesign-paper/toc.md

KernBench — 2026 1H HW-SW Codesign Report — Table of Contents

Agreed outline (the /paper contract). /paper build follows this. Period: 2026 1H. Language: English. References: external literature only (no ADR/SPEC named anywhere in the paper). PDF via Tectonic.

Section order → files

1. Executive Summary — sections/00-exec-summary.tex The attention-optimization goal, the two enabling optimizations (GEMM composite command + PE_IPCQ communication), the fused GQA capstone, the headline results, and the bottom-line recommendation on which HW changes are worth keeping. Front-loads conclusions; may run slightly longer than a terse abstract.

2. Introduction — sections/01-introduction.tex 1H focus is attention-kernel optimization: FlashAttention-style tiling and Grouped Query Attention (GQA), building on prior work on Multi-Head Attention (MHA, studied earlier — referenced as the established baseline, not re-derived). Optimizing a fused attention kernel requires two enabling optimizations, each studied in its own right and then combined inside the fused kernel: (a) GEMM optimization (the composite command), and (b) communication optimization (PE_IPCQ). Narrative arc: two enablers → the fused GQA capstone that uses both. Motivate why HW-SW codesign (not software alone) is required.

3. The KernBench Platform — sections/02-platform.tex

   • 3.1 Why KernBench — source-level kernel execution, no compiler / SW-stack dependency; isolate algorithm-level optimization.
   • 3.2 Execution model — discrete-event graph; runtime API → sim_engine → components; PE pipeline + composite commands; 2-pass data/timing.
   • 3.3 Latency model & accuracy — traversal-based golden invariants; structural CPU-dispatch cost model; known simplifications + calibration.
   • 3.4 Modeled Hardware Configuration — shared platform-wide config from topology.yaml (hierarchy counts, clocks, capacities/BW, link params, cost-model constants). Per-experiment workload params live in §4–§6.

4. GEMM Acceleration via Composite Command — sections/03-gemm.tex necessity · design · results · analysis. (+ GEMM workload params)

5. All-Reduce Acceleration via PE_IPCQ — sections/04-allreduce.tex necessity · design · results · analysis. (+ All-Reduce topologies/sizes)

6. Fused Grouped Query Attention (composite + PE_IPCQ) — sections/05-gqa.tex necessity · design · results · analysis. (+ GQA seq/head/user configs)

7. Supporting Agentic Workloads — sections/06-agentic.tex How the §6 GQA design extends to agentic fan-out/fan-in: shared-prefix KV reuse, batched replicated-Q attention, schema/pointer-based fan-in. Implementation split across three levels (agentic framework / runtime / kernel) with each component's responsibilities. Design, not yet measured.

8. Hardware Performance-Spec Search for GQA — sections/07-hw-spec-search.tex Work in progress. Joint sweep of GEMM TFLOPS, MATH-engine ALU count, CUBE↔CUBE (die-to-die) BW, SIP↔SIP (card-to-card) BW to find the balanced spec for GQA. Intent + method only; data/figures/conclusions deferred.

9. Discussion — sections/08-discussion.tex Which HW changes are meaningful, and under what regimes (cross-cutting).

10. Conclusion — sections/09-conclusion.tex The codesign thesis, stated plainly, supported by the measured results.

11. Future Work — 2H — sections/10-future-work.tex Add the FFN/MoE layer toward full LLM decoding; how compute & data should be distributed for agentic / MoE workloads.

12. References (optional) — external literature only (FlashAttention, Megatron-LM, GPT-3, Llama 3). No ADR/SPEC entries.

Per-section structure (§4/§5/§6)

Each of the three optimization sections follows the same four beats: (a) necessity → (b) design → (c) experimental results → (d) analysis & meaning.

Build notes

• GEMM (§4) and All-Reduce (§5): reuse committed figures/CSVs under src/kernbench/benches/1H_milestone_output/{gemm,ccl}/.
• GQA (§6): committed sweep.json holds op-counts only — latency + figures are generated by an isolated harness under scripts/paper/ at build time. Measure only the implemented path; mark proposed designs (e.g. the flat-ops softmax_merge decode opt2) as designed-not-measured.