Two new long-ctx decode attention kernels for the d_head-TP sharding
variants the 6-case chart predicts:
· _gqa_attention_decode_long_ctx_cube_sp_pe_tp_dhead.py
Cube-SP × PE-TP-d_head (Case 4 in chart). Per cube holds
S_kv/C tokens of full d_head; per PE holds same tokens but
only d_head/P dims. Partial Q·Kᵀ scores reduced intra-cube
before softmax; outer (m,ℓ,O) merge two-phase (intra+inter).
· _gqa_attention_decode_long_ctx_cube_tp_dhead_pe_sp.py
Cube-TP-d_head × PE-SP (Case 5). Per cube holds full S_kv
with only d_head/C dims; per PE holds S_kv/P of those dims.
Partial scores reduced inter-cube (UCIe) before softmax.
Sweep dispatch (gqa_decode_long_ctx_4cases.py) extended with two
new panels so the milestone-1h-gqa sweep covers all 6 cases.
Smoke test scripts/verify_case4_dhead_tp.py runs Cases 4/5/6 at
S_kv=2K to validate the kernels load and execute end-to-end.
Plus the figure-generation toolchain that produced the committed
PNGs in the prior commit (dd3337f):
· paper_plot_gqa_4cases_summary.py - 3-panel summary +
2-panel (analytical / paired-measured) chart generator.
_plot_comm now takes mode="analytical" | "paired".
· paper_plot_gqa_kv_sharding_diagram.py - 6-case 2-D KV-tensor
diagram + companion comparison-table PNG.
· measure_gqa_decode_placement_comm.py - runs all 6 kernels at
S_kv=8K, sums actual IPCQ-copy bytes from engine.op_log,
scales partial-score AR ×128 to S_kv=1M, writes
gqa_3cases_measured_comm.json (committed).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the user-orchestrated async GEMM variants used in paper §3.4 to
contrast with the composite (load_ref) path:
- matmul-async : naive (tl.load full A+B, one tl.dot)
- matmul-async-chunked : depth-inf prefetch (all B-tiles queued)
- matmul-async-chunked-db : depth-2 double-buffer (TCM-bounded)
plus scripts/paper/paper_plot_gemm_async_vs_composite.py (4-way per-PE
TFLOPS sweep + figure) and the regenerated outputs under
1H_milestone_output/gemm/.
Sweep regenerated under the ADR-0064 D8 single-op cost model (commit
2d8271c). At K=3072 the composite-vs-async-tiled gap narrows from the
pre-D8 ~6.3x to ~2.8x (composite 7.18 / async-tiled 2.54 TFLOPS): D8
makes the async-tiled kernel's ~191 single-op dispatches 5x cheaper, so
its dispatch overhead drops to ~1.5us. Qualitative order persists:
composite > async-full (3.91) > async-tiled (2.54).
test_bench_registry.py: register matmul-async / -chunked / -chunked-db
(also picks up the earlier milestone-gqa-headline -> milestone-1h-gqa
rename already present in the tree).
--- Remaining work (resume here if interrupted) ---
- Paper §3.4 (03-gemm.tex sec:gemm-vs-async): finish naive->async-full /
chunked->async-tiled rename AND reframe "FIXED_DMA=8 for DMA descriptors"
to single-op(8) vs composite(40). Figure already copied to
docs/report/1H-codesign-paper/figures/gemm_composite_vs_async_tflops.png.
- Paper §3.4 K=3072 para + mechanism #3: update dispatch breakdown to new
model (191 single-op * 8c ~= 1.5us, was 4.6us) and headline gap
~6.3x -> ~2.8x.
- Rebuild build/main.pdf (tectonic) + verify via pdftotext.
- Then commit Group 3 (paper + diagrams + PDF).
- Table 2 / §2 prose / ADR-0064 D8 already done in 2d8271c.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Analytical plot generator + PNG covering the 4 sharding cases from
slide 17 plus the 2 d_head-TP variants (Cases 4, 5). Three panels:
per-PE HBM budget breakdown (Wq + Wk + Wv + Wo + 4.24 GB KV
headroom), per-PE KV memory at S_kv=1M, and per-PE communication
per output token (decode). Numbers match slide 17 (max KV context
~7.1M for the 64-way sharded cases).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirrors the existing decode 4-cases study for prefill on the LLaMA-3.1-70B
single-KV-head group (h_q=8, h_kv=1, d_head=128) across 8 cubes × 8 PEs:
Case 1 Cube-SP × PE-TP → KV S_kv-Ring across cubes; Q/O T_q-split 64-way
Case 2 Cube-Repl × PE-TP → full KV per cube; only CUBE 0's P PEs split T_q
Case 3 Cube-Repl × PE-SP → full KV per cube; PEs SP on S_kv (intra-cube AR)
Case 4 Cube-SP × PE-SP → KV split 64-way + Q T_q-split across cubes
(Ring KV + per-cube intra-CUBE AR) ★ optimal
Cases 2 and 3 use Q-axis tiling (TILE_Q) so the per-PE scratch is bounded
by TILE_Q × TILE_S_KV regardless of T_q (the full Q tensor would be
G·T_q·d_head·2 bytes which scales linearly with T_q and blew the 1 MB
budget at T_q≥128 without tiling).
Per-panel try/except in run_sweep tolerates per-panel failures so
sweep.json always lands with whichever cases succeed plus a failures
list. Case 4 uses configure_sfr_intercube_ring with the snake submesh
(2,4) which installs both the Ring E/W lanes and the intra_* lanes
needed for the intra-CUBE reduce — single SFR config for all 4 cases.
The plot script generates 4 PNGs (latency, traffic, memory, parallelism)
into 1H_milestone_output/gqa/long_ctx/. The parallelism chart shows
total compute work (active_PE × T_q_per_PE × S_kv_processed) — exposes
Case 3's 8× redundancy vs. Cases 1/2/4 which all do unique work.
gqa_prefill_long_ctx_4cases.py exposes run_sweep() rather than a @bench
decorator — the umbrella bench in the next commit invokes it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Splits the GQA helpers into a dedicated subpackage to make room for the
prefill 4-cases study (next commit) and a single umbrella bench
(milestone-1h-gqa, after that).
Layout:
benches/gqa_helpers/
long_ctx/ — decode 4-cases kernels + sweep runner
short_ctx/ — prefill/decode short-context kernels
shared/ — _gqa_panel_helpers + decode_opt2 (context-agnostic)
The registry audit now skips subpackages so gqa_helpers/ (without a
leading underscore) doesn't get audited for @bench decorators.
Also drops the legacy milestone-gqa-headline bench, its
_gqa_attention_prefill_long kernel, 6 dependent prefill tests, the
paper_gqa_latency.py report harness, and the 3 stale headline-derived
PNGs the §6 wire-up referenced (paper will re-pull from the new
1H_milestone_output/gqa/long_ctx/ once §6 is updated).
The _ccl_cfg and _summarize_op_log helpers used to live in the
headline bench; extracted them to gqa_helpers/shared/_gqa_panel_helpers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bench changes:
- new _end_to_end_ns(op_log) and _engine_occupancy_ns(op_log) helpers
in milestone_gqa_decode_long_ctx_4cases.py (mirror paper_gqa_latency.py)
- _run_panel return dict now carries latency_ns + engine_occupancy_ns
alongside op_log_summary, so sweep.json is the single source of truth
for the comparative figures
Plot script:
- new scripts/paper/paper_plot_gqa_decode_long_ctx_4cases.py reads
sweep.json and emits 3 PNGs to docs/report/1H-codesign-paper/figures/:
gqa_decode_long_ctx_4cases_latency.png (end-to-end latency / case)
gqa_decode_long_ctx_4cases_traffic.png (ipcq/dma op counts / case)
gqa_decode_long_ctx_4cases_memory.png (KV bytes per cube / case)
Test changes:
- 2 new tests verifying the helpers + _run_panel dict shape
- lower smoke S_kv from 8192 -> 2048 (4x faster Case 2; assertions
are S_kv-independent; one fold-loop iteration preserved)
18/18 tests pass in ~4 min.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Reordered §2 subsections to follow the SIP → CUBE → PE → graph
flow: Why KernBench → Device and execution model → Latency model
→ Modeled hardware configuration. Readers now meet the device
hierarchy before the graph abstraction that re-uses it.
- §2.2 Device and execution model: starts with the SIP/CUBE/PE
hierarchy paragraphs (each anchoring fig:sip-arch, fig:cube-arch,
fig:pe-arch); then the runtime-API/sim-engine/components bullet
list; then the atomic-vs-composite command distinction (corrects
the prior over-narrow framing that read every PE command as
composite -- atomic single-engine commands exist too, and PE_CPU
itself runs control-plane work directly).
- §2.3 Latency model: opens with the four-contribution decomposition
(per-node overhead, per-edge transmission, drain, queuing delay)
and the latency_model schematic; retains existing The hardware as
a graph / From graph to DES / Latency contributions / Congestion
/ Control-plane cost model / Accuracy paragraphs. Accuracy
paragraph now closes on KernBench's sufficiency for *relative*
HW/SW design trade-offs given analytic + external-simulator
agreement.
- New figures and assets:
- figures/sip_architecture.pdf (SIP-level graph view)
- figures/cube_architecture.pdf (CUBE-level zoom-in)
- figures/latency_model.png (conceptual latency-model
schematic with per-node /
per-edge / drain / queuing-delay
colour coding)
- figures/pe_architecture.png (carried over)
- Source-of-truth generator for the latency schematic:
scripts/paper/paper_latency_model_diagram.py (a report-only
harness under scripts/paper/ per the /paper isolation rule).
- main.tex preamble: \usepackage{tikz} added (kept from prior
sequence-diagram draft -- harmless now that the latency model is
a PNG; left in to keep paragraph numbering stable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The legacy panel names suggested batched serving semantics they never
had — all four modeled a single user with KV sharded differently
(C=1 single-cube; C=4 multi-cube Cube-SP), at toy dims (T_q=4, S_kv≤128).
The single-KV-group C=8 panel + the new milestone-gqa-decode-4cases
bench cover the meaningful comparisons; pytest regression already
covers C=1/C=4 configurations end-to-end at richer scale.
Changes:
- milestone_gqa_headline.py: drop the 4 legacy panels; _PANELS now
contains only single_kv_group_prefill_gqa_c8_p8. Update docstring.
- tests/attention/test_milestone_gqa_headline.py: drop the 3 legacy-
panel architectural tests (Ring-KV traffic, root-only decode write,
per-CUBE distributed output) and test_decode_panels_use_real_gqa
(no decode panels in this bench anymore). Equivalent properties
are asserted in test_milestone_gqa_single_kv_group_prefill_panel.py
(64 dma_writes, 896 ipcq_copy) and test_milestone_gqa_decode_4cases.py
(1 dma_write at cube 6, 189 ipcq_copy).
- tests/attention/test_milestone_gqa_single_kv_group_prefill_panel.py:
drop test_existing_prefill_panel_runner_backward_compat (it exercised
multi_user_prefill_gqa which no longer exists).
- scripts/paper/paper_plot_gqa.py: replace the 4 legacy _LABELS entries
with the single single_kv_group_prefill_gqa_c8_p8 label.
- Regenerate 1H_milestone_output/gqa_headline/sweep.json from the new
panel set.
Verification: 9/9 tests pass across the 3 affected test files.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New `/paper` slash-command skill that synthesizes ADR/SPEC content and live
KernBench benchmark results into a sectioned LaTeX technical paper compiled
to PDF with Tectonic (auto-installed). The skill negotiates a TOC, grounds
every number in committed artifacts or fresh bench runs, and keeps
report-only benches isolated.
This commit also includes the first generated report:
- docs/report/1H-codesign-paper/ — main.tex + per-section .tex, figures,
toc.md contract, and the built 8-page main.pdf. Covers the platform
(source-level kernels, latency model + accuracy, HW config from
topology.yaml), GEMM via composite command, All-Reduce via PE_IPCQ, and
fused GQA combining both, plus discussion/conclusion/2H future work.
- scripts/paper/ — isolated report harnesses (not registered benches):
paper_gqa_latency.py harvests per-panel GQA end-to-end latency + engine
occupancy (the milestone only emitted op-counts); paper_plot_gqa.py
renders the GQA figures.
GEMM/All-Reduce reuse committed milestone figures/CSVs; GQA results are
generated fresh. Honest flags retained: PE_CPU dispatch cost is 0 in this
config, and the proposed two-composite softmax_merge decode is marked
designed-not-measured.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mukesh
ywkang