End-to-end wires C=8 P=8 d_head=128 prefill with snake-ring inter-CUBE
SFR, intra-CUBE PE-SP (all 64 ranks active), and the milestone bench
panel. Decode kernel gains lrab-adapted center-root reduce for the
2×4 sub-mesh per ADR-0060 §4.2.
Increment 1 — SFR multi-row snake
src/kernbench/ccl/sfr_config.py: configure_sfr_intercube_ring gains
submesh_shape / submesh_origin kwargs; installs a Hamiltonian snake
ring through a rectangular sub-mesh (every hop is 1-hop physical
neighbour). Backward-compat: 1D-row behaviour preserved when
submesh_shape is None.
tests/test_intercube_snake_ring.py (12 tests)
Increment 2 — Decode lrab-adapted center-root reduce
src/kernbench/benches/_gqa_attention_decode_long.py: new sub_w param
(default 0 = existing 1D-chain). sub_w >= 2 selects the ADR-0060
§4.2 prescribed lrab-adapted Phase 1+2 reduce (bidirectional row +
bidirectional col converge to the center cube), with log-sum-exp
_merge_running replacing the plain + of lrab.
tests/attention/test_gqa_decode_long_2d_reduce.py (4 tests)
Increment 3 — Prefill kernel at C=8 (no production change)
Verified by inspection that the existing prefill_long kernel +
Increment 1's snake SFR already work at C=8 without any kernel
edit. The kernel speaks logical W/E; the snake routes it.
tests/attention/test_gqa_prefill_long_c8_snake.py (3 tests)
Increment 4 — Intra-CUBE PE-SP in prefill (all 64 ranks)
src/kernbench/benches/_gqa_attention_prefill_long.py: new P param
(default 1 = existing PE-0-only). P > 1 splits T_q query-axis-wise
across the P PEs of each CUBE; output rows are disjoint per PE so
no intra-CUBE reduce is needed; each PE drives its own same-lane
ring (P parallel rings).
tests/attention/test_gqa_prefill_long_pe_sp.py (5 tests)
Increment 5 — LLaMA-scale milestone bench panel
src/kernbench/benches/milestone_gqa_headline.py: new panel
single_kv_group_prefill_gqa_c8_p8 (C=8, P=8, T_q=S_kv=32K,
d_head=128). _run_prefill_panel extended with P/T_q/d_head
defaults; routes snake SFR when C > mesh_w.
tests/attention/test_milestone_gqa_single_kv_group_prefill_panel.py (3 tests)
Total: 4 production files modified, 5 new test files, 27 new tests.
Followed the Phase 1/2 protocol per CLAUDE.md throughout.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Land the new GQA fused-attention kernels (ADR-0060) for prefill/decode
across long and short context, the TL discipline primitives they depend
on (ADR-0062 lazy load, ADR-0063 scratch_scope + copy_to), and the
per-op-type CPU issue cost model (ADR-0064). Remove the pre-ADR-0060
mesh-attention baseline now that the unified kernels supersede it.
ADR-0060 (long context)
- _gqa_decode.py: M-fold + 2-level chain reduce-to-root (Level-2
intra-CUBE row-then-col + Level-1 inter-CUBE) — root-only output.
- _gqa_prefill.py: head-parallel + Ring KV rotation around C CUBEs,
online-softmax merge per ring step, per-CUBE distributed output.
- Each merge stage wraps in scratch_scope() and persists running
(m, l, O) via copy_to() to lift the 1 MiB scratch ceiling.
ADR-0060 §B.split.2 (short context, kv_per_cube in {1,2,4,8})
- _gqa_decode_short.py / _gqa_prefill_short.py: no cube-SP; each CUBE
owns whole KV heads; PE-parallel heads with intra-group chain
reduce. Prefill has no Ring KV (each head fully resident).
ADR-0062 (lazy tl.load): future-bearing TensorHandle, auto-wait at
first consuming op (dot/MATH/store/send/copy_to/composite).
ADR-0063 (tl.scratch_scope + tl.copy_to): scoped per-tile arena with
copy_to writeback primitive for persistent running state.
ADR-0064 (CPU issue cost model)
- common/cpu_issue_cost.py: per-op-type table (composite=40 ns,
primitives=5 ns); ratios are load-bearing per D1.
- TLContext: issue_cost_table param; _emit_dispatch_overhead(kind)
consults table with dispatch_cycles fallback (ADR-0046 §D6
back-compat).
- Live PE_CPU paths (greenlet + legacy) construct TLContext with
DEFAULT_CPU_ISSUE_COST so saturation lever (ADR-0060 §1) is
measurable end-to-end.
P7 headline bench: milestone-gqa-headline writes per-panel
op_log_summary to 1H_milestone_output/gqa_headline/sweep.json. No
figure renderers yet (deferred).
Removals (pre-ADR-0060 baseline now superseded):
- benches: _attention_mesh_kv.py, _attention_mesh_mlo.py,
_attention_mesh_mlo_2d.py, milestone_gqa_llama70b.py
- tests: test_attention_*, test_mesh_*, test_milestone_gqa_llama70b
- topology: llama70b_4sip.yaml (only consumer was the deleted diag)
- artifacts: 1H_milestone_output/gqa/ (sweep.json + 5 PNGs)
- tests/gqa/ plot helper + test (broken on Windows Tcl/Tkinter)
- ADR-0060/0061 references to deleted file paths cleaned up
(EN + KO kept in sync).
Tests: 124/124 focused regression green (attention + Phase E + TL
discipline + triton_emu + pe_components). Full regression: 764 pass,
2 pre-existing test_bench_registry failures (stale EXPECTED_NAMES
across multiple benches, not introduced here).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The single_user_prefill/single_user_decode panels of milestone-gqa-llama70b
(landed in e748a62) import configure_sfr_intracube_pe_ring from
kernbench.ccl.sfr_config, but the function definition was left in the
local working tree and never made it into that commit. Because the
bench registry eager-imports every bench module at CLI startup, the
missing symbol breaks `kernbench` entirely and cascades into 7 test
failures + 18 collection errors for anyone on a fresh checkout.
Installs an 8-PE logical ring inside every cube on every SIP (E/W
direction labels, no intercube or inter-SIP edges). Mutually exclusive
with configure_sfr_intercube_multisip on the same engine — the bench
picks one per panel-run (ADR-0058 D2).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mesh_2d, torus_2d, and mesh_2d_no_wrap accept optional w,h kwargs;
sqrt fall-back preserved for square layouts (back-compat tests
confirm 4-SIP and 9-SIP square configs still work). sfr_config
reads system.sips.w/h from spec and threads dims through to the
topology fn.
test_allreduce_multidevice CONFIGS switched from 4 SIPs (square)
to 6 SIPs: ring_1d_6sip, torus_2d_6sip_2x3, mesh_2d_no_wrap_6sip_2x3.
_write_temp_configs writes system.sips.w/h when supplied;
_sip_topo_dims reads them back. Latency sweep loop also moved to
6-SIP layouts. Linear-scale plot variants dropped -- only log-scale
*.png + summary.csv emitted. Plots in tests/allreduce_latency_plots
regenerated.
New tests/test_sip_topology_rectangular.py asserts neighbor
correctness for 2x3 layouts and back-compat for square fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mukesh