Research artifact (NOT wired into the production sweep) for the decode composite-vs-primitive investigation. Models a primitive decode kernel that hand-blocks each tl.dot into 16x16x16 GemmCmds, charging per-MAC-block PE_CPU dispatch -- testing whether faithfully charging the dispatch that the composite form offloads to PE_SCHEDULER flips the 'composite gives no decode latency benefit' result.
Finding: it does NOT. Even with up to 512 serialized blocking GemmCmds per matmul (vs 2 coarse tl.dot), end-to-end decode latency is unchanged (8192: 28.9us, 32768: 114.9us) -- the inter-cube (m,l,O) reduce DMA tail dominates and the local-attention GEMM/dispatch sits in critical-path slack. Confirms the two-regime conclusion (24c7054): composite helps compute-bound prefill, not memory/reduce-bound decode.
Caveats: measured at S_kv 8K/32K (reduce-dominated); large-S_kv streaming regime extrapolated only. The -5.5% tiled<untiled is reduce-tail ordering noise, not fully nailed down.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complete the composite-command study across both attention regimes and
write up the result, resolving when the composite command helps latency.
Decode (memory-bound, T_q=1, M=8): the command form is latency-neutral —
the kernel is bound by KV streaming and the MAC array is near-idle, so the
composite's only benefit here is host-issue offload (O(n_tiles) -> O(1)
PE_CPU commands). Figure: gqa_decode_long_ctx_composite (latency flat,
command count saturates).
Prefill (compute-bound, large M=G*T_q tile-filling): add three command-form
variants of a single-rank FlashAttention prefill kernel
(_gqa_prefill_compute_bound: primitive / composite / composite_extended).
Here the composite's scheduler-internal per-tile DMA<->compute pipeline
keeps the MAC array fed while the primitive's blocking tl.dot starves it,
so composite wins on MAC utilization (68% flat -> 83%) and wall-clock
(19% faster at ctx=1024), with the margin growing with context (deeper
P.V reduction = more tiles to pipeline) — the compute-bound mirror of the
GEMM result in section 3. Figure: gqa_prefill_compute_bound (latency +
MAC util). Sweep wired as GQA_1H_SWEEPS=prefill_cb; tests cover structure,
e2e, and composite<primitive in the compute-bound regime.
The synthesis: composite has two benefits set by roofline position —
host-issue offload (always) and MAC-array feeding (compute-bound only).
Decode exercises the first, prefill both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add two command-form variants of the Case-6 (Cube-SP x PE-SP) long-context
decode kernel alongside the primitive baseline:
- composite: per-tile GEMMs issued as one coarse tl.composite over the
full S_local (PE_SCHEDULER tiles/streams K,V) -- O(1) PE_CPU commands
vs the primitive kernel's O(n_tiles).
- composite_extended: Q.K^T composite + softmax_merge recipe composite
(ADR-0065), folding the per-tile online merge + P.V.
Extract the shared two-level (m,l,O) reduce into _gqa_mlo_reduce and
refactor the baseline to use it (byte-equal, guarded by a 64-rank
command-stream digest test).
Engine fix: _make_compute_out omitted pinned=True, so an on-chip TCM
compute result consumed as a composite GEMM operand was scheduled as an
HBM DMA_READ of its bit-61 scratch address -> PhysAddrError in data mode.
Mark compute outputs pinned (resident), matching the composite
auto-output and recipe scratch handles. .pinned is read only by the
tiling DMA gate, so this is byte-equal for existing benches.
Add the composite sweep (emit-level PE_CPU dispatch to 1M + data-mode
latency to 128K), its plot, umbrella wiring (GQA_1H_SWEEPS=composite),
and tests (refactor guard, dispatch saturation, engine-fix, e2e).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- §6 GQA: rewrite long-context decode from 4-case to 6-case. Data
Placement Policy now presents the six placement options and their
intrinsic per-PE memory / per-token comm costs (no winner predicted);
the long-context subsection selects the best placement for that regime
(Case 6 ★, both-axes S_kv shard) from the measured 6-case sweep. Add
KV-sharding diagram + analytical budget/summary figures.
- Regenerate the 6-row decode sweep (milestone-1h-gqa) and the
sweep-dependent decode panels (latency/traffic/parallelism/memory) so
figures, prose, and sweep_decode.json are mutually consistent.
- §5 All-Reduce: add IPCQ design alternatives (architecture + decision
matrix) as design-rationale schematics (illustrative step-counts, not
measured) and the bench-generated topology diagram.
- §4 GEMM: rename "user-orchestrated/user-level" -> "kernel-orchestrated/
kernel-level" (orchestration runs in the kernel program vs the
scheduler-orchestrated composite); minor accuracy fixes (7.18 TFLOP/s
~10% below peak; ~781 ns DMA).
- Recompile build/main.pdf.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the unsourced 32 KB/layer placeholder in the analytical chart
with T*(N-1)/N where T = h_q * S_q * (d_head*2 + 8) ~ 2.1 KB
(per-PE (m, l, O) payload) and N is the participant count of the
reduce-only chain per stage. Cases 2/3/6 analytical bars now match
the simulator-measured numbers within ~10% (was 6-30x over).
Also:
- Bumps the measurement S_kv from 8 K to 64 K to verify Cases 1, 2, 3, 6
are genuinely S_kv-independent and Cases 4, 5 scale linearly.
- In-bar descriptor on the paired chart is now wrapped to fit the bar
width, black, no background box, and centered between the analytical
and measured bars so the label applies to both.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drops the per-case panel outline from linewidth 2.2 to 0.9 with 85% alpha.
The case-color frame still anchors each panel but no longer competes
with the swimlanes, flit packets, and annotations inside.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds two new IPCQ architecture views in the latency_model.png aesthetic:
a 2x2 flow figure (one per case) and a 4x1 stacked figure for taller
horizontal panels. Refactors per-case panel drawing into shared helpers
and disambiguates the flit labels (HMQ desc -> Hd so it does not
collide with data D). Two-row legend keys.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drops the control-data overlay from the paper figures folder and
adds the per-send latency cycle-stack PNG instead.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removes the (measured) comm column, widens the Notes column with
shorter wrapped text, scales the table figure down, and wraps the
heading to two lines so it sits flush with the table.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
6 generated figures comparing IPCQ (chosen) against the 3 HW alternatives
(Doorbell+Polling, HMQ, RDMA-CQ): architecture blocks, latency stack,
op counts, sequence flow, decision matrix, and control/data plane overlay.
Three figures also copied into the paper's figures/ folder for inclusion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous commit (8102ddb) renamed the figures-dir
gqa_decode_long_ctx_4cases_*.png → 6cases_*.png but only the rename
was staged; the cp that brought the regenerated 6-case content
landed after `git add`, so the committed bytes were still the old
4-case ones. This commit syncs the bytes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Filename cleanup so every long-ctx GQA artifact has a consistent
"gqa_long_ctx_6cases_*" prefix (or "gqa_decode_long_ctx_6cases_*"
for decode-only charts). Old "4cases" / mixed names retired.
Renames (long_ctx + figures, content unchanged):
gqa_hbm_budget.png -> gqa_long_ctx_6cases_hbm_budget.png
gqa_4cases_summary.png -> gqa_long_ctx_6cases_summary.png
gqa_4cases_memory_comm_analytical -> gqa_long_ctx_6cases_memory_comm_analytical.png
gqa_4cases_memory_comm_paired -> gqa_long_ctx_6cases_memory_comm_paired.png
gqa_kv_sharding_6cases_diagram -> gqa_long_ctx_6cases_kv_sharding_diagram.png
gqa_kv_sharding_6cases_table -> gqa_long_ctx_6cases_kv_sharding_table.png
gqa_3cases_measured_comm.json -> gqa_long_ctx_6cases_measured_comm.json
gqa_decode_long_ctx_4cases_*.png -> gqa_decode_long_ctx_6cases_*.png
(figures dir; long_ctx never had old)
New 4-chart 6-case set in long_ctx output dir (regenerated by
paper_plot_gqa_decode_long_ctx_4cases.py, which now reads all 6
sweep_decode.json panels — Cases 1-6 with the same colour scheme
used elsewhere: red = overflow per-PE HBM, grey = neutral, blue
= Pareto-best ★):
gqa_decode_long_ctx_6cases_latency.png
gqa_decode_long_ctx_6cases_memory.png
gqa_decode_long_ctx_6cases_parallelism.png
gqa_decode_long_ctx_6cases_traffic.png
Generator scripts updated to write the new filenames + handle the
two new d_head-TP variants (Cases 4, 5) in their per-PE memory and
active-PE-count helpers. Figure widths bumped 10 -> 12 in to fit 6
multi-line case labels.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two new long-ctx decode attention kernels for the d_head-TP sharding
variants the 6-case chart predicts:
· _gqa_attention_decode_long_ctx_cube_sp_pe_tp_dhead.py
Cube-SP × PE-TP-d_head (Case 4 in chart). Per cube holds
S_kv/C tokens of full d_head; per PE holds same tokens but
only d_head/P dims. Partial Q·Kᵀ scores reduced intra-cube
before softmax; outer (m,ℓ,O) merge two-phase (intra+inter).
· _gqa_attention_decode_long_ctx_cube_tp_dhead_pe_sp.py
Cube-TP-d_head × PE-SP (Case 5). Per cube holds full S_kv
with only d_head/C dims; per PE holds S_kv/P of those dims.
Partial scores reduced inter-cube (UCIe) before softmax.
Sweep dispatch (gqa_decode_long_ctx_4cases.py) extended with two
new panels so the milestone-1h-gqa sweep covers all 6 cases.
Smoke test scripts/verify_case4_dhead_tp.py runs Cases 4/5/6 at
S_kv=2K to validate the kernels load and execute end-to-end.
Plus the figure-generation toolchain that produced the committed
PNGs in the prior commit (dd3337f):
· paper_plot_gqa_4cases_summary.py - 3-panel summary +
2-panel (analytical / paired-measured) chart generator.
_plot_comm now takes mode="analytical" | "paired".
· paper_plot_gqa_kv_sharding_diagram.py - 6-case 2-D KV-tensor
diagram + companion comparison-table PNG.
· measure_gqa_decode_placement_comm.py - runs all 6 kernels at
S_kv=8K, sums actual IPCQ-copy bytes from engine.op_log,
scales partial-score AR ×128 to S_kv=1M, writes
gqa_3cases_measured_comm.json (committed).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Six PNGs covering the 6-case decode KV-placement story, copied
into both the kernbench 1H_milestone_output staging dir and the
1H-codesign-paper figures/ dir:
· gqa_hbm_budget.png - per-PE HBM budget bar
· gqa_4cases_summary.png - 3-panel: HBM + KV mem + comm
· gqa_4cases_memory_comm_analytical - 2-panel: KV mem + comm (analytical)
· gqa_4cases_memory_comm_paired - 2-panel: + measured overlay
· gqa_kv_sharding_6cases_diagram - 2-D KV-tensor view (Y=S_kv, X=d_head)
· gqa_kv_sharding_6cases_table - companion comparison table
The original 4-case gqa_4cases_summary.png is updated to the new
6-case 3-panel layout. Generator scripts and Phase-1 d_head-TP
kernels remain untracked - separate commit scope.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two strands bundled as the 1H-codesign-paper refresh unit:
(A) This session — single-op cost-model reflection (depends on 2d8271c):
- §2 Table 2 (tab:hw): split "FIXED per command" into "FIXED per
single-op command" (8 cycles) and "FIXED per composite command"
(40 cycles); §2 dispatch-overhead prose updated to the two-class split.
- §3.4 (sec:gemm-vs-async): rename paragraph headers + prose to
async-full / async-tiled; "atomic" -> "single-op" throughout; reframe
mechanism #3 from the old DMA-only fast-path to the single-op
fast-path. Headline narrative now: even with EVERY single-op cmd
(96 DMA + 48 dot + 47 add) charged the light 8-cycle FIXED, composite
still wins ~2.8x at K=3072 purely on command-count structure (1 vs
192 commands) -- down from the pre-D8 ~6.3x, and explicitly NOT a
modelling artifact. Numbers refreshed from the regenerated sweep:
async-full 3.83->3.91, async-tiled 1.14->~2.53, under-tile corner
1.06->1.21, depth-2 vs depth-inf spread <1%. New figure wired in.
- build/main.pdf rebuilt (tectonic); pdftotext-verified (no broken
refs; Table 2 split, single-op terms, 2.8x/2.53/192-host-commands
all present).
(B) Prior-session paper work riding along uncommitted: §4 all-reduce
deep-edit, §5 GQA, §6 discussion trims; milestone_1h_ccl.py plot
label "FSIM" -> "H2 2025 SW queue baseline"; regenerated diagrams
under docs/diagrams/** and gemm output PNGs under
1H_milestone_output/gemm/. (Composite-window gemm plots are
unaffected by D8 — D8 only changes single-op dispatch FIXED, which
the composite window excludes.)
All TODO items for the D8 single-op extension are now complete and
pushed across 3 commits (2d8271c cost-model+ADR+tests, 821bbf2 bench
harness, this paper refresh). Full regression green (826 passed, 1
skipped). No remaining work.
NOTE for review (carried from 2d8271c): ADR-0065's "2x CPU-offload win"
headline for GQA decode opt2 may want a refresh to the post-D8 ~1.87x.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the user-orchestrated async GEMM variants used in paper §3.4 to
contrast with the composite (load_ref) path:
- matmul-async : naive (tl.load full A+B, one tl.dot)
- matmul-async-chunked : depth-inf prefetch (all B-tiles queued)
- matmul-async-chunked-db : depth-2 double-buffer (TCM-bounded)
plus scripts/paper/paper_plot_gemm_async_vs_composite.py (4-way per-PE
TFLOPS sweep + figure) and the regenerated outputs under
1H_milestone_output/gemm/.
Sweep regenerated under the ADR-0064 D8 single-op cost model (commit
2d8271c). At K=3072 the composite-vs-async-tiled gap narrows from the
pre-D8 ~6.3x to ~2.8x (composite 7.18 / async-tiled 2.54 TFLOPS): D8
makes the async-tiled kernel's ~191 single-op dispatches 5x cheaper, so
its dispatch overhead drops to ~1.5us. Qualitative order persists:
composite > async-full (3.91) > async-tiled (2.54).
test_bench_registry.py: register matmul-async / -chunked / -chunked-db
(also picks up the earlier milestone-gqa-headline -> milestone-1h-gqa
rename already present in the tree).
--- Remaining work (resume here if interrupted) ---
- Paper §3.4 (03-gemm.tex sec:gemm-vs-async): finish naive->async-full /
chunked->async-tiled rename AND reframe "FIXED_DMA=8 for DMA descriptors"
to single-op(8) vs composite(40). Figure already copied to
docs/report/1H-codesign-paper/figures/gemm_composite_vs_async_tflops.png.
- Paper §3.4 K=3072 para + mechanism #3: update dispatch breakdown to new
model (191 single-op * 8c ~= 1.5us, was 4.6us) and headline gap
~6.3x -> ~2.8x.
- Rebuild build/main.pdf (tectonic) + verify via pdftotext.
- Then commit Group 3 (paper + diagrams + PDF).
- Table 2 / §2 prose / ADR-0064 D8 already done in 2d8271c.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0064 D8 broadened from "DMA fast-path" to "single-op-cmd fast-path":
every single-op command (DmaRead/DmaWrite/Gemm/Math/Copy) now pays the
lighter FIXED=8; only CompositeCmd keeps the 40-cycle control-path FIXED
(it alone needs scheduler plan generation + per-tile RW-hazard tracking +
completion wiring). Renamed knob fixed_per_dma_cmd_cycles ->
fixed_per_single_op_cmd_cycles. dispatch_cycles now branches on
"is CompositeCmd" rather than enumerating DMA types.
Term choice: "single-op" (not "atomic", which read as sync/async) — the
axis is composition (one engine op vs fused multi-op plan), orthogonal to
timing. single-op <-> composite.
Tests: test_pe_cost_model.py updated to the single-op surface (defaults,
fast-path over all 5 single-op cmd types, composite general path, yaml
override). All green.
Recalibrated tests/attention/test_gqa_decode_opt2.py
::test_opt3_dispatch_exceeds_opt2 — NOT a regression: D8 makes single-op
cmds 5x cheaper, so opt2's two-composite fusion win over opt3's many
single-ops narrowed from pre-D8 ~3.7x to ~1.87x (opt3=224 > opt2=120).
The CPU-offload invariant (opt2 cheaper) still holds; only the model-
dependent ">2x" constant was over-fit to the old uniform-40 model. Gate
now: direction + >1.5x margin (matches sibling R-sweep test's stated
"absolute ratio informative-only" philosophy).
NOTE for review: ADR-0065's "2x CPU-offload win" headline may want a
refresh to reflect the post-D8 ~1.87x — left to user (architectural doc).
Full regression: 826 passed, 1 skipped (tests/ excl. tests/gemm).
--- Remaining work (resume here if interrupted) ---
5. Re-run scripts/paper/paper_plot_gemm_async_vs_composite.py with new
cost model; verify async-tiled dispatch overhead drops (~4576ns ->
~1536ns expected) and the composite-vs-async-tiled gap narrows from
the prior ~6.3x at K=3072.
6. Copy regenerated gemm_composite_vs_async_tflops.png to
docs/report/1H-codesign-paper/figures/.
7. Paper §3.4 (03-gemm.tex sec:gemm-vs-async): finish naive->async-full /
chunked->async-tiled rename AND reframe FIXED_DMA wording to single-op
vs composite (currently still says "lighter FIXED for DMA descriptors,
FIXED_DMA=8"). Table 2 (02-platform) + §2 dispatch prose already done.
8. Paper §3.4 K=3072 corner para + mechanism #3: update dispatch breakdown
to new model (96 DMA*8 + 95 single-op*8 ≈ 1.5us vs old 4.6us); update
headline ratio if it changed.
9. Rebuild docs/report/1H-codesign-paper/build/main.pdf (tectonic) +
verify via pdftotext.
10. Then this is the bench-harness + paper commits (Groups 2 & 3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five-stage chain was set as a centered display equation, which
overflowed the column and clipped the last 'DMA_WRITE' label. The
inline prose right below it also repeated the same chain in a
shorter form. Consolidate into one inline parenthetical listing the
five stage names, and reuse 'flows through the chain' for the
self-routing description — no clipping, no repetition.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The matplotlib title and gray subtitle were duplicating what the
figure caption already says in the LaTeX source, and crowding the
top of every chart. Drop both — chart now leaves the title/subtitle
slots blank and the caption is the single source of truth for
chart titling.
Re-render all 5 GEMM figures (mac util measured, mac util theoretical
vs measured, stage breakdown, HBM BW util, per-PE TFLOPS) and resync
into docs/report/1H-codesign-paper/figures/. No data changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new evaluation plots for §3 GEMM, both using the composite window
as the denominator (the up-front tl.load of A in load_ref is therefore
excluded — only HBM traffic and compute inside the composite count):
- gemm_hbm_bw_util.png: per-PE HBM bandwidth utilization against the
256 GB/s per-PE ceiling. Answers "is this workload memory-bound on
this configuration?" — small/single-tile shapes sit at 22–46%
(pipeline-fill-limited), large or output-write-dominated shapes
saturate (≥85%).
- gemm_per_pe_tflops.png: per-PE GEMM throughput against the 8 TFLOP/s
engine peak. The deep-K K=3072 load_ref case lands at
7.18 TFLOP/s (~90% of peak) — the configuration's clean win.
ref_ref at the same shape drops to 3.83 TFLOP/s because the second
operand doubles HBM pressure and the kernel hits the BW ceiling.
The variant gap on this chart is the operational cost of NOT
pre-staging the activation.
Both charts compare two operand-staging variants on every shape:
load_ref ("activation pre-staged", weight-only HBM streaming) and
ref_ref (both A and W streamed from HBM). load_load is omitted —
with both operands pre-staged the composite carries no HBM traffic
and the BW metric collapses to 0.
Analytic / measured peak unified to the single hardware spec
(8.0 TFLOP/s = 8000 flops/ns). T_STAGE is now derived as
tile_flops / peak (= 16.384 ns for the 32×64×32 tile) rather than
the previous hardcoded 16 ns, so MAC efficiency and per-PE TFLOPS%
share the same denominator and never disagree. Side effect: max
analytic-vs-measured gap tightens from 2.2 ppt to 1.4 ppt across
the sweep.
§3 Results restructured: lead with the two new evaluation plots
(BW saturation, achieved throughput), then the MAC-utilization
analytic-vs-measured chart serves as the validation step, then the
per-stage engine wall-clock as the supporting diagnostic. §3
Analysis updated to use the load_ref / ref_ref contrast as the
hardware-software boundary line that motivates the GQA kernel of §5.
§2.4 Accuracy claim updated to match: GEMM analytic-vs-measured
agreement is now "within 1.4 ppt across every swept shape", from
~7.7% at single-tile up to ~90% at K=3072. Subtitle/title vertical
stacking glitch in the matplotlib charts fixed in passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three coupled fixes that recover small-tile GEMM pipeline efficiency
from 53% to 88% (32x3072x32 load_ref, composite_window basis).
1. PE_DMA channel-hold (ADR-0014 D4 clarified): both the
_handle_with_hooks (PeInternalTxn) and _pipeline_process
(TileToken) paths used to hold the cap=1 DMA channel through the
full HBM round-trip, which double-serialized with the HBM_CTRL's
own per-PC `available_at` model and prevented back-to-back tile
DMAs from amortizing their per-request head latency. Channel is
now released after the request is enqueued onto the next hop;
HBM serialization is HBM_CTRL's responsibility alone.
Tests: new test_pe_dma_back_to_back_pipelining as the oracle
(asserts wall < 75% of strict-serialized N x single_op). Existing
test_pe_dma_record_start_after_channel_acquire rewritten to assert
t_start clustering (channel released fast) instead of the old
round-trip-hold invariant. test_pe_dma_same_channel_serializes
still passes — HBM_CTRL preserves ordering.
Probe regression: PE→local-HBM 32 KiB stays at 141 ns
(single-request, unaffected).
2. milestone_1h_gemm bench: matmul_composite was reading MATMUL_M/K/N
env vars at module load, so every sweep row replayed the cached
256³ result; values now read inside run(). Drops the stale
sys.modules deletion hack.
3. Analytic ideal-pipeline model: dropped the (n_mn-1)·dma_w_per_pair
penalty (over-pessimistic for under-tile shapes — it pushed
measured > theoretical) and replaced the D_STAGES-derived head
with empirical T_PIPELINE_FILL=60 ns / T_PIPELINE_TAIL=30 ns.
Max analytic-vs-measured gap across all 7 swept shapes now 2.2 ppt
(was 9-44 ppt under the old constants).
Paper updates:
- §3 (GEMM): 78%→88% measured at 48 tiles, 23%→15% at 1 tile,
stage breakdown numbers refreshed (DMA in / Fetch / GEMM all
~785 ns at K=3072), analytic-vs-measured agreement tightened
to "within 2.2 ppt".
- §2.4 (Accuracy): GEMM tracking claim refreshed accordingly.
- §5 (GQA): restore long-ctx 4-cases figures into figures/
(they were dropped from bench output dir as derived artifacts in
92b9221 / e45626c but §5 still cites them by name).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These 8 prefill/decode 4-case PNGs are derived artifacts regenerated
on demand from scripts/paper/paper_plot_gqa_{prefill,decode}_long_ctx_4cases.py
and no longer belong in the committed bench output.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Analytical plot generator + PNG covering the 4 sharding cases from
slide 17 plus the 2 d_head-TP variants (Cases 4, 5). Three panels:
per-PE HBM budget breakdown (Wq + Wk + Wv + Wo + 4.24 GB KV
headroom), per-PE KV memory at S_kv=1M, and per-PE communication
per output token (decode). Numbers match slide 17 (max KV context
~7.1M for the 64-way sharded cases).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single @bench entry that drives the prefill + decode long-context
4-cases sweeps in one invocation, mirroring milestone-1h-gemm. The
individual gqa_helpers/long_ctx/gqa_{prefill,decode}_long_ctx_4cases
modules are now pure helpers (no @bench), reached only via this
umbrella.
Env var contract:
GQA_1H_RUN=1 (required gate)
GQA_1H_SWEEPS=prefill,decode (default: both; pick subset to skip)
GQA_1H_TOPOLOGY=topology.yaml (override)
Output layout:
benches/1H_milestone_output/gqa/long_ctx/
sweep_prefill.json
sweep_decode.json
gqa_prefill_long_ctx_4cases_{latency,traffic,memory,parallelism}.png
gqa_decode_long_ctx_4cases_{latency,traffic,memory,parallelism}.png
The decode bench config drops S_kv from 131_072 to 8_192 so the umbrella
finishes in minutes — the comparative-story bars are the same shape at
8K. The 131K production headline number is recoverable by overriding
the dispatch; the helper docstring notes this.
Outputs (2 sweep JSONs + 8 PNGs) ship in this commit alongside the
umbrella that produced them, following the milestone-1h-gemm pattern
where derived artifacts live with the bench that emits them.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirrors the existing decode 4-cases study for prefill on the LLaMA-3.1-70B
single-KV-head group (h_q=8, h_kv=1, d_head=128) across 8 cubes × 8 PEs:
Case 1 Cube-SP × PE-TP → KV S_kv-Ring across cubes; Q/O T_q-split 64-way
Case 2 Cube-Repl × PE-TP → full KV per cube; only CUBE 0's P PEs split T_q
Case 3 Cube-Repl × PE-SP → full KV per cube; PEs SP on S_kv (intra-cube AR)
Case 4 Cube-SP × PE-SP → KV split 64-way + Q T_q-split across cubes
(Ring KV + per-cube intra-CUBE AR) ★ optimal
Cases 2 and 3 use Q-axis tiling (TILE_Q) so the per-PE scratch is bounded
by TILE_Q × TILE_S_KV regardless of T_q (the full Q tensor would be
G·T_q·d_head·2 bytes which scales linearly with T_q and blew the 1 MB
budget at T_q≥128 without tiling).
Per-panel try/except in run_sweep tolerates per-panel failures so
sweep.json always lands with whichever cases succeed plus a failures
list. Case 4 uses configure_sfr_intercube_ring with the snake submesh
(2,4) which installs both the Ring E/W lanes and the intra_* lanes
needed for the intra-CUBE reduce — single SFR config for all 4 cases.
The plot script generates 4 PNGs (latency, traffic, memory, parallelism)
into 1H_milestone_output/gqa/long_ctx/. The parallelism chart shows
total compute work (active_PE × T_q_per_PE × S_kv_processed) — exposes
Case 3's 8× redundancy vs. Cases 1/2/4 which all do unique work.
gqa_prefill_long_ctx_4cases.py exposes run_sweep() rather than a @bench
decorator — the umbrella bench in the next commit invokes it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Splits the GQA helpers into a dedicated subpackage to make room for the
prefill 4-cases study (next commit) and a single umbrella bench
(milestone-1h-gqa, after that).
Layout:
benches/gqa_helpers/
long_ctx/ — decode 4-cases kernels + sweep runner
short_ctx/ — prefill/decode short-context kernels
shared/ — _gqa_panel_helpers + decode_opt2 (context-agnostic)
The registry audit now skips subpackages so gqa_helpers/ (without a
leading underscore) doesn't get audited for @bench decorators.
Also drops the legacy milestone-gqa-headline bench, its
_gqa_attention_prefill_long kernel, 6 dependent prefill tests, the
paper_gqa_latency.py report harness, and the 3 stale headline-derived
PNGs the §6 wire-up referenced (paper will re-pull from the new
1H_milestone_output/gqa/long_ctx/ once §6 is updated).
The _ccl_cfg and _summarize_op_log helpers used to live in the
headline bench; extracted them to gqa_helpers/shared/_gqa_panel_helpers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Probe table (Table 1, tab:probe-pe-dma) promoted to table* so it
spans both columns and no longer gets clipped at the column edge.
- Fig 1 (hw-arch) row 1 (SIP, CUBE subfigures) trimmed: widths
0.495 -> 0.42 textwidth and a height=0.6\linewidth cap added so
the figure occupies less vertical space and floats can land
earlier in the document.
- Fig 1 and Fig 2 source blocks relocated to the top of §2 so they
enter LaTeX's float queue before any §2 body text. As a result
Fig 1 now lands on page 3 and Fig 2 on page 4 (was 4 and 5).
- Table 2 (tab:hw, modeled hardware config) reverted to plain
\begin{table}; the previous \begin{table*} reshape was not
requested.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- subfig widths 0.46 -> 0.495 textwidth so the \hfill gap between
(a) SIP and (b) CUBE shrinks to ~2% textwidth instead of ~8%.
- Removed the height=0.85\linewidth cap on both subfigures; with the
cap, the (roughly 1:1) SIP and CUBE diagrams were being scaled
down to 85% even when the column had room. Let them render at
their natural aspect ratio.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§2 platform:
- Accuracy promoted from \paragraph to \subsection (sec:accuracy);
it sits at the same heading level as Why / Device / Latency / HW
config and so reads as a first-class component of the platform
description rather than a tail-end footnote.
- Three architecture diagrams (SIP, CUBE, PE) collapsed into one
figure* using subcaption: Fig.~\ref{fig:hw-arch}(a) SIP and (b)
CUBE side-by-side on row 1, (c) PE wide on row 2, total height
capped at ~40% of the page. Subfigure cross-references rewritten
to Fig.~ref{fig:hw-arch}\subref{...} in the body text. main.tex
now pulls in the subcaption package.
- Table 1 (modeled hardware configuration) promoted to table*
(two-column / full page width) and the row layout rewritten as a
4-column tabular so Hierarchy + PE + Command-issue sit on the
left and Memory + Interconnect sit on the right. The previous
single-column rendering was getting cut at the right edge of the
printed column.
§6 GQA:
- Removed the historical four-panel headline table (tab:gqa), the
two figures (fig:gqa-lat, fig:gqa-break), and the prose paragraph
that cited their per-panel numbers. Reason: the underlying
milestone-gqa-headline bench has been simplified in collaborator
commit 65c365f ("drop misleading single_user_/multi_user_
panels") and no longer reproduces that data, so the section was
left referencing a dataset the current bench cannot regenerate.
- The previously-added 4-cases long-context decode comparison is
now §6's only Results subsection, retitled to
"Results: long-context decode and parallelism strategies".
The headline GQA result that survives into the paper is therefore
the parallelism trade-off study (Case 4 / Cube-SP × PE-SP at 34 us
buying an 8x KV-memory reduction over the latency leaders).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The collaborator commit 7c346de added the comparative figures
(gqa_decode_long_ctx_4cases_{latency,memory,traffic}.png) but the
fused-GQA section still only referenced the four headline panels.
This commit closes that loop:
- New §6 subsection "Long-context decode: parallelism strategies"
added between Results and Analysis. It lays out the four
parallelism strategies (Cube-SP/Repl x PE-TP/SP) and pulls in the
three new figures.
- The discussion highlights the central trade: the fastest strategy
(Case 3, Cube-Repl x PE-SP at 20.2 us) requires replicating the
full KV cache to every CUBE, while Case 4 (Cube-SP x PE-SP, the
chosen design marked *) gives back ~14 us in exchange for an 8x
KV-memory reduction. Case 4's ~190 IPCQ copies + ~190 DMA reads
are precisely the on-device collective traffic PE_IPCQ and the
torus links of §5 are provisioned to absorb -- a direct payoff
of the communication-side codesign work.
- Connects back to §5 (PE_IPCQ / all-reduce) so the reader sees the
capstone arc: the GEMM enabler exposes the data-movement bound,
the communication enabler attacks it, and the long-context
parallelism study shows how the choice between the two extremes is
framed by KV memory vs. on-device collective traffic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§2 platform:
- Add a "memory-centric AHBM" sentence to the device-and-execution
intro: each PE is paired with dedicated HBM bandwidth and on-PE
TCM, so the performance question is about feeding compute from
locally-attached memory + moving the unavoidable inter-PE traffic
efficiently. Makes the AHBM character of the platform visible
before §2.3 starts unpacking the latency model.
- Promote "Congestion and contention modeling" from \paragraph to
\subsubsection: this is the platform's core differentiator over a
peak-BW roofline, so it deserves its own heading.
- Rename "Control-plane (issue) cost model" to "Command dispatch
overhead model" -- describes what it actually charges (the PE_CPU
paying a fixed + per-byte cost to push a command to one of the
accelerator engines) without the more abstract "control-plane"
framing.
- Add a third Accuracy cross-check from kernbench probe: a
PE→HBM DMA-distance sweep that confirms monotonic hop progression,
bandwidth saturation matching the per-edge model, and the built-in
invariants (D2H >= H2D, cross-CUBE best < worst). New
Table~\ref{tab:probe-pe-dma} reports per-traversal latency and
utilisation at 32KiB / 1MiB across five hop classes.
- Captured probe output as figures/probe_pe_dma_summary.txt for
reproducibility.
§5 PE_IPCQ / all-reduce:
- Re-ran milestone-1h-ccl on current sim_engine (post-ADR-0064 Rev2
and the IPCQ slot-wrap Phase-2 race fix). Updated the topology-
comparison table and the buffer-kind caption to the fresh raw
latencies. Headline ratios are preserved:
torus vs mesh saving at 96 KB/PE: 24.8% (was 24.8%) -> ~25%
torus vs ring saving at 96 KB/PE: 19.3% (was 19.3%) -> ~19%
TCM vs HBM saving at 64 KB/PE: 13.4% (was 13.9%) -> ~13%
TCM vs SRAM saving at 64 KB/PE: 37.2% (was 38.3%) -> ~37%
The Executive Summary's "up to ~14%" / "up to ~38%" framing stays
consistent with these post values.
Artifacts refreshed in src/kernbench/benches/1H_milestone_output/:
- ccl/summary.csv + per-topology PNGs + buffer-kind CSV/PNG
- ccl/comparison_mesh_vs_ring_vs_2DTorus_vs_theoretical_vs_fsim.png
- gemm/* (re-run yields identical pe_window structure; PNGs
refreshed)
Paper figures synced to the fresh artifacts:
- figures/allreduce_comparison.png
- figures/allreduce_buffer_kind.png
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Generated by scripts/paper/paper_plot_gqa_decode_long_ctx_4cases.py
from the bench's sweep.json (S_kv=8192, the headline 128K run was
sidestepped — Case 2's single-PE 128-tile sweep is the long pole
and shape/ordering is preserved across scales).
gqa_decode_long_ctx_4cases_latency.png end-to-end latency / case
gqa_decode_long_ctx_4cases_traffic.png ipcq / dma op-count breakdown
gqa_decode_long_ctx_4cases_memory.png KV bytes per cube (slide-11)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bench changes:
- new _end_to_end_ns(op_log) and _engine_occupancy_ns(op_log) helpers
in milestone_gqa_decode_long_ctx_4cases.py (mirror paper_gqa_latency.py)
- _run_panel return dict now carries latency_ns + engine_occupancy_ns
alongside op_log_summary, so sweep.json is the single source of truth
for the comparative figures
Plot script:
- new scripts/paper/paper_plot_gqa_decode_long_ctx_4cases.py reads
sweep.json and emits 3 PNGs to docs/report/1H-codesign-paper/figures/:
gqa_decode_long_ctx_4cases_latency.png (end-to-end latency / case)
gqa_decode_long_ctx_4cases_traffic.png (ipcq/dma op counts / case)
gqa_decode_long_ctx_4cases_memory.png (KV bytes per cube / case)
Test changes:
- 2 new tests verifying the helpers + _run_panel dict shape
- lower smoke S_kv from 8192 -> 2048 (4x faster Case 2; assertions
are S_kv-independent; one fold-loop iteration preserved)
18/18 tests pass in ~4 min.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Reordered §2 subsections to follow the SIP → CUBE → PE → graph
flow: Why KernBench → Device and execution model → Latency model
→ Modeled hardware configuration. Readers now meet the device
hierarchy before the graph abstraction that re-uses it.
- §2.2 Device and execution model: starts with the SIP/CUBE/PE
hierarchy paragraphs (each anchoring fig:sip-arch, fig:cube-arch,
fig:pe-arch); then the runtime-API/sim-engine/components bullet
list; then the atomic-vs-composite command distinction (corrects
the prior over-narrow framing that read every PE command as
composite -- atomic single-engine commands exist too, and PE_CPU
itself runs control-plane work directly).
- §2.3 Latency model: opens with the four-contribution decomposition
(per-node overhead, per-edge transmission, drain, queuing delay)
and the latency_model schematic; retains existing The hardware as
a graph / From graph to DES / Latency contributions / Congestion
/ Control-plane cost model / Accuracy paragraphs. Accuracy
paragraph now closes on KernBench's sufficiency for *relative*
HW/SW design trade-offs given analytic + external-simulator
agreement.
- New figures and assets:
- figures/sip_architecture.pdf (SIP-level graph view)
- figures/cube_architecture.pdf (CUBE-level zoom-in)
- figures/latency_model.png (conceptual latency-model
schematic with per-node /
per-edge / drain / queuing-delay
colour coding)
- figures/pe_architecture.png (carried over)
- Source-of-truth generator for the latency schematic:
scripts/paper/paper_latency_model_diagram.py (a report-only
harness under scripts/paper/ per the /paper isolation rule).
- main.tex preamble: \usepackage{tikz} added (kept from prior
sequence-diagram draft -- harmless now that the latency model is
a PNG; left in to keep paragraph numbering stable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Option B fold: the lrab math previously in
``_gqa_attention_decode_long.py`` (used only as a thin re-export by
the Case 4 wrapper and as the import source for ``_merge_running``
in Cases 1 / 3) now lives inline in the Case 4 kernel file. ``sub_w``
is hardcoded to 4 (the 4×2 cube sub-mesh geometry; root cube 6);
the legacy ``sub_w=0`` 1D-chain backward-compat path is removed
along with its dedicated tests — the dropped ``single_user_*`` /
``multi_user_*`` panels that exercised it are already gone.
Production changes:
- inline lrab math into _gqa_attention_decode_long_ctx_cube_sp_pe_sp.py
(drops sub_w param; _ROOT_CUBE=6 baked in)
- inline _merge_running into Cases 1 (cube_sp_pe_tp) and 3
(cube_repl_pe_sp) so they no longer depend on the deleted file
- delete src/kernbench/benches/_gqa_attention_decode_long.py
- remove dead _run_decode_panel / _DECODE_* constants /
decode-side _PANEL_DISPATCH / _make_bench_fn decode branch
from milestone_gqa_headline.py (only the prefill panel remains)
- update _gqa_attention_decode_opt2.py docstring reference
Test changes:
- delete 7 legacy test_gqa_*.py files that pre-dated the 4-cases
architectural split (coverage now subsumed by the 16 4-cases tests)
- remove test_opt2_matches_opt3_data_mode + _run_decode_data helper
from test_gqa_decode_opt2.py (the parity check required sub_w=0
which no longer exists; opt2's other 4 tests preserved)
20/20 tests pass (16 4-cases + 4 opt2 smoke/dispatch).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cases 1-3 each have a dedicated _gqa_attention_decode_<case>.py
kernel file; Case 4 previously reached into _gqa_attention_decode_long.py
via the headline bench's _run_decode_panel helper, breaking the
one-file-per-case convention. Adds _gqa_attention_decode_cube_sp_pe_sp.py
as a 20-line wrapper that bakes in sub_w=4 (the C=8 lrab geometry)
and gives Case 4 its own kind ("decode_cube_sp_pe_sp") and helper
(_run_decode_panel_cube_sp_pe_sp). decode_long.py is unchanged
(still serves the legacy decode_long tests). 16 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the fourth and final 4-cases panel: KV split S_kv-wise across
the 8 cubes (cube=row_wise, S_local = S_kv/C), replicated within
each cube (pe=replicate). PE-TP at B=1 means only PE 0 of each cube
has work; PEs 1-7 early-return (slide-11 PE-TP-at-B=1 waste). No
intra-CUBE comm; inter-CUBE 8-way reduce reuses the lrab-adapted
center-root pattern (root cube 6) — same structural cost as Case 4's
inter-CUBE phase (21 ipcq_copy). 16 tests pass (4 per case).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the third 4-cases panel: KV replicated per cube (8× memory
waste), PEs SP on S_kv within each cube, intra-CUBE 8-way reduce on
(m, ℓ, O), and no inter-CUBE comm (every cube ends with full answer;
designated writer = cube 0). Reuses the row-chain + col-bridge
intra-CUBE pattern that anchors Case 4 (21 ipcq_copy per cube × 8
cubes = 168 total). 12 tests pass (4 Case 4 + 4 Case 2 + 4 Case 3).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Second case in the GQA decode 4-cases comparative study per
GQA_full_deck.pptx slide 11. Case 2 replicates K, V across all 8
cubes × 8 PEs (the slide-11 8 KB/tok/PE memory waste) and has zero
inter-rank comm. For B=1 (single-user decode default), only PE 0
of CUBE 0 has work — the inherent PE-TP waste slide 11 calls out.
Changes:
- New kernel: src/kernbench/benches/_gqa_attention_decode_cube_repl_pe_tp.py
Simplest of the 4 cases. Active rank loads full Q/K/V from HBM,
computes attention via S_kv tile sweep with online-softmax merge,
writes O. All non-(0,0) ranks early-return. No tl.send/recv.
- src/kernbench/benches/milestone_gqa_decode_4cases.py:
- Add panel single_kv_group_decode_gqa_cube_repl_pe_tp (Case 2)
to _PANELS and _PANEL_DISPATCH.
- Add _run_decode_panel_cube_repl_pe_tp helper: DPPolicy K/V/Q/O
= cube=replicate, pe=replicate (models 8× memory waste).
- Extend _make_bench_fn to dispatch kind="decode_cube_repl_pe_tp"
to the new runner.
- tests/attention/test_milestone_gqa_decode_4cases.py:
4 new tests assert Case 2 contract: panel registered, smoke
completion, zero ipcq_copy (no comm), single dma_write from cube 0.
Verification: 8/8 tests pass (4 Case 4 anchor + 4 new Case 2).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The legacy panel names suggested batched serving semantics they never
had — all four modeled a single user with KV sharded differently
(C=1 single-cube; C=4 multi-cube Cube-SP), at toy dims (T_q=4, S_kv≤128).
The single-KV-group C=8 panel + the new milestone-gqa-decode-4cases
bench cover the meaningful comparisons; pytest regression already
covers C=1/C=4 configurations end-to-end at richer scale.
Changes:
- milestone_gqa_headline.py: drop the 4 legacy panels; _PANELS now
contains only single_kv_group_prefill_gqa_c8_p8. Update docstring.
- tests/attention/test_milestone_gqa_headline.py: drop the 3 legacy-
panel architectural tests (Ring-KV traffic, root-only decode write,
per-CUBE distributed output) and test_decode_panels_use_real_gqa
(no decode panels in this bench anymore). Equivalent properties
are asserted in test_milestone_gqa_single_kv_group_prefill_panel.py
(64 dma_writes, 896 ipcq_copy) and test_milestone_gqa_decode_4cases.py
(1 dma_write at cube 6, 189 ipcq_copy).
- tests/attention/test_milestone_gqa_single_kv_group_prefill_panel.py:
drop test_existing_prefill_panel_runner_backward_compat (it exercised
multi_user_prefill_gqa which no longer exists).
- scripts/paper/paper_plot_gqa.py: replace the 4 legacy _LABELS entries
with the single single_kv_group_prefill_gqa_c8_p8 label.
- Regenerate 1H_milestone_output/gqa_headline/sweep.json from the new
panel set.
Verification: 9/9 tests pass across the 3 affected test files.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First milestone of the decode 4-cases comparative study per
GQA_full_deck.pptx slides 11-17. Case 4 (Cube-SP × PE-SP, the optimal
case per slide 11) is structurally the existing _gqa_attention_decode_long
at sub_w=4 (Increment 2's lrab-adapted center-root reduce). This commit
wires it into a dedicated bench so the remaining cases land alongside.
Changes:
- New bench: src/kernbench/benches/milestone_gqa_decode_4cases.py
Houses the 4 case panels under one milestone-gqa-decode-4cases entry
(gated by GQA_DECODE_4CASES_RUN=1; output to
1H_milestone_output/gqa_decode_4cases/sweep.json). Cases 1-3 are
TBD in subsequent sub-increments (5C.A/B/C).
- New panel: single_kv_group_decode_gqa_cube_sp_pe_sp
C=8, P=8, sub_w=4, T_q=1, S_kv=131_072, d_head=128, h_q=8, h_kv=1.
- src/kernbench/benches/milestone_gqa_headline.py: _run_decode_panel
extended with keyword-only sub_w/T_q/d_head/h_q/h_kv overrides
(defaults preserve existing-panel behaviour).
- tests/attention/test_milestone_gqa_decode_4cases.py: 4 new tests
asserting registration, smoke completion, reduce-to-root at the lrab
center cube (cube 6), and the predicted 189-ipcq Case-4 traffic
pattern (168 intra-CUBE + 21 inter-CUBE lrab Phase 1+2).
- tests/attention/test_milestone_gqa_headline.py: rename
test_sweep_json_has_four_panels -> test_sweep_json_has_expected_panels
and switch hardcoded 4 to len(PANELS) (the panel set grew to 5
with Increment 5's single_kv_group_prefill_gqa_c8_p8).
Deviation noted: slide 13 prescribes AllReduce on (m,ℓ,O); our kernel
does reduce-to-root (only the lrab center cube has the answer) per
ADR-0060 §4. Treated as the kernbench Case-4 baseline.
Verification: all 4 new tests pass; 90 regression tests pass; the
previously-failing test_sweep_json_has_four_panels now passes under
its renamed form.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The headline T_q=S_kv=32K target overflows the 1 MB per-PE scratch
pool: at T_q_local=4K and tile_s=1024 the scores matrix alone is 8 MB.
The prefill kernel's bootstrap section also leaves K_t/V_t/scores/
exp_scores persistent (outside tl.scratch_scope), inflating baseline.
Scale-down to T_q=S_kv=1K (T_q_local=128, fits comfortably) preserves
the C=8 + P=8 architecture demonstration; the true LLaMA 32K headline
awaits a future increment to add Q-axis tiling and tighten bootstrap
scratch discipline.
Verified end-to-end: kernbench run --bench milestone-gqa-headline now
produces sweep.json with all 5 panels. The new panel shows
ipcq_copy=896 (matches (C-1)·n_tiles·2·C·P = 7·1·2·8·8) and
dma_write=64 (one per PE, head-parallel + intra-CUBE PE-SP).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
End-to-end wires C=8 P=8 d_head=128 prefill with snake-ring inter-CUBE
SFR, intra-CUBE PE-SP (all 64 ranks active), and the milestone bench
panel. Decode kernel gains lrab-adapted center-root reduce for the
2×4 sub-mesh per ADR-0060 §4.2.
Increment 1 — SFR multi-row snake
src/kernbench/ccl/sfr_config.py: configure_sfr_intercube_ring gains
submesh_shape / submesh_origin kwargs; installs a Hamiltonian snake
ring through a rectangular sub-mesh (every hop is 1-hop physical
neighbour). Backward-compat: 1D-row behaviour preserved when
submesh_shape is None.
tests/test_intercube_snake_ring.py (12 tests)
Increment 2 — Decode lrab-adapted center-root reduce
src/kernbench/benches/_gqa_attention_decode_long.py: new sub_w param
(default 0 = existing 1D-chain). sub_w >= 2 selects the ADR-0060
§4.2 prescribed lrab-adapted Phase 1+2 reduce (bidirectional row +
bidirectional col converge to the center cube), with log-sum-exp
_merge_running replacing the plain + of lrab.
tests/attention/test_gqa_decode_long_2d_reduce.py (4 tests)
Increment 3 — Prefill kernel at C=8 (no production change)
Verified by inspection that the existing prefill_long kernel +
Increment 1's snake SFR already work at C=8 without any kernel
edit. The kernel speaks logical W/E; the snake routes it.
tests/attention/test_gqa_prefill_long_c8_snake.py (3 tests)
Increment 4 — Intra-CUBE PE-SP in prefill (all 64 ranks)
src/kernbench/benches/_gqa_attention_prefill_long.py: new P param
(default 1 = existing PE-0-only). P > 1 splits T_q query-axis-wise
across the P PEs of each CUBE; output rows are disjoint per PE so
no intra-CUBE reduce is needed; each PE drives its own same-lane
ring (P parallel rings).
tests/attention/test_gqa_prefill_long_pe_sp.py (5 tests)
Increment 5 — LLaMA-scale milestone bench panel
src/kernbench/benches/milestone_gqa_headline.py: new panel
single_kv_group_prefill_gqa_c8_p8 (C=8, P=8, T_q=S_kv=32K,
d_head=128). _run_prefill_panel extended with P/T_q/d_head
defaults; routes snake SFR when C > mesh_w.
tests/attention/test_milestone_gqa_single_kv_group_prefill_panel.py (3 tests)
Total: 4 production files modified, 5 new test files, 27 new tests.
Followed the Phase 1/2 protocol per CLAUDE.md throughout.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mid-pass through §2. Captured progress so far:
- §2 intro previews the three new threads (graph view, DES engine,
congestion) the rewritten latency section weaves together.
- §2.3 retitled to "Latency model: a graph traversed by events" and
restructured into bold-led paragraphs:
- The hardware as a graph (nodes = components, edges = links).
- From graph to discrete-event simulation (node/edge events,
deterministic ordering, correlation-ID trace).
- Latency contributions (per-node fixed, per-edge size-aware,
per-service occupancy).
- Congestion: per-edge FIFO BW occupancy, HBM per-PC parallelism,
component serial workers — the mechanisms that surface real
bottlenecks instead of peak-BW roofline.
- Control-plane cost model (FIXED + b·R) — unchanged.
- Accuracy: extended with a second cross-check from the all-reduce
study (torus vs. analytic startup-plus-per-packet model + FSIM
external single-device reference). Existing GEMM analytic-vs-
measured 10-20% check retained.
- §2.4 Hardware-configuration table unchanged.
Still TODO: cross-check accuracy numbers against a fresh
milestone-1h-gemm/ccl re-run; the current artifacts predate the
ADR-0064 Rev2 cost-model and IPCQ Phase-2 race fix and may need a
refresh before §2.3 final lock-in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- §1 Introduction rewritten with AHBM-first opening, in-paper KernBench
platform overview, GQA motivation and three architectural requirements,
three matching HW-SW co-design mechanisms (GQA-aware placement,
PE_IPCQ, composite-command GEMM under PE_SCHEDULER), broader-
applicability outlook (PE_IPCQ as collective substrate, composite
command for fused operator pipelines), and outline. Consistent with
the Executive Summary structure.
- Executive Summary: extended broader-applicability closing with a
forward-looking line previewing FFN/MoE integration.
- §5 section heading renamed from
"All-Reduce Acceleration via PE_IPCQ" to
"PE_IPCQ and Collective Communication".
- Title page date line refined to "2026 H1 Report".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
test_bench_registry EXPECTED_NAMES: add the 3 milestone benches the pull registered (milestone-1h-ccl, milestone-1h-gemm, milestone-gqa-headline) — 11 benches, alphabetical. test_memory_store::test_shape_mismatch_raises: the pull made same-size reshapes byte-conserving (allowed) and an over-large read raise 'Out-of-bounds read' (not 'Shape mismatch') — match the new message. Full suite now 820 passed / 1 skipped / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ywkang
mukesh