ADR-0064 D8 broadened from "DMA fast-path" to "single-op-cmd fast-path":
every single-op command (DmaRead/DmaWrite/Gemm/Math/Copy) now pays the
lighter FIXED=8; only CompositeCmd keeps the 40-cycle control-path FIXED
(it alone needs scheduler plan generation + per-tile RW-hazard tracking +
completion wiring). Renamed knob fixed_per_dma_cmd_cycles ->
fixed_per_single_op_cmd_cycles. dispatch_cycles now branches on
"is CompositeCmd" rather than enumerating DMA types.
Term choice: "single-op" (not "atomic", which read as sync/async) — the
axis is composition (one engine op vs fused multi-op plan), orthogonal to
timing. single-op <-> composite.
Tests: test_pe_cost_model.py updated to the single-op surface (defaults,
fast-path over all 5 single-op cmd types, composite general path, yaml
override). All green.
Recalibrated tests/attention/test_gqa_decode_opt2.py
::test_opt3_dispatch_exceeds_opt2 — NOT a regression: D8 makes single-op
cmds 5x cheaper, so opt2's two-composite fusion win over opt3's many
single-ops narrowed from pre-D8 ~3.7x to ~1.87x (opt3=224 > opt2=120).
The CPU-offload invariant (opt2 cheaper) still holds; only the model-
dependent ">2x" constant was over-fit to the old uniform-40 model. Gate
now: direction + >1.5x margin (matches sibling R-sweep test's stated
"absolute ratio informative-only" philosophy).
NOTE for review: ADR-0065's "2x CPU-offload win" headline may want a
refresh to reflect the post-D8 ~1.87x — left to user (architectural doc).
Full regression: 826 passed, 1 skipped (tests/ excl. tests/gemm).
--- Remaining work (resume here if interrupted) ---
5. Re-run scripts/paper/paper_plot_gemm_async_vs_composite.py with new
cost model; verify async-tiled dispatch overhead drops (~4576ns ->
~1536ns expected) and the composite-vs-async-tiled gap narrows from
the prior ~6.3x at K=3072.
6. Copy regenerated gemm_composite_vs_async_tflops.png to
docs/report/1H-codesign-paper/figures/.
7. Paper §3.4 (03-gemm.tex sec:gemm-vs-async): finish naive->async-full /
chunked->async-tiled rename AND reframe FIXED_DMA wording to single-op
vs composite (currently still says "lighter FIXED for DMA descriptors,
FIXED_DMA=8"). Table 2 (02-platform) + §2 dispatch prose already done.
8. Paper §3.4 K=3072 corner para + mechanism #3: update dispatch breakdown
to new model (96 DMA*8 + 95 single-op*8 ≈ 1.5us vs old 4.6us); update
headline ratio if it changed.
9. Rebuild docs/report/1H-codesign-paper/build/main.pdf (tectonic) +
verify via pdftotext.
10. Then this is the bench-harness + paper commits (Groups 2 & 3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five-stage chain was set as a centered display equation, which
overflowed the column and clipped the last 'DMA_WRITE' label. The
inline prose right below it also repeated the same chain in a
shorter form. Consolidate into one inline parenthetical listing the
five stage names, and reuse 'flows through the chain' for the
self-routing description — no clipping, no repetition.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The matplotlib title and gray subtitle were duplicating what the
figure caption already says in the LaTeX source, and crowding the
top of every chart. Drop both — chart now leaves the title/subtitle
slots blank and the caption is the single source of truth for
chart titling.
Re-render all 5 GEMM figures (mac util measured, mac util theoretical
vs measured, stage breakdown, HBM BW util, per-PE TFLOPS) and resync
into docs/report/1H-codesign-paper/figures/. No data changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new evaluation plots for §3 GEMM, both using the composite window
as the denominator (the up-front tl.load of A in load_ref is therefore
excluded — only HBM traffic and compute inside the composite count):
- gemm_hbm_bw_util.png: per-PE HBM bandwidth utilization against the
256 GB/s per-PE ceiling. Answers "is this workload memory-bound on
this configuration?" — small/single-tile shapes sit at 22–46%
(pipeline-fill-limited), large or output-write-dominated shapes
saturate (≥85%).
- gemm_per_pe_tflops.png: per-PE GEMM throughput against the 8 TFLOP/s
engine peak. The deep-K K=3072 load_ref case lands at
7.18 TFLOP/s (~90% of peak) — the configuration's clean win.
ref_ref at the same shape drops to 3.83 TFLOP/s because the second
operand doubles HBM pressure and the kernel hits the BW ceiling.
The variant gap on this chart is the operational cost of NOT
pre-staging the activation.
Both charts compare two operand-staging variants on every shape:
load_ref ("activation pre-staged", weight-only HBM streaming) and
ref_ref (both A and W streamed from HBM). load_load is omitted —
with both operands pre-staged the composite carries no HBM traffic
and the BW metric collapses to 0.
Analytic / measured peak unified to the single hardware spec
(8.0 TFLOP/s = 8000 flops/ns). T_STAGE is now derived as
tile_flops / peak (= 16.384 ns for the 32×64×32 tile) rather than
the previous hardcoded 16 ns, so MAC efficiency and per-PE TFLOPS%
share the same denominator and never disagree. Side effect: max
analytic-vs-measured gap tightens from 2.2 ppt to 1.4 ppt across
the sweep.
§3 Results restructured: lead with the two new evaluation plots
(BW saturation, achieved throughput), then the MAC-utilization
analytic-vs-measured chart serves as the validation step, then the
per-stage engine wall-clock as the supporting diagnostic. §3
Analysis updated to use the load_ref / ref_ref contrast as the
hardware-software boundary line that motivates the GQA kernel of §5.
§2.4 Accuracy claim updated to match: GEMM analytic-vs-measured
agreement is now "within 1.4 ppt across every swept shape", from
~7.7% at single-tile up to ~90% at K=3072. Subtitle/title vertical
stacking glitch in the matplotlib charts fixed in passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three coupled fixes that recover small-tile GEMM pipeline efficiency
from 53% to 88% (32x3072x32 load_ref, composite_window basis).
1. PE_DMA channel-hold (ADR-0014 D4 clarified): both the
_handle_with_hooks (PeInternalTxn) and _pipeline_process
(TileToken) paths used to hold the cap=1 DMA channel through the
full HBM round-trip, which double-serialized with the HBM_CTRL's
own per-PC `available_at` model and prevented back-to-back tile
DMAs from amortizing their per-request head latency. Channel is
now released after the request is enqueued onto the next hop;
HBM serialization is HBM_CTRL's responsibility alone.
Tests: new test_pe_dma_back_to_back_pipelining as the oracle
(asserts wall < 75% of strict-serialized N x single_op). Existing
test_pe_dma_record_start_after_channel_acquire rewritten to assert
t_start clustering (channel released fast) instead of the old
round-trip-hold invariant. test_pe_dma_same_channel_serializes
still passes — HBM_CTRL preserves ordering.
Probe regression: PE→local-HBM 32 KiB stays at 141 ns
(single-request, unaffected).
2. milestone_1h_gemm bench: matmul_composite was reading MATMUL_M/K/N
env vars at module load, so every sweep row replayed the cached
256³ result; values now read inside run(). Drops the stale
sys.modules deletion hack.
3. Analytic ideal-pipeline model: dropped the (n_mn-1)·dma_w_per_pair
penalty (over-pessimistic for under-tile shapes — it pushed
measured > theoretical) and replaced the D_STAGES-derived head
with empirical T_PIPELINE_FILL=60 ns / T_PIPELINE_TAIL=30 ns.
Max analytic-vs-measured gap across all 7 swept shapes now 2.2 ppt
(was 9-44 ppt under the old constants).
Paper updates:
- §3 (GEMM): 78%→88% measured at 48 tiles, 23%→15% at 1 tile,
stage breakdown numbers refreshed (DMA in / Fetch / GEMM all
~785 ns at K=3072), analytic-vs-measured agreement tightened
to "within 2.2 ppt".
- §2.4 (Accuracy): GEMM tracking claim refreshed accordingly.
- §5 (GQA): restore long-ctx 4-cases figures into figures/
(they were dropped from bench output dir as derived artifacts in
92b9221 / e45626c but §5 still cites them by name).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Splits the GQA helpers into a dedicated subpackage to make room for the
prefill 4-cases study (next commit) and a single umbrella bench
(milestone-1h-gqa, after that).
Layout:
benches/gqa_helpers/
long_ctx/ — decode 4-cases kernels + sweep runner
short_ctx/ — prefill/decode short-context kernels
shared/ — _gqa_panel_helpers + decode_opt2 (context-agnostic)
The registry audit now skips subpackages so gqa_helpers/ (without a
leading underscore) doesn't get audited for @bench decorators.
Also drops the legacy milestone-gqa-headline bench, its
_gqa_attention_prefill_long kernel, 6 dependent prefill tests, the
paper_gqa_latency.py report harness, and the 3 stale headline-derived
PNGs the §6 wire-up referenced (paper will re-pull from the new
1H_milestone_output/gqa/long_ctx/ once §6 is updated).
The _ccl_cfg and _summarize_op_log helpers used to live in the
headline bench; extracted them to gqa_helpers/shared/_gqa_panel_helpers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Probe table (Table 1, tab:probe-pe-dma) promoted to table* so it
spans both columns and no longer gets clipped at the column edge.
- Fig 1 (hw-arch) row 1 (SIP, CUBE subfigures) trimmed: widths
0.495 -> 0.42 textwidth and a height=0.6\linewidth cap added so
the figure occupies less vertical space and floats can land
earlier in the document.
- Fig 1 and Fig 2 source blocks relocated to the top of §2 so they
enter LaTeX's float queue before any §2 body text. As a result
Fig 1 now lands on page 3 and Fig 2 on page 4 (was 4 and 5).
- Table 2 (tab:hw, modeled hardware config) reverted to plain
\begin{table}; the previous \begin{table*} reshape was not
requested.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- subfig widths 0.46 -> 0.495 textwidth so the \hfill gap between
(a) SIP and (b) CUBE shrinks to ~2% textwidth instead of ~8%.
- Removed the height=0.85\linewidth cap on both subfigures; with the
cap, the (roughly 1:1) SIP and CUBE diagrams were being scaled
down to 85% even when the column had room. Let them render at
their natural aspect ratio.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§2 platform:
- Accuracy promoted from \paragraph to \subsection (sec:accuracy);
it sits at the same heading level as Why / Device / Latency / HW
config and so reads as a first-class component of the platform
description rather than a tail-end footnote.
- Three architecture diagrams (SIP, CUBE, PE) collapsed into one
figure* using subcaption: Fig.~\ref{fig:hw-arch}(a) SIP and (b)
CUBE side-by-side on row 1, (c) PE wide on row 2, total height
capped at ~40% of the page. Subfigure cross-references rewritten
to Fig.~ref{fig:hw-arch}\subref{...} in the body text. main.tex
now pulls in the subcaption package.
- Table 1 (modeled hardware configuration) promoted to table*
(two-column / full page width) and the row layout rewritten as a
4-column tabular so Hierarchy + PE + Command-issue sit on the
left and Memory + Interconnect sit on the right. The previous
single-column rendering was getting cut at the right edge of the
printed column.
§6 GQA:
- Removed the historical four-panel headline table (tab:gqa), the
two figures (fig:gqa-lat, fig:gqa-break), and the prose paragraph
that cited their per-panel numbers. Reason: the underlying
milestone-gqa-headline bench has been simplified in collaborator
commit 65c365f ("drop misleading single_user_/multi_user_
panels") and no longer reproduces that data, so the section was
left referencing a dataset the current bench cannot regenerate.
- The previously-added 4-cases long-context decode comparison is
now §6's only Results subsection, retitled to
"Results: long-context decode and parallelism strategies".
The headline GQA result that survives into the paper is therefore
the parallelism trade-off study (Case 4 / Cube-SP × PE-SP at 34 us
buying an 8x KV-memory reduction over the latency leaders).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The collaborator commit 7c346de added the comparative figures
(gqa_decode_long_ctx_4cases_{latency,memory,traffic}.png) but the
fused-GQA section still only referenced the four headline panels.
This commit closes that loop:
- New §6 subsection "Long-context decode: parallelism strategies"
added between Results and Analysis. It lays out the four
parallelism strategies (Cube-SP/Repl x PE-TP/SP) and pulls in the
three new figures.
- The discussion highlights the central trade: the fastest strategy
(Case 3, Cube-Repl x PE-SP at 20.2 us) requires replicating the
full KV cache to every CUBE, while Case 4 (Cube-SP x PE-SP, the
chosen design marked *) gives back ~14 us in exchange for an 8x
KV-memory reduction. Case 4's ~190 IPCQ copies + ~190 DMA reads
are precisely the on-device collective traffic PE_IPCQ and the
torus links of §5 are provisioned to absorb -- a direct payoff
of the communication-side codesign work.
- Connects back to §5 (PE_IPCQ / all-reduce) so the reader sees the
capstone arc: the GEMM enabler exposes the data-movement bound,
the communication enabler attacks it, and the long-context
parallelism study shows how the choice between the two extremes is
framed by KV memory vs. on-device collective traffic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§2 platform:
- Add a "memory-centric AHBM" sentence to the device-and-execution
intro: each PE is paired with dedicated HBM bandwidth and on-PE
TCM, so the performance question is about feeding compute from
locally-attached memory + moving the unavoidable inter-PE traffic
efficiently. Makes the AHBM character of the platform visible
before §2.3 starts unpacking the latency model.
- Promote "Congestion and contention modeling" from \paragraph to
\subsubsection: this is the platform's core differentiator over a
peak-BW roofline, so it deserves its own heading.
- Rename "Control-plane (issue) cost model" to "Command dispatch
overhead model" -- describes what it actually charges (the PE_CPU
paying a fixed + per-byte cost to push a command to one of the
accelerator engines) without the more abstract "control-plane"
framing.
- Add a third Accuracy cross-check from kernbench probe: a
PE→HBM DMA-distance sweep that confirms monotonic hop progression,
bandwidth saturation matching the per-edge model, and the built-in
invariants (D2H >= H2D, cross-CUBE best < worst). New
Table~\ref{tab:probe-pe-dma} reports per-traversal latency and
utilisation at 32KiB / 1MiB across five hop classes.
- Captured probe output as figures/probe_pe_dma_summary.txt for
reproducibility.
§5 PE_IPCQ / all-reduce:
- Re-ran milestone-1h-ccl on current sim_engine (post-ADR-0064 Rev2
and the IPCQ slot-wrap Phase-2 race fix). Updated the topology-
comparison table and the buffer-kind caption to the fresh raw
latencies. Headline ratios are preserved:
torus vs mesh saving at 96 KB/PE: 24.8% (was 24.8%) -> ~25%
torus vs ring saving at 96 KB/PE: 19.3% (was 19.3%) -> ~19%
TCM vs HBM saving at 64 KB/PE: 13.4% (was 13.9%) -> ~13%
TCM vs SRAM saving at 64 KB/PE: 37.2% (was 38.3%) -> ~37%
The Executive Summary's "up to ~14%" / "up to ~38%" framing stays
consistent with these post values.
Artifacts refreshed in src/kernbench/benches/1H_milestone_output/:
- ccl/summary.csv + per-topology PNGs + buffer-kind CSV/PNG
- ccl/comparison_mesh_vs_ring_vs_2DTorus_vs_theoretical_vs_fsim.png
- gemm/* (re-run yields identical pe_window structure; PNGs
refreshed)
Paper figures synced to the fresh artifacts:
- figures/allreduce_comparison.png
- figures/allreduce_buffer_kind.png
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Generated by scripts/paper/paper_plot_gqa_decode_long_ctx_4cases.py
from the bench's sweep.json (S_kv=8192, the headline 128K run was
sidestepped — Case 2's single-PE 128-tile sweep is the long pole
and shape/ordering is preserved across scales).
gqa_decode_long_ctx_4cases_latency.png end-to-end latency / case
gqa_decode_long_ctx_4cases_traffic.png ipcq / dma op-count breakdown
gqa_decode_long_ctx_4cases_memory.png KV bytes per cube (slide-11)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Reordered §2 subsections to follow the SIP → CUBE → PE → graph
flow: Why KernBench → Device and execution model → Latency model
→ Modeled hardware configuration. Readers now meet the device
hierarchy before the graph abstraction that re-uses it.
- §2.2 Device and execution model: starts with the SIP/CUBE/PE
hierarchy paragraphs (each anchoring fig:sip-arch, fig:cube-arch,
fig:pe-arch); then the runtime-API/sim-engine/components bullet
list; then the atomic-vs-composite command distinction (corrects
the prior over-narrow framing that read every PE command as
composite -- atomic single-engine commands exist too, and PE_CPU
itself runs control-plane work directly).
- §2.3 Latency model: opens with the four-contribution decomposition
(per-node overhead, per-edge transmission, drain, queuing delay)
and the latency_model schematic; retains existing The hardware as
a graph / From graph to DES / Latency contributions / Congestion
/ Control-plane cost model / Accuracy paragraphs. Accuracy
paragraph now closes on KernBench's sufficiency for *relative*
HW/SW design trade-offs given analytic + external-simulator
agreement.
- New figures and assets:
- figures/sip_architecture.pdf (SIP-level graph view)
- figures/cube_architecture.pdf (CUBE-level zoom-in)
- figures/latency_model.png (conceptual latency-model
schematic with per-node /
per-edge / drain / queuing-delay
colour coding)
- figures/pe_architecture.png (carried over)
- Source-of-truth generator for the latency schematic:
scripts/paper/paper_latency_model_diagram.py (a report-only
harness under scripts/paper/ per the /paper isolation rule).
- main.tex preamble: \usepackage{tikz} added (kept from prior
sequence-diagram draft -- harmless now that the latency model is
a PNG; left in to keep paragraph numbering stable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mid-pass through §2. Captured progress so far:
- §2 intro previews the three new threads (graph view, DES engine,
congestion) the rewritten latency section weaves together.
- §2.3 retitled to "Latency model: a graph traversed by events" and
restructured into bold-led paragraphs:
- The hardware as a graph (nodes = components, edges = links).
- From graph to discrete-event simulation (node/edge events,
deterministic ordering, correlation-ID trace).
- Latency contributions (per-node fixed, per-edge size-aware,
per-service occupancy).
- Congestion: per-edge FIFO BW occupancy, HBM per-PC parallelism,
component serial workers — the mechanisms that surface real
bottlenecks instead of peak-BW roofline.
- Control-plane cost model (FIXED + b·R) — unchanged.
- Accuracy: extended with a second cross-check from the all-reduce
study (torus vs. analytic startup-plus-per-packet model + FSIM
external single-device reference). Existing GEMM analytic-vs-
measured 10-20% check retained.
- §2.4 Hardware-configuration table unchanged.
Still TODO: cross-check accuracy numbers against a fresh
milestone-1h-gemm/ccl re-run; the current artifacts predate the
ADR-0064 Rev2 cost-model and IPCQ Phase-2 race fix and may need a
refresh before §2.3 final lock-in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- §1 Introduction rewritten with AHBM-first opening, in-paper KernBench
platform overview, GQA motivation and three architectural requirements,
three matching HW-SW co-design mechanisms (GQA-aware placement,
PE_IPCQ, composite-command GEMM under PE_SCHEDULER), broader-
applicability outlook (PE_IPCQ as collective substrate, composite
command for fused operator pipelines), and outline. Consistent with
the Executive Summary structure.
- Executive Summary: extended broader-applicability closing with a
forward-looking line previewing FFN/MoE integration.
- §5 section heading renamed from
"All-Reduce Acceleration via PE_IPCQ" to
"PE_IPCQ and Collective Communication".
- Title page date line refined to "2026 H1 Report".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The reconciliation table, corrected file/test lists, measured 5.22x ratio, and the new section 15 (data-mode numerics D8 + N1-N4) — the content the git-mv in 30a0451 left uncommitted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updated the detailed design to match the shipped implementation: an as-built reconciliation table (P1-first, pinned-based DMA, D7 hard-cap-error, D8 output handle, prologue 1-stage-tile feed, D6.7 prologue-scoped), corrected file/test lists, the measured 5.22x dispatch ratio, and a new section 15 documenting the data-mode numeric subsystem (D8 + N1-N4) the original DDD omitted. Status Draft->Accepted; moved docs/adr-proposed -> docs/ddd/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tl.composite now returns the output TensorHandle (not a CompletionHandle) so its result chains like tl.dot's; the handle carries the completion in a CompositeFuture pending so downstream ops and tl.wait auto-await it. out is a handle: out=tl.ref(addr,shape) (HBM, DMA_WRITE inside the composite) or an in-place TCM handle (STORE only); omitted -> TLContext auto-allocates a TCM scratch. out_ptr kept as HBM shorthand (= out=tl.ref(out_ptr, shape)) to avoid churning ~30 existing call sites.
tl.ref now returns space=hbm (it references HBM data; operand-input DMA stays pinned-based per D4 so input streaming is unchanged). tiling: the tile loop's DMA_WRITE is gated on out.space==hbm (out analog of the operand pinned rule) — a TCM output stays on-chip (chainable) and its high-bit scratch address no longer hits the DMA PA decoder.
Fixes the opt2 data-mode crash: the recipe accumulator O is TCM -> no DMA_WRITE -> opt2 now RUNS end-to-end in data mode (enable_data=True). Numeric parity of the recipe MATH ops is the next step. Suite 812 pass / 3 pre-existing fail; existing composite benches use out_ptr->hbm->DMA_WRITE unchanged (byte-equal).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New `/paper` slash-command skill that synthesizes ADR/SPEC content and live
KernBench benchmark results into a sectioned LaTeX technical paper compiled
to PDF with Tectonic (auto-installed). The skill negotiates a TOC, grounds
every number in committed artifacts or fresh bench runs, and keeps
report-only benches isolated.
This commit also includes the first generated report:
- docs/report/1H-codesign-paper/ — main.tex + per-section .tex, figures,
toc.md contract, and the built 8-page main.pdf. Covers the platform
(source-level kernels, latency model + accuracy, HW config from
topology.yaml), GEMM via composite command, All-Reduce via PE_IPCQ, and
fused GQA combining both, plus discussion/conclusion/2H future work.
- scripts/paper/ — isolated report harnesses (not registered benches):
paper_gqa_latency.py harvests per-panel GQA end-to-end latency + engine
occupancy (the milestone only emitted op-counts); paper_plot_gqa.py
renders the GQA figures.
GEMM/All-Reduce reuse committed milestone figures/CSVs; GQA results are
generated fresh. Honest flags retained: PE_CPU dispatch cost is 0 in this
config, and the proposed two-composite softmax_merge decode is marked
designed-not-measured.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tiling.generate_plan_from_ops: scan flat ops for the GEMM (<=1); pre-GEMM KERNEL ops become single-shot prologue MATH stages, post-GEMM ops split by scope (K_TILE/OUTPUT_TILE epilogue + KERNEL post-loop). MATH-only composite reproduces the legacy math-head plan. Prologue/post-loop stages fold into the first/last tile so the feeder + completion counting are untouched (existing benches have neither -> byte-equal op_log).
PipelinePlan gains prologue_stages/epilogue_stages. pe_scheduler._generate_plan delegates to generate_plan_from_ops. DMA keeps the existing pinned signal (NOT a space flip); recipe scratch/primary-out handles are pinned=True so the head GEMM's auto-bound a (=P) is consumed in place.
ADR-0065 D4/D6.7/Test#5 amended (EN+KO): DMA decision from pinned, not space; prologue recipe ops TCM-only (head op exempt -- it may stream from HBM).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New triton_emu/tl_recipes.py: RecipeDescriptor/EngineOp/PrimaryOutSpec + RECIPE_DESCRIPTORS['softmax_merge'] (8-step engine_seq). PE_SCHEDULER does not import it (ADR-0065 D5 boundary).
TLContext.composite(): add prologue=[...] + out=TensorHandle kwargs, a optional. _expand_prologue lowers a recipe into flat MATH OpSpecs (scope=KERNEL), allocates TCM scratch, derives the primary-out slot 'P' and auto-binds it into the head GEMM a (D6.6 conflict check); rw_handles=(m,l,O). decode-opt2 #2 lowers to 10 ops [rmax,max_elem,exp_diff,exp_diff,rsum,fma,mul_bcast,copy,gemm,add].
D6.7 (MATH operand TCM-only) scoped to prologue recipe ops only — the head op (gemm or math) keeps existing DMA-staged-from-HBM behavior. D6.1 (GEMM count <=1) on the whole composite. Host-side lowering only; PE_SCHEDULER position-scan is P3.
Also commit the ADR-0064 Rev2 promotion content that the prior commit's git mv dropped: Status Proposed->Accepted + D7 amended to hard-cap ValueError (no segmentation), EN+KO.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0064 Revision 2 second-pass review fixes:
- D7 added 1024-byte cap rationale: explicitly framed as a *safe
engineering limit* (intentionally above all known composites at ~322
bytes, finite descriptor capacity placeholder), not a measured HW
number. Default is the discipline; topology override for real HW.
- D7 segment ordering: strict FIFO is the *ordering source*; rw_handles
is dependency metadata, not the ordering primitive. Explicit note for
the future RW-aware reorder migration path.
- Tests rewritten around the formula (D1) instead of specific numbers:
- Test #1 → "formula preservation" (parametrized across composites,
asserts dispatch == FIXED + bytes × R, not anchor 43 ns).
- Test #2 → robust "opt3 > 2 × opt2" qualitative gate; ≈ 4.0× is
informative only, not the gate, so calibration changes don't break.
- Test #9 → qualitative (opt2 < opt3 at all R), not a numeric ratio.
- Anchor description in D3 stays as *informative* (shows the model
produces ≈ 43 ns at defaults), but is no longer a test gate.
ADR-0065 second-pass review:
- D5 step 5 aligned with D6.6 invariant: explicit conflict check before
auto-binding (previously written as if auto-bind always happens).
Mirrors D6.6 wording: error if both kernel explicit `a` and prologue
primary_out are present.
- Test #7 made robust: `opt3 > 2 × opt2` gate (was ≈ 4.0×).
Default-calibration model expectation (≈ 4.0×) recorded as informative
reference, not the gate.
DDD-0065 follow-on robust thresholds:
- §1.4 success criteria: ratio condition rewritten as `opt3 > 2 × opt2`
with model-expected ≈ 4.0× as informative.
- §9 P6 phase gate: `opt3 > 2 × opt2` (was 4.0× ± 15%).
- §10 dispatch ratio test: assert `opt3 > 2 × opt2`; record observed
ratio for performance tracking but do not gate on it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ADR-0065 review fixes:
- D3 semantics clarified: position determines *phase* (pre-loop / head /
in-accumulation / per-output-tile / post-loop), scope determines
*repetition*. KERNEL = "once per CompositeCmd invocation", not "once
per kernel launch" — removes ambiguity around softmax_merge's KERNEL
scope at different positions before/after GEMM.
- D4 MATH-TCM invariant: Phase 1 limits DMA auto-insertion to GEMM
operands and head outputs; MATH operand with space="hbm" raises
validation error at TLContext emit. Catches kernel misuse early; a
MATH-with-HBM path is future work (explicit tl.load + MATH composite).
- D6 #6 NEW: auto-bind conflict — explicit GEMM `a=` + prologue
primary_out simultaneously → validation error (prevents "which value
wins" ambiguity).
- D6 #7 NEW: MATH operand TCM-only restatement.
- Consequences/Negative: strict-FIFO conservatism note — safe but may
under-expose composite-level overlap when handle-sharing composites
could in principle run in parallel.
- Test Requirements #10, #11: auto-bind conflict + MATH-HBM validation
error tests.
ADR-0064 Revision 2 clarification:
- D2 counting rule: per-op summation, no deduplication. Same handle
appearing in multiple OpSpecs or in rw_handles is counted
independently — matches HW reality (each descriptor field is a
distinct address slot, rw_handles is separate metadata block).
Example walked through (mul_bcast with O in src_a + out + rw_handles).
DDD-0065 follow-on:
- §3.2: new test file test_tl_composite_validation.py covers auto-bind
conflict, MATH-TCM, GEMM-count invariants.
- §4.3: phase vs repetition split aligned with ADR-0065 D3.
- §6 lowering pipeline: step 5 adds auto-bind conflict check; step 8
NEW — operand space validation; step 9 (size cap); step 10 (emit).
- §8 strict-FIFO conservatism note: explains why next-tile #1 cross-tile
pipelining still works under FIFO (no rw overlap), and where the
conservatism would hurt (future handle-sharing recipes).
- §10 verification: invariant-guard tests (auto-bind conflict, MATH
TCM, per-op summation accuracy).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ADR-0064 Revision 2 review fixes:
- D7 NEW: composite size cap (MAX_COMPOSITE_LOGICAL_BYTES default 1024
bytes). Oversized recipes deterministically segmented into N
CompositeCmds; each segment incurs its own dispatch cost. Models real
HW limits (descriptor queue entry, scheduler parser buffer, command
SRAM) and prevents the model from rewarding pathologically-large
fused composites.
- D2: type-aware extra-field byte counting (int/float=4, bool=1,
tuple/list=1+4N, str=1) — replaces uniform 4 bytes per extra.
- D3: recalibrated defaults to FIXED=40 cycles, R=0.0625 cycles/byte
(16 B/cycle — typical on-die descriptor queue width); anchor stays at
~43 ns for typical 1-OpSpec composite. Clarified anchor description:
DMA stages do not appear in logical_bytes (auto-inserted by
PE_SCHEDULER from operand.space per ADR-0065 D4).
- D4: removed clock_freq_ghz from pe_cost_model: override block;
conversion uses the PE node's existing clock_freq_ghz attr. Added
max_composite_logical_bytes knob.
- Context: emphasized command-count reduction (FIXED) as the primary
signal; byte term as secondary refinement.
- Open review: added large-composite scheduler-cost stress test.
- Test req: added composite-size-cap (#8) and R-sensitivity sweep (#9).
ADR-0065 + DDD-0065 follow-on updates:
- opt2 vs opt3 dispatch ratio updated 2.4× → ≈4.0× under new defaults
(FIXED-dominated, reflecting the corrected framing).
- Test req #9: decode opt2 composite fits within 1024-byte cap; no
segmentation needed for the GQA workload.
- DDD §6: TLContext lowering checks logical_bytes against cap (step 8).
- DDD §11: performance model recomputed with new defaults + sensitivity
table across R ∈ {0.25, 0.0625, 0.03125} confirming opt2 < opt3 holds.
- DDD §9 P6 gate: ratio band 2.4×±10% → 4.0×±15%; sensitivity sweep added.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three logically distinct changes, bundled for atomic test green:
1. **P3c — prefill_long tile-granular Ring KV** (ADR-0060 §5.5.1 amendment).
Convert the ring from slice-granular (one full ``(d_head, S_local)``
KV slice per step) to tile-granular (``n_tiles`` tiles of
``TILE_S_KV`` per step). Nested loop with outer tile, inner ring step:
each tile propagates through all C ring positions before the next
tile starts, so IPCQ in-flight depth stays at 1 per direction.
Bootstrap at ``(t=0, k=0)`` outside the scratch_scope establishes the
persistent ``(m, ℓ, O)``; every other iteration scope-wraps + persists
via ``copy_to``. Per-rank persistent scratch shrinks to ~1 KB; per-tile
scope bounded by TILE_S_KV regardless of S_local. Headline:
prefill_long now completes at S_kv=128K (previously overflowed).
New: ``tests/attention/test_gqa_prefill_long_tile_ring.py``
(3 tests — ceiling-lift + tile-granular ipcq_copy count +
per-CUBE distributed output regression guard).
2. **Rename ``gqa_*`` → ``gqa_attention_*``** across kernel files,
function names, and importers. The "attention" name makes the role
explicit (GQA is grouped-query attention) and matches upstream Triton
FlashAttention naming conventions. Renames:
_gqa_decode_long.py -> _gqa_attention_decode_long.py
_gqa_decode_short.py -> _gqa_attention_decode_short.py
_gqa_prefill_long.py -> _gqa_attention_prefill_long.py
_gqa_prefill_short.py -> _gqa_attention_prefill_short.py
And function names ``gqa_<phase>_<context>_kernel`` →
``gqa_attention_<phase>_<context>_kernel``. Updated 1 bench file
(milestone_gqa_headline.py) and 10 test files.
3. **ADR-0060 / 0062 / 0063 / 0064: Proposed → Accepted**.
All four are reflected in production code and covered by tests:
- ADR-0060 (GQA fused attention): 4 kernels deployed; §5.5.1
amendment added for the tile-granular Ring KV introduced by P3c
(EN + KO mirror).
- ADR-0062 (lazy tl.load): LoadFuture + _await_pending live in
tl_context.py.
- ADR-0063 (tl.scratch_scope + tl.copy_to): used in every chain
reduce + tile sweep + ring step. EN-only previously; KO
translation authored as part of this commit (CLAUDE.md
bidirectional rule).
- ADR-0064 (per-op-type CPU issue cost): cpu_issue_cost.py +
issue_cost_table wiring in tl_context.py (Phase E).
Files git mv'd from docs/adr-proposed/ to docs/adr/ (EN) and
docs/adr-ko/ (KO). ADR-0061 (tl.broadcast) stays Proposed — no
implementation; documented as optional convenience primitive in
the ADR itself.
Tests: 88/88 focused regression green
(tests/attention/ + Phase E + TL discipline).
ADR pair verification: ``python tools/verify_adr_lang_pairs.py`` OK.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Land the new GQA fused-attention kernels (ADR-0060) for prefill/decode
across long and short context, the TL discipline primitives they depend
on (ADR-0062 lazy load, ADR-0063 scratch_scope + copy_to), and the
per-op-type CPU issue cost model (ADR-0064). Remove the pre-ADR-0060
mesh-attention baseline now that the unified kernels supersede it.
ADR-0060 (long context)
- _gqa_decode.py: M-fold + 2-level chain reduce-to-root (Level-2
intra-CUBE row-then-col + Level-1 inter-CUBE) — root-only output.
- _gqa_prefill.py: head-parallel + Ring KV rotation around C CUBEs,
online-softmax merge per ring step, per-CUBE distributed output.
- Each merge stage wraps in scratch_scope() and persists running
(m, l, O) via copy_to() to lift the 1 MiB scratch ceiling.
ADR-0060 §B.split.2 (short context, kv_per_cube in {1,2,4,8})
- _gqa_decode_short.py / _gqa_prefill_short.py: no cube-SP; each CUBE
owns whole KV heads; PE-parallel heads with intra-group chain
reduce. Prefill has no Ring KV (each head fully resident).
ADR-0062 (lazy tl.load): future-bearing TensorHandle, auto-wait at
first consuming op (dot/MATH/store/send/copy_to/composite).
ADR-0063 (tl.scratch_scope + tl.copy_to): scoped per-tile arena with
copy_to writeback primitive for persistent running state.
ADR-0064 (CPU issue cost model)
- common/cpu_issue_cost.py: per-op-type table (composite=40 ns,
primitives=5 ns); ratios are load-bearing per D1.
- TLContext: issue_cost_table param; _emit_dispatch_overhead(kind)
consults table with dispatch_cycles fallback (ADR-0046 §D6
back-compat).
- Live PE_CPU paths (greenlet + legacy) construct TLContext with
DEFAULT_CPU_ISSUE_COST so saturation lever (ADR-0060 §1) is
measurable end-to-end.
P7 headline bench: milestone-gqa-headline writes per-panel
op_log_summary to 1H_milestone_output/gqa_headline/sweep.json. No
figure renderers yet (deferred).
Removals (pre-ADR-0060 baseline now superseded):
- benches: _attention_mesh_kv.py, _attention_mesh_mlo.py,
_attention_mesh_mlo_2d.py, milestone_gqa_llama70b.py
- tests: test_attention_*, test_mesh_*, test_milestone_gqa_llama70b
- topology: llama70b_4sip.yaml (only consumer was the deleted diag)
- artifacts: 1H_milestone_output/gqa/ (sweep.json + 5 PNGs)
- tests/gqa/ plot helper + test (broken on Windows Tcl/Tkinter)
- ADR-0060/0061 references to deleted file paths cleaned up
(EN + KO kept in sync).
Tests: 124/124 focused regression green (attention + Phase E + TL
discipline + triton_emu + pe_components). Full regression: 764 pass,
2 pre-existing test_bench_registry failures (stale EXPECTED_NAMES
across multiple benches, not introduced here).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Amends ADR-0026 to reflect the cube_start field added in e2fe331.
The production code shipped without an ADR update; this fills that
gap. Documentation-only change (no production code, no test code).
Changes (mirrored in both EN and KO):
- Status: Revision 5 → Revision 6
- D1: add ``cube_start: int = 0`` to the canonical DPPolicy dataclass
- D3: ``cube = policy.cube_start + cube_id`` in resolve_dp_policy
- D8: new section explaining purpose (disjoint cube sub-meshes for
GQA Llama-70B 8-KV-group headline), semantics, default-0 backward
compatibility, intra-device constraint, design rationale
(scalar vs 2D origin vs cube_ids list), and the kernel-side
cube_start subtraction needed to compensate for ADR-0022's
physical-cube-id ``program_id(axis=1)`` semantics.
tools/verify_adr_lang_pairs.py passes (EN/KO Status keyword and
title in sync).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per review:
- Placement unified to *contiguous* C×P blocks throughout (S0/S2.1/S2.2/S0.5);
round-robin demoted to the rejected alternative. Adds driver SP-enable
threshold fallback (smaller C / C=P=1 for short/early decode).
- Ring-vs-reduce cost model in S5.5: reduce ~ G*T_q*log(C*P) (O dominant;
m,l scalars), ring ~ 2*S (total K+V bytes a CUBE injects over C-1
rotations; recv_async pipelines so latency ~ max-step) -> ring wins when
T_q > 2S/(G*log(C*P)).
- opt3 'removes the bubble' -> 'hides (subject to scheduler+engine balance)'
everywhere; table 'hidden*' with footnote.
- 'rank' defined (SP participant = a PE in a CUBE, KV shard in its HBM->TCM).
- out-proj handoff contract (S0.5.4); S11 gate-type note (absolute latency
deferred to ADR-0064; structural + relative-to-baseline gates now).
- greenlet-as-contrast tightened to 'primitive-op (tl.dot) path' (S1, SB).
KO mirror synced; DDD gets the SP-enable threshold fallback. Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DDD-0060 was rewritten to the two-kernel composite-hybrid design, so SB
item 1 ('DDD not yet synced') is stale. Update EN + KO mirror to record the
DDD as synced (ADR authoritative, DDD = impl how-to). Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Supersede the old greenlet-primitive/load_async/single-kernel DDD. Align to
ADR-0060's composite-hybrid + hierarchical CUBE-Group SP with two kernels:
decode+SP (head-replicated, contiguous C×P static shard, 2-level
reduce-to-root) and prefill+SP (1 Q head/CUBE, Ring KV, no reduce). Updated
file plan (lazy tl.load, scratch_scope, broadcast-opt; evolve
_attention_mesh_mlo_2d/_kv; DPPolicy.cube_start; llama70b_4sip.yaml),
placement (contiguous shared KV), phase plan (P1..P8), verification, perf
model, risks, glossary. Defers design rationale to ADR-0060 (now
authoritative); open items point to ADR-0060 §B. ADR-0064 cost model noted.
Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Regenerate the KO mirror to match the current EN: two SP kernels
(decode=reduce / prefill=ring), TL;DR full code for the 3 decode variants
+ prefill, 'Q replicated / M-fold' and '1 Q head per CUBE' terminology,
contiguous shared KV layout, opt2 tl.wait, and all SB items. Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ex_composite #2 updates acc=(m,l,O) asynchronously on the scheduler and
the kernel never reads its output, so no auto-wait fires; acc is only final
after the last #2 in the serial chain. Add tl.wait() (drain all composites)
before hierarchical_reduce_and_store reads acc. opt1/opt3 don't need it
(their composite outputs are consumed in-iteration -> auto-wait). Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ring rotates K/V over IPCQ, so they must be kernel-held TCM handles
(tl.load), not HBM tl.ref streamed inside the composite. Correct the
prefill kernel: load K pre-transposed [d, S/C] and pass the TCM-resident
Kc/Vc directly to the composite (drop the bogus tl.ref(Kc,(d,TILE_S)) and
TILE_S); recv shapes match ([d,S/C] for K, [S/C,d] for V). Comments note
why K/V live in TCM. Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the 3 decode CPU-pipelining variants (opt1 current / opt3 sw-pipe /
opt2 ex_composite) up into the TL;DR as full standalone kernels alongside
the full prefill ring kernel, with the comparison table. S5.6 is reduced to
a brief anchor (still referenced by S8/SB) pointing to the TL;DR code +
keeping the recommend/cost-model linkage. Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The prior commit updated S0/S10/S5.5 terminology and added S5.6 (3 decode
variants) but left the TL;DR with the old 'all G heads replicated' wording
and no pointer to S5.6. Sync the TL;DR: 'Q replicated (G heads stacked into
the GEMM M-dim, M-fold)', '1 Q head per CUBE', and a one-line pointer to the
3 CPU-pipelining variants in S5.6. Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Terminology: 'Q replicated' (all G query heads stacked into the GEMM M-dim;
M-fold explained) for decode; 'one Q head per CUBE' precise for prefill.
- New S5.6: three decode CPU-pipelining variants — opt1 current CompositeCmd
(has GEMM-engine bubble), opt3 software pipelining (issue next Q.Kt before
this tile's softmax; Sj in persistent double buffer; ships now, no new cmd),
opt2 ex_composite split into two (#1 = existing GEMM+scale reads K first;
#2 = softmax+P.V+accumulator merge, the only new flash-epilogue machinery,
gives DMA K-before-V priority). MATH engine already has max/sum/exp — the
new part is the stateful flash accumulator, not the ops.
- S2.1/SB: shared prefill/decode KV layout = contiguous CxP blocks (prefill
causal-skip needs contiguous; avoids prefill->decode reshard; short-context
under-use caveat). S8 item 4 sizing note for the two-composite split.
Prefill note: opt2/opt3 give little for prefill (causal if can't enter a
composite; recv_async already overlaps). Docs only; KO mirror deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Decode-SP and prefill-SP are structurally different and cannot share one
kernel (principle: move the smaller thing):
- Decode: O=[G,d] tiny, KV cache big -> keep KV statically sharded/resident,
G heads replicated (M-fold), move only (m,l,O) via the 2-level reduce (S4).
- Prefill: O=[S,d] big -> shard heads (1 query head per CUBE, C=G),
rotate KV (Ring KV, S5.5), no (m,l,O) reduce; each CUBE writes its own head.
Rewrites TL;DR (two kernels), S0 (head map differs by case), S0.5.4 (output
head distribution differs -> downstream out-proj impact), S4 (scoped to
decode; S4.1 = intra-CUBE KV-split + PE reduce, the only way decode uses P
PEs), S5.1 (decode skeleton), S5.5 (head-parallel Ring KV), S9/S10/S11, and
adds SB items (two head mappings, output asymmetry, prefill within-CUBE PE,
C=G coupling, reconcile with _attention_mesh_mlo_2d). KO mirror deferred
until the design stabilizes (adr-proposed is mirror-exempt). Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Placement pivot (user-approved): a CUBE Group = C CUBEs within one SIP that
jointly own one KV head + its G=8 query heads. KV sequence sharded 2 levels
(Level-1 inter-CUBE over C, Level-2 intra-CUBE PE over P=8 = C*P ranks); G
folded into matmul M. C is a knob (8/4; C=1 = single-CUBE). Reverses the old
'a query head never spans CUBEs' non-goal (held only because the baseline
was H_kv=1). device=SIP; the for-kv loop is gone (head picked by CUBE coord).
Reduction is a 2-level reduce-to-root (not all-reduce): Level-2 PE tree ->
Level-1 center-root CUBE-mesh reduce, adapting lrab_hierarchical_allreduce's
inter-CUBE pattern as reduce-only + log-sum-exp. Data-driven (send on local
P.V completion, no global barrier) + level-pipelined; per-level topology
configurable (tree for decode, ring for long prefill).
Rewrites SS0/2/4/5/0.5/8-11, pseudocode, and adds SSB items (4-SIP config,
mesh partition, C knob, invariant-reversal check, index-math test). KO mirror
updated. Topology grounded in topology.yaml (4x4 CUBE mesh/SIP, 8 PE/cube).
Docs only; no production code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ADR-0060: GEMMs (Q.Kt, P.V) via existing tl.composite (scheduler-managed
tiling + K/V DMA streaming); softmax merge + IPCQ tree reduction stay in
kernel. Front TL;DR pseudocode of the final composite kernel; new section
B lists open design items (DDD sync, K pre-transpose, dma_read lever,
kernel-vs-scheduler tiling, ring path).
- ADR-0062: redefined from a new load_async op to global lazy tl.load
(non-blocking + auto-wait on first use; API unchanged; goldens regenerate).
- ADR-0064 (new): per-op-type CPU issue cost model (composite ~40ns >>
primitive) so the hybrid's CPU-saturation win becomes measurable
(currently dispatch_cycles=0 hides it). Cost-model impl deferred.
- KO mirrors for ADR-0060/0062/0064 (-ko suffix, adr-proposed).
Rationale: non-blocking CompositeCmd offloads tiling to PE_SCHEDULER,
decoupling CPU issue-rate from execution so the CPU can saturate the
engines; the prior 'composite = no latency benefit' claim was an artifact
of dispatch_cycles=0. Docs only; no production code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified against sim_engine data path (memory_store, data_executor):
- GQA reuse does NOT need a broadcast op. The baseline's h_q==h_kv limit
is its head-packing reshape hack, not a missing primitive. Correct fix:
per-KV-head loop with G folded into matmul M dim (byte-conserving
reshape) — runs today, timing correct (m=G*T_q), data mode runs.
- ADR-0061 broadcast demoted from 'the blocker' to optional convenience.
- Surfaced tl.trans = reshape-not-transpose (memory_store reshapes;
data_executor np.matmul on reshaped operands) -> numeric parity is
bounded; verification is structural/timing/determinism-first (matches
SPEC perf-model purpose). Optional tl.transpose deferred.
- Reordered DDD phase plan (P1 GQA needs no new feature; P3 scratch_scope
is the key scale feature); added open decisions 10.10 (transpose) and
10.11 (GQA-via-M-fold finding).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reframes the proposed GQA FlashAttention design onto kernbench's actual
execution model (greenlet tl API + IPCQ), replacing the composite-centric
mechanism that does not match the simulator:
- Records relationship to existing baseline kernels (_attention_mesh_kv/mlo,
milestone-gqa-llama70b) and their 3 deliberate limitations.
- Mechanism is greenlet tl (per-op latency; no fusion benefit), not
composites; running (m,l,O) is Python handles; reduction is tl.send/recv.
- Tree reduction (log N) replaces baseline all-to-all fan-out (N-1).
- Pseudocode rewritten in real tl.* signatures; depends on ADR-0061/62/63.
- Rejects composite-IPCQ-push + composite-carried-state + flash-composite
with documented efficient alternatives.
- Adds verification plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Proposed prerequisites surfaced while evaluating the GQA fused-attention
ADR against the actual kernbench tl/sim_engine implementation:
- ADR-0061 tl.broadcast: data-faithful GQA head reuse (fixes the
MemoryStore nbytes check that forces h_q==h_kv==1 today).
- ADR-0062 tl.load_async: non-blocking HBM tile load for KV prefetch
(KV-load-bound decode/long-context overlap).
- ADR-0063 tl.scratch_scope: per-tile scratch recycling (removes the
1 MiB bump-allocator ceiling that caps context at S=16).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per request, the milestone bench output is now tracked in git instead of
gitignored, so the figures/results are viewable on the remote:
- src/kernbench/benches/1H_milestone_output/gemm/ (3 PNGs + gemm_sweep.json)
- src/kernbench/benches/1H_milestone_output/ccl/ (3 per-topology PNGs,
buffer-kind PNG+CSV, FSIM comparison PNG, topology.png, summary.csv)
Drop the .gitignore rule; update ADR-0054 D3 + Negative (EN+KO) to say the
output is committed (regenerable by rerunning the bench). Artifacts produced
by full bench runs (milestone-1h-gemm non-FAST, milestone-1h-ccl).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ywkang
mukesh
Yangwook Kang