Adds the memory-bound mirror of the compute-bound prefill figure. New single-rank decode sweep (T_q=1, M=8) reuses the prefill_compute_bound kernels, sweeping per-rank S_kv with NO cross-CUBE reduce — isolating the local attention that the 64-way Case-6 decode masks under its reduce tail.
Finding: even memory-bound decode benefits from the composite command. Its scheduler-streamed concurrent per-tile DMAs reach ~233 GB/s (91% of the 256 GB/s per-rank roofline) while the primitive's blocking tl.dot serializes one tile DMA and plateaus at ~166 GB/s — a 25-28% latency win that widens with context. This refines the paper's 'decode is latency-neutral' claim: neutral at 64-way production scale (reduce-dominated), but the composite extracts bandwidth at the local level. The bandwidth roofline here mirrors the MAC roofline in prefill.
Adds: gqa_decode_streaming.py sweep (wired into the milestone umbrella as GQA_1H_SWEEPS=decode_streaming), paper_plot_gqa_decode_streaming.py, the figure, and a new fig:gqa-decode-stream + paragraph in 05-gqa.tex; rebuilt main.pdf. Prefill (Experiment B) re-verified byte-identical (794.1/668.1/646.9us).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complete the composite-command study across both attention regimes and
write up the result, resolving when the composite command helps latency.
Decode (memory-bound, T_q=1, M=8): the command form is latency-neutral —
the kernel is bound by KV streaming and the MAC array is near-idle, so the
composite's only benefit here is host-issue offload (O(n_tiles) -> O(1)
PE_CPU commands). Figure: gqa_decode_long_ctx_composite (latency flat,
command count saturates).
Prefill (compute-bound, large M=G*T_q tile-filling): add three command-form
variants of a single-rank FlashAttention prefill kernel
(_gqa_prefill_compute_bound: primitive / composite / composite_extended).
Here the composite's scheduler-internal per-tile DMA<->compute pipeline
keeps the MAC array fed while the primitive's blocking tl.dot starves it,
so composite wins on MAC utilization (68% flat -> 83%) and wall-clock
(19% faster at ctx=1024), with the margin growing with context (deeper
P.V reduction = more tiles to pipeline) — the compute-bound mirror of the
GEMM result in section 3. Figure: gqa_prefill_compute_bound (latency +
MAC util). Sweep wired as GQA_1H_SWEEPS=prefill_cb; tests cover structure,
e2e, and composite<primitive in the compute-bound regime.
The synthesis: composite has two benefits set by roofline position —
host-issue offload (always) and MAC-array feeding (compute-bound only).
Decode exercises the first, prefill both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- §6 GQA: rewrite long-context decode from 4-case to 6-case. Data
Placement Policy now presents the six placement options and their
intrinsic per-PE memory / per-token comm costs (no winner predicted);
the long-context subsection selects the best placement for that regime
(Case 6 ★, both-axes S_kv shard) from the measured 6-case sweep. Add
KV-sharding diagram + analytical budget/summary figures.
- Regenerate the 6-row decode sweep (milestone-1h-gqa) and the
sweep-dependent decode panels (latency/traffic/parallelism/memory) so
figures, prose, and sweep_decode.json are mutually consistent.
- §5 All-Reduce: add IPCQ design alternatives (architecture + decision
matrix) as design-rationale schematics (illustrative step-counts, not
measured) and the bench-generated topology diagram.
- §4 GEMM: rename "user-orchestrated/user-level" -> "kernel-orchestrated/
kernel-level" (orchestration runs in the kernel program vs the
scheduler-orchestrated composite); minor accuracy fixes (7.18 TFLOP/s
~10% below peak; ~781 ns DMA).
- Recompile build/main.pdf.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the unsourced 32 KB/layer placeholder in the analytical chart
with T*(N-1)/N where T = h_q * S_q * (d_head*2 + 8) ~ 2.1 KB
(per-PE (m, l, O) payload) and N is the participant count of the
reduce-only chain per stage. Cases 2/3/6 analytical bars now match
the simulator-measured numbers within ~10% (was 6-30x over).
Also:
- Bumps the measurement S_kv from 8 K to 64 K to verify Cases 1, 2, 3, 6
are genuinely S_kv-independent and Cases 4, 5 scale linearly.
- In-bar descriptor on the paired chart is now wrapped to fit the bar
width, black, no background box, and centered between the analytical
and measured bars so the label applies to both.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drops the per-case panel outline from linewidth 2.2 to 0.9 with 85% alpha.
The case-color frame still anchors each panel but no longer competes
with the swimlanes, flit packets, and annotations inside.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds two new IPCQ architecture views in the latency_model.png aesthetic:
a 2x2 flow figure (one per case) and a 4x1 stacked figure for taller
horizontal panels. Refactors per-case panel drawing into shared helpers
and disambiguates the flit labels (HMQ desc -> Hd so it does not
collide with data D). Two-row legend keys.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drops the control-data overlay from the paper figures folder and
adds the per-send latency cycle-stack PNG instead.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removes the (measured) comm column, widens the Notes column with
shorter wrapped text, scales the table figure down, and wraps the
heading to two lines so it sits flush with the table.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
6 generated figures comparing IPCQ (chosen) against the 3 HW alternatives
(Doorbell+Polling, HMQ, RDMA-CQ): architecture blocks, latency stack,
op counts, sequence flow, decision matrix, and control/data plane overlay.
Three figures also copied into the paper's figures/ folder for inclusion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous commit (8102ddb) renamed the figures-dir
gqa_decode_long_ctx_4cases_*.png → 6cases_*.png but only the rename
was staged; the cp that brought the regenerated 6-case content
landed after `git add`, so the committed bytes were still the old
4-case ones. This commit syncs the bytes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Filename cleanup so every long-ctx GQA artifact has a consistent
"gqa_long_ctx_6cases_*" prefix (or "gqa_decode_long_ctx_6cases_*"
for decode-only charts). Old "4cases" / mixed names retired.
Renames (long_ctx + figures, content unchanged):
gqa_hbm_budget.png -> gqa_long_ctx_6cases_hbm_budget.png
gqa_4cases_summary.png -> gqa_long_ctx_6cases_summary.png
gqa_4cases_memory_comm_analytical -> gqa_long_ctx_6cases_memory_comm_analytical.png
gqa_4cases_memory_comm_paired -> gqa_long_ctx_6cases_memory_comm_paired.png
gqa_kv_sharding_6cases_diagram -> gqa_long_ctx_6cases_kv_sharding_diagram.png
gqa_kv_sharding_6cases_table -> gqa_long_ctx_6cases_kv_sharding_table.png
gqa_3cases_measured_comm.json -> gqa_long_ctx_6cases_measured_comm.json
gqa_decode_long_ctx_4cases_*.png -> gqa_decode_long_ctx_6cases_*.png
(figures dir; long_ctx never had old)
New 4-chart 6-case set in long_ctx output dir (regenerated by
paper_plot_gqa_decode_long_ctx_4cases.py, which now reads all 6
sweep_decode.json panels — Cases 1-6 with the same colour scheme
used elsewhere: red = overflow per-PE HBM, grey = neutral, blue
= Pareto-best ★):
gqa_decode_long_ctx_6cases_latency.png
gqa_decode_long_ctx_6cases_memory.png
gqa_decode_long_ctx_6cases_parallelism.png
gqa_decode_long_ctx_6cases_traffic.png
Generator scripts updated to write the new filenames + handle the
two new d_head-TP variants (Cases 4, 5) in their per-PE memory and
active-PE-count helpers. Figure widths bumped 10 -> 12 in to fit 6
multi-line case labels.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Six PNGs covering the 6-case decode KV-placement story, copied
into both the kernbench 1H_milestone_output staging dir and the
1H-codesign-paper figures/ dir:
· gqa_hbm_budget.png - per-PE HBM budget bar
· gqa_4cases_summary.png - 3-panel: HBM + KV mem + comm
· gqa_4cases_memory_comm_analytical - 2-panel: KV mem + comm (analytical)
· gqa_4cases_memory_comm_paired - 2-panel: + measured overlay
· gqa_kv_sharding_6cases_diagram - 2-D KV-tensor view (Y=S_kv, X=d_head)
· gqa_kv_sharding_6cases_table - companion comparison table
The original 4-case gqa_4cases_summary.png is updated to the new
6-case 3-panel layout. Generator scripts and Phase-1 d_head-TP
kernels remain untracked - separate commit scope.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two strands bundled as the 1H-codesign-paper refresh unit:
(A) This session — single-op cost-model reflection (depends on 2d8271c):
- §2 Table 2 (tab:hw): split "FIXED per command" into "FIXED per
single-op command" (8 cycles) and "FIXED per composite command"
(40 cycles); §2 dispatch-overhead prose updated to the two-class split.
- §3.4 (sec:gemm-vs-async): rename paragraph headers + prose to
async-full / async-tiled; "atomic" -> "single-op" throughout; reframe
mechanism #3 from the old DMA-only fast-path to the single-op
fast-path. Headline narrative now: even with EVERY single-op cmd
(96 DMA + 48 dot + 47 add) charged the light 8-cycle FIXED, composite
still wins ~2.8x at K=3072 purely on command-count structure (1 vs
192 commands) -- down from the pre-D8 ~6.3x, and explicitly NOT a
modelling artifact. Numbers refreshed from the regenerated sweep:
async-full 3.83->3.91, async-tiled 1.14->~2.53, under-tile corner
1.06->1.21, depth-2 vs depth-inf spread <1%. New figure wired in.
- build/main.pdf rebuilt (tectonic); pdftotext-verified (no broken
refs; Table 2 split, single-op terms, 2.8x/2.53/192-host-commands
all present).
(B) Prior-session paper work riding along uncommitted: §4 all-reduce
deep-edit, §5 GQA, §6 discussion trims; milestone_1h_ccl.py plot
label "FSIM" -> "H2 2025 SW queue baseline"; regenerated diagrams
under docs/diagrams/** and gemm output PNGs under
1H_milestone_output/gemm/. (Composite-window gemm plots are
unaffected by D8 — D8 only changes single-op dispatch FIXED, which
the composite window excludes.)
All TODO items for the D8 single-op extension are now complete and
pushed across 3 commits (2d8271c cost-model+ADR+tests, 821bbf2 bench
harness, this paper refresh). Full regression green (826 passed, 1
skipped). No remaining work.
NOTE for review (carried from 2d8271c): ADR-0065's "2x CPU-offload win"
headline for GQA decode opt2 may want a refresh to the post-D8 ~1.87x.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0064 D8 broadened from "DMA fast-path" to "single-op-cmd fast-path":
every single-op command (DmaRead/DmaWrite/Gemm/Math/Copy) now pays the
lighter FIXED=8; only CompositeCmd keeps the 40-cycle control-path FIXED
(it alone needs scheduler plan generation + per-tile RW-hazard tracking +
completion wiring). Renamed knob fixed_per_dma_cmd_cycles ->
fixed_per_single_op_cmd_cycles. dispatch_cycles now branches on
"is CompositeCmd" rather than enumerating DMA types.
Term choice: "single-op" (not "atomic", which read as sync/async) — the
axis is composition (one engine op vs fused multi-op plan), orthogonal to
timing. single-op <-> composite.
Tests: test_pe_cost_model.py updated to the single-op surface (defaults,
fast-path over all 5 single-op cmd types, composite general path, yaml
override). All green.
Recalibrated tests/attention/test_gqa_decode_opt2.py
::test_opt3_dispatch_exceeds_opt2 — NOT a regression: D8 makes single-op
cmds 5x cheaper, so opt2's two-composite fusion win over opt3's many
single-ops narrowed from pre-D8 ~3.7x to ~1.87x (opt3=224 > opt2=120).
The CPU-offload invariant (opt2 cheaper) still holds; only the model-
dependent ">2x" constant was over-fit to the old uniform-40 model. Gate
now: direction + >1.5x margin (matches sibling R-sweep test's stated
"absolute ratio informative-only" philosophy).
NOTE for review: ADR-0065's "2x CPU-offload win" headline may want a
refresh to reflect the post-D8 ~1.87x — left to user (architectural doc).
Full regression: 826 passed, 1 skipped (tests/ excl. tests/gemm).
--- Remaining work (resume here if interrupted) ---
5. Re-run scripts/paper/paper_plot_gemm_async_vs_composite.py with new
cost model; verify async-tiled dispatch overhead drops (~4576ns ->
~1536ns expected) and the composite-vs-async-tiled gap narrows from
the prior ~6.3x at K=3072.
6. Copy regenerated gemm_composite_vs_async_tflops.png to
docs/report/1H-codesign-paper/figures/.
7. Paper §3.4 (03-gemm.tex sec:gemm-vs-async): finish naive->async-full /
chunked->async-tiled rename AND reframe FIXED_DMA wording to single-op
vs composite (currently still says "lighter FIXED for DMA descriptors,
FIXED_DMA=8"). Table 2 (02-platform) + §2 dispatch prose already done.
8. Paper §3.4 K=3072 corner para + mechanism #3: update dispatch breakdown
to new model (96 DMA*8 + 95 single-op*8 ≈ 1.5us vs old 4.6us); update
headline ratio if it changed.
9. Rebuild docs/report/1H-codesign-paper/build/main.pdf (tectonic) +
verify via pdftotext.
10. Then this is the bench-harness + paper commits (Groups 2 & 3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five-stage chain was set as a centered display equation, which
overflowed the column and clipped the last 'DMA_WRITE' label. The
inline prose right below it also repeated the same chain in a
shorter form. Consolidate into one inline parenthetical listing the
five stage names, and reuse 'flows through the chain' for the
self-routing description — no clipping, no repetition.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The matplotlib title and gray subtitle were duplicating what the
figure caption already says in the LaTeX source, and crowding the
top of every chart. Drop both — chart now leaves the title/subtitle
slots blank and the caption is the single source of truth for
chart titling.
Re-render all 5 GEMM figures (mac util measured, mac util theoretical
vs measured, stage breakdown, HBM BW util, per-PE TFLOPS) and resync
into docs/report/1H-codesign-paper/figures/. No data changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new evaluation plots for §3 GEMM, both using the composite window
as the denominator (the up-front tl.load of A in load_ref is therefore
excluded — only HBM traffic and compute inside the composite count):
- gemm_hbm_bw_util.png: per-PE HBM bandwidth utilization against the
256 GB/s per-PE ceiling. Answers "is this workload memory-bound on
this configuration?" — small/single-tile shapes sit at 22–46%
(pipeline-fill-limited), large or output-write-dominated shapes
saturate (≥85%).
- gemm_per_pe_tflops.png: per-PE GEMM throughput against the 8 TFLOP/s
engine peak. The deep-K K=3072 load_ref case lands at
7.18 TFLOP/s (~90% of peak) — the configuration's clean win.
ref_ref at the same shape drops to 3.83 TFLOP/s because the second
operand doubles HBM pressure and the kernel hits the BW ceiling.
The variant gap on this chart is the operational cost of NOT
pre-staging the activation.
Both charts compare two operand-staging variants on every shape:
load_ref ("activation pre-staged", weight-only HBM streaming) and
ref_ref (both A and W streamed from HBM). load_load is omitted —
with both operands pre-staged the composite carries no HBM traffic
and the BW metric collapses to 0.
Analytic / measured peak unified to the single hardware spec
(8.0 TFLOP/s = 8000 flops/ns). T_STAGE is now derived as
tile_flops / peak (= 16.384 ns for the 32×64×32 tile) rather than
the previous hardcoded 16 ns, so MAC efficiency and per-PE TFLOPS%
share the same denominator and never disagree. Side effect: max
analytic-vs-measured gap tightens from 2.2 ppt to 1.4 ppt across
the sweep.
§3 Results restructured: lead with the two new evaluation plots
(BW saturation, achieved throughput), then the MAC-utilization
analytic-vs-measured chart serves as the validation step, then the
per-stage engine wall-clock as the supporting diagnostic. §3
Analysis updated to use the load_ref / ref_ref contrast as the
hardware-software boundary line that motivates the GQA kernel of §5.
§2.4 Accuracy claim updated to match: GEMM analytic-vs-measured
agreement is now "within 1.4 ppt across every swept shape", from
~7.7% at single-tile up to ~90% at K=3072. Subtitle/title vertical
stacking glitch in the matplotlib charts fixed in passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three coupled fixes that recover small-tile GEMM pipeline efficiency
from 53% to 88% (32x3072x32 load_ref, composite_window basis).
1. PE_DMA channel-hold (ADR-0014 D4 clarified): both the
_handle_with_hooks (PeInternalTxn) and _pipeline_process
(TileToken) paths used to hold the cap=1 DMA channel through the
full HBM round-trip, which double-serialized with the HBM_CTRL's
own per-PC `available_at` model and prevented back-to-back tile
DMAs from amortizing their per-request head latency. Channel is
now released after the request is enqueued onto the next hop;
HBM serialization is HBM_CTRL's responsibility alone.
Tests: new test_pe_dma_back_to_back_pipelining as the oracle
(asserts wall < 75% of strict-serialized N x single_op). Existing
test_pe_dma_record_start_after_channel_acquire rewritten to assert
t_start clustering (channel released fast) instead of the old
round-trip-hold invariant. test_pe_dma_same_channel_serializes
still passes — HBM_CTRL preserves ordering.
Probe regression: PE→local-HBM 32 KiB stays at 141 ns
(single-request, unaffected).
2. milestone_1h_gemm bench: matmul_composite was reading MATMUL_M/K/N
env vars at module load, so every sweep row replayed the cached
256³ result; values now read inside run(). Drops the stale
sys.modules deletion hack.
3. Analytic ideal-pipeline model: dropped the (n_mn-1)·dma_w_per_pair
penalty (over-pessimistic for under-tile shapes — it pushed
measured > theoretical) and replaced the D_STAGES-derived head
with empirical T_PIPELINE_FILL=60 ns / T_PIPELINE_TAIL=30 ns.
Max analytic-vs-measured gap across all 7 swept shapes now 2.2 ppt
(was 9-44 ppt under the old constants).
Paper updates:
- §3 (GEMM): 78%→88% measured at 48 tiles, 23%→15% at 1 tile,
stage breakdown numbers refreshed (DMA in / Fetch / GEMM all
~785 ns at K=3072), analytic-vs-measured agreement tightened
to "within 2.2 ppt".
- §2.4 (Accuracy): GEMM tracking claim refreshed accordingly.
- §5 (GQA): restore long-ctx 4-cases figures into figures/
(they were dropped from bench output dir as derived artifacts in
92b9221 / e45626c but §5 still cites them by name).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Splits the GQA helpers into a dedicated subpackage to make room for the
prefill 4-cases study (next commit) and a single umbrella bench
(milestone-1h-gqa, after that).
Layout:
benches/gqa_helpers/
long_ctx/ — decode 4-cases kernels + sweep runner
short_ctx/ — prefill/decode short-context kernels
shared/ — _gqa_panel_helpers + decode_opt2 (context-agnostic)
The registry audit now skips subpackages so gqa_helpers/ (without a
leading underscore) doesn't get audited for @bench decorators.
Also drops the legacy milestone-gqa-headline bench, its
_gqa_attention_prefill_long kernel, 6 dependent prefill tests, the
paper_gqa_latency.py report harness, and the 3 stale headline-derived
PNGs the §6 wire-up referenced (paper will re-pull from the new
1H_milestone_output/gqa/long_ctx/ once §6 is updated).
The _ccl_cfg and _summarize_op_log helpers used to live in the
headline bench; extracted them to gqa_helpers/shared/_gqa_panel_helpers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Probe table (Table 1, tab:probe-pe-dma) promoted to table* so it
spans both columns and no longer gets clipped at the column edge.
- Fig 1 (hw-arch) row 1 (SIP, CUBE subfigures) trimmed: widths
0.495 -> 0.42 textwidth and a height=0.6\linewidth cap added so
the figure occupies less vertical space and floats can land
earlier in the document.
- Fig 1 and Fig 2 source blocks relocated to the top of §2 so they
enter LaTeX's float queue before any §2 body text. As a result
Fig 1 now lands on page 3 and Fig 2 on page 4 (was 4 and 5).
- Table 2 (tab:hw, modeled hardware config) reverted to plain
\begin{table}; the previous \begin{table*} reshape was not
requested.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- subfig widths 0.46 -> 0.495 textwidth so the \hfill gap between
(a) SIP and (b) CUBE shrinks to ~2% textwidth instead of ~8%.
- Removed the height=0.85\linewidth cap on both subfigures; with the
cap, the (roughly 1:1) SIP and CUBE diagrams were being scaled
down to 85% even when the column had room. Let them render at
their natural aspect ratio.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§2 platform:
- Accuracy promoted from \paragraph to \subsection (sec:accuracy);
it sits at the same heading level as Why / Device / Latency / HW
config and so reads as a first-class component of the platform
description rather than a tail-end footnote.
- Three architecture diagrams (SIP, CUBE, PE) collapsed into one
figure* using subcaption: Fig.~\ref{fig:hw-arch}(a) SIP and (b)
CUBE side-by-side on row 1, (c) PE wide on row 2, total height
capped at ~40% of the page. Subfigure cross-references rewritten
to Fig.~ref{fig:hw-arch}\subref{...} in the body text. main.tex
now pulls in the subcaption package.
- Table 1 (modeled hardware configuration) promoted to table*
(two-column / full page width) and the row layout rewritten as a
4-column tabular so Hierarchy + PE + Command-issue sit on the
left and Memory + Interconnect sit on the right. The previous
single-column rendering was getting cut at the right edge of the
printed column.
§6 GQA:
- Removed the historical four-panel headline table (tab:gqa), the
two figures (fig:gqa-lat, fig:gqa-break), and the prose paragraph
that cited their per-panel numbers. Reason: the underlying
milestone-gqa-headline bench has been simplified in collaborator
commit 65c365f ("drop misleading single_user_/multi_user_
panels") and no longer reproduces that data, so the section was
left referencing a dataset the current bench cannot regenerate.
- The previously-added 4-cases long-context decode comparison is
now §6's only Results subsection, retitled to
"Results: long-context decode and parallelism strategies".
The headline GQA result that survives into the paper is therefore
the parallelism trade-off study (Case 4 / Cube-SP × PE-SP at 34 us
buying an 8x KV-memory reduction over the latency leaders).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The collaborator commit 7c346de added the comparative figures
(gqa_decode_long_ctx_4cases_{latency,memory,traffic}.png) but the
fused-GQA section still only referenced the four headline panels.
This commit closes that loop:
- New §6 subsection "Long-context decode: parallelism strategies"
added between Results and Analysis. It lays out the four
parallelism strategies (Cube-SP/Repl x PE-TP/SP) and pulls in the
three new figures.
- The discussion highlights the central trade: the fastest strategy
(Case 3, Cube-Repl x PE-SP at 20.2 us) requires replicating the
full KV cache to every CUBE, while Case 4 (Cube-SP x PE-SP, the
chosen design marked *) gives back ~14 us in exchange for an 8x
KV-memory reduction. Case 4's ~190 IPCQ copies + ~190 DMA reads
are precisely the on-device collective traffic PE_IPCQ and the
torus links of §5 are provisioned to absorb -- a direct payoff
of the communication-side codesign work.
- Connects back to §5 (PE_IPCQ / all-reduce) so the reader sees the
capstone arc: the GEMM enabler exposes the data-movement bound,
the communication enabler attacks it, and the long-context
parallelism study shows how the choice between the two extremes is
framed by KV memory vs. on-device collective traffic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§2 platform:
- Add a "memory-centric AHBM" sentence to the device-and-execution
intro: each PE is paired with dedicated HBM bandwidth and on-PE
TCM, so the performance question is about feeding compute from
locally-attached memory + moving the unavoidable inter-PE traffic
efficiently. Makes the AHBM character of the platform visible
before §2.3 starts unpacking the latency model.
- Promote "Congestion and contention modeling" from \paragraph to
\subsubsection: this is the platform's core differentiator over a
peak-BW roofline, so it deserves its own heading.
- Rename "Control-plane (issue) cost model" to "Command dispatch
overhead model" -- describes what it actually charges (the PE_CPU
paying a fixed + per-byte cost to push a command to one of the
accelerator engines) without the more abstract "control-plane"
framing.
- Add a third Accuracy cross-check from kernbench probe: a
PE→HBM DMA-distance sweep that confirms monotonic hop progression,
bandwidth saturation matching the per-edge model, and the built-in
invariants (D2H >= H2D, cross-CUBE best < worst). New
Table~\ref{tab:probe-pe-dma} reports per-traversal latency and
utilisation at 32KiB / 1MiB across five hop classes.
- Captured probe output as figures/probe_pe_dma_summary.txt for
reproducibility.
§5 PE_IPCQ / all-reduce:
- Re-ran milestone-1h-ccl on current sim_engine (post-ADR-0064 Rev2
and the IPCQ slot-wrap Phase-2 race fix). Updated the topology-
comparison table and the buffer-kind caption to the fresh raw
latencies. Headline ratios are preserved:
torus vs mesh saving at 96 KB/PE: 24.8% (was 24.8%) -> ~25%
torus vs ring saving at 96 KB/PE: 19.3% (was 19.3%) -> ~19%
TCM vs HBM saving at 64 KB/PE: 13.4% (was 13.9%) -> ~13%
TCM vs SRAM saving at 64 KB/PE: 37.2% (was 38.3%) -> ~37%
The Executive Summary's "up to ~14%" / "up to ~38%" framing stays
consistent with these post values.
Artifacts refreshed in src/kernbench/benches/1H_milestone_output/:
- ccl/summary.csv + per-topology PNGs + buffer-kind CSV/PNG
- ccl/comparison_mesh_vs_ring_vs_2DTorus_vs_theoretical_vs_fsim.png
- gemm/* (re-run yields identical pe_window structure; PNGs
refreshed)
Paper figures synced to the fresh artifacts:
- figures/allreduce_comparison.png
- figures/allreduce_buffer_kind.png
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Generated by scripts/paper/paper_plot_gqa_decode_long_ctx_4cases.py
from the bench's sweep.json (S_kv=8192, the headline 128K run was
sidestepped — Case 2's single-PE 128-tile sweep is the long pole
and shape/ordering is preserved across scales).
gqa_decode_long_ctx_4cases_latency.png end-to-end latency / case
gqa_decode_long_ctx_4cases_traffic.png ipcq / dma op-count breakdown
gqa_decode_long_ctx_4cases_memory.png KV bytes per cube (slide-11)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Reordered §2 subsections to follow the SIP → CUBE → PE → graph
flow: Why KernBench → Device and execution model → Latency model
→ Modeled hardware configuration. Readers now meet the device
hierarchy before the graph abstraction that re-uses it.
- §2.2 Device and execution model: starts with the SIP/CUBE/PE
hierarchy paragraphs (each anchoring fig:sip-arch, fig:cube-arch,
fig:pe-arch); then the runtime-API/sim-engine/components bullet
list; then the atomic-vs-composite command distinction (corrects
the prior over-narrow framing that read every PE command as
composite -- atomic single-engine commands exist too, and PE_CPU
itself runs control-plane work directly).
- §2.3 Latency model: opens with the four-contribution decomposition
(per-node overhead, per-edge transmission, drain, queuing delay)
and the latency_model schematic; retains existing The hardware as
a graph / From graph to DES / Latency contributions / Congestion
/ Control-plane cost model / Accuracy paragraphs. Accuracy
paragraph now closes on KernBench's sufficiency for *relative*
HW/SW design trade-offs given analytic + external-simulator
agreement.
- New figures and assets:
- figures/sip_architecture.pdf (SIP-level graph view)
- figures/cube_architecture.pdf (CUBE-level zoom-in)
- figures/latency_model.png (conceptual latency-model
schematic with per-node /
per-edge / drain / queuing-delay
colour coding)
- figures/pe_architecture.png (carried over)
- Source-of-truth generator for the latency schematic:
scripts/paper/paper_latency_model_diagram.py (a report-only
harness under scripts/paper/ per the /paper isolation rule).
- main.tex preamble: \usepackage{tikz} added (kept from prior
sequence-diagram draft -- harmless now that the latency model is
a PNG; left in to keep paragraph numbering stable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mid-pass through §2. Captured progress so far:
- §2 intro previews the three new threads (graph view, DES engine,
congestion) the rewritten latency section weaves together.
- §2.3 retitled to "Latency model: a graph traversed by events" and
restructured into bold-led paragraphs:
- The hardware as a graph (nodes = components, edges = links).
- From graph to discrete-event simulation (node/edge events,
deterministic ordering, correlation-ID trace).
- Latency contributions (per-node fixed, per-edge size-aware,
per-service occupancy).
- Congestion: per-edge FIFO BW occupancy, HBM per-PC parallelism,
component serial workers — the mechanisms that surface real
bottlenecks instead of peak-BW roofline.
- Control-plane cost model (FIXED + b·R) — unchanged.
- Accuracy: extended with a second cross-check from the all-reduce
study (torus vs. analytic startup-plus-per-packet model + FSIM
external single-device reference). Existing GEMM analytic-vs-
measured 10-20% check retained.
- §2.4 Hardware-configuration table unchanged.
Still TODO: cross-check accuracy numbers against a fresh
milestone-1h-gemm/ccl re-run; the current artifacts predate the
ADR-0064 Rev2 cost-model and IPCQ Phase-2 race fix and may need a
refresh before §2.3 final lock-in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- §1 Introduction rewritten with AHBM-first opening, in-paper KernBench
platform overview, GQA motivation and three architectural requirements,
three matching HW-SW co-design mechanisms (GQA-aware placement,
PE_IPCQ, composite-command GEMM under PE_SCHEDULER), broader-
applicability outlook (PE_IPCQ as collective substrate, composite
command for fused operator pipelines), and outline. Consistent with
the Executive Summary structure.
- Executive Summary: extended broader-applicability closing with a
forward-looking line previewing FFN/MoE integration.
- §5 section heading renamed from
"All-Reduce Acceleration via PE_IPCQ" to
"PE_IPCQ and Collective Communication".
- Title page date line refined to "2026 H1 Report".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The reconciliation table, corrected file/test lists, measured 5.22x ratio, and the new section 15 (data-mode numerics D8 + N1-N4) — the content the git-mv in 30a0451 left uncommitted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updated the detailed design to match the shipped implementation: an as-built reconciliation table (P1-first, pinned-based DMA, D7 hard-cap-error, D8 output handle, prologue 1-stage-tile feed, D6.7 prologue-scoped), corrected file/test lists, the measured 5.22x dispatch ratio, and a new section 15 documenting the data-mode numeric subsystem (D8 + N1-N4) the original DDD omitted. Status Draft->Accepted; moved docs/adr-proposed -> docs/ddd/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tl.composite now returns the output TensorHandle (not a CompletionHandle) so its result chains like tl.dot's; the handle carries the completion in a CompositeFuture pending so downstream ops and tl.wait auto-await it. out is a handle: out=tl.ref(addr,shape) (HBM, DMA_WRITE inside the composite) or an in-place TCM handle (STORE only); omitted -> TLContext auto-allocates a TCM scratch. out_ptr kept as HBM shorthand (= out=tl.ref(out_ptr, shape)) to avoid churning ~30 existing call sites.
tl.ref now returns space=hbm (it references HBM data; operand-input DMA stays pinned-based per D4 so input streaming is unchanged). tiling: the tile loop's DMA_WRITE is gated on out.space==hbm (out analog of the operand pinned rule) — a TCM output stays on-chip (chainable) and its high-bit scratch address no longer hits the DMA PA decoder.
Fixes the opt2 data-mode crash: the recipe accumulator O is TCM -> no DMA_WRITE -> opt2 now RUNS end-to-end in data mode (enable_data=True). Numeric parity of the recipe MATH ops is the next step. Suite 812 pass / 3 pre-existing fail; existing composite benches use out_ptr->hbm->DMA_WRITE unchanged (byte-equal).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New `/paper` slash-command skill that synthesizes ADR/SPEC content and live
KernBench benchmark results into a sectioned LaTeX technical paper compiled
to PDF with Tectonic (auto-installed). The skill negotiates a TOC, grounds
every number in committed artifacts or fresh bench runs, and keeps
report-only benches isolated.
This commit also includes the first generated report:
- docs/report/1H-codesign-paper/ — main.tex + per-section .tex, figures,
toc.md contract, and the built 8-page main.pdf. Covers the platform
(source-level kernels, latency model + accuracy, HW config from
topology.yaml), GEMM via composite command, All-Reduce via PE_IPCQ, and
fused GQA combining both, plus discussion/conclusion/2H future work.
- scripts/paper/ — isolated report harnesses (not registered benches):
paper_gqa_latency.py harvests per-panel GQA end-to-end latency + engine
occupancy (the milestone only emitted op-counts); paper_plot_gqa.py
renders the GQA figures.
GEMM/All-Reduce reuse committed milestone figures/CSVs; GQA results are
generated fresh. Honest flags retained: PE_CPU dispatch cost is 0 in this
config, and the proposed two-composite softmax_merge decode is marked
designed-not-measured.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tiling.generate_plan_from_ops: scan flat ops for the GEMM (<=1); pre-GEMM KERNEL ops become single-shot prologue MATH stages, post-GEMM ops split by scope (K_TILE/OUTPUT_TILE epilogue + KERNEL post-loop). MATH-only composite reproduces the legacy math-head plan. Prologue/post-loop stages fold into the first/last tile so the feeder + completion counting are untouched (existing benches have neither -> byte-equal op_log).
PipelinePlan gains prologue_stages/epilogue_stages. pe_scheduler._generate_plan delegates to generate_plan_from_ops. DMA keeps the existing pinned signal (NOT a space flip); recipe scratch/primary-out handles are pinned=True so the head GEMM's auto-bound a (=P) is consumed in place.
ADR-0065 D4/D6.7/Test#5 amended (EN+KO): DMA decision from pinned, not space; prologue recipe ops TCM-only (head op exempt -- it may stream from HBM).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New triton_emu/tl_recipes.py: RecipeDescriptor/EngineOp/PrimaryOutSpec + RECIPE_DESCRIPTORS['softmax_merge'] (8-step engine_seq). PE_SCHEDULER does not import it (ADR-0065 D5 boundary).
TLContext.composite(): add prologue=[...] + out=TensorHandle kwargs, a optional. _expand_prologue lowers a recipe into flat MATH OpSpecs (scope=KERNEL), allocates TCM scratch, derives the primary-out slot 'P' and auto-binds it into the head GEMM a (D6.6 conflict check); rw_handles=(m,l,O). decode-opt2 #2 lowers to 10 ops [rmax,max_elem,exp_diff,exp_diff,rsum,fma,mul_bcast,copy,gemm,add].
D6.7 (MATH operand TCM-only) scoped to prologue recipe ops only — the head op (gemm or math) keeps existing DMA-staged-from-HBM behavior. D6.1 (GEMM count <=1) on the whole composite. Host-side lowering only; PE_SCHEDULER position-scan is P3.
Also commit the ADR-0064 Rev2 promotion content that the prior commit's git mv dropped: Status Proposed->Accepted + D7 amended to hard-cap ValueError (no segmentation), EN+KO.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0064 Revision 2 second-pass review fixes:
- D7 added 1024-byte cap rationale: explicitly framed as a *safe
engineering limit* (intentionally above all known composites at ~322
bytes, finite descriptor capacity placeholder), not a measured HW
number. Default is the discipline; topology override for real HW.
- D7 segment ordering: strict FIFO is the *ordering source*; rw_handles
is dependency metadata, not the ordering primitive. Explicit note for
the future RW-aware reorder migration path.
- Tests rewritten around the formula (D1) instead of specific numbers:
- Test #1 → "formula preservation" (parametrized across composites,
asserts dispatch == FIXED + bytes × R, not anchor 43 ns).
- Test #2 → robust "opt3 > 2 × opt2" qualitative gate; ≈ 4.0× is
informative only, not the gate, so calibration changes don't break.
- Test #9 → qualitative (opt2 < opt3 at all R), not a numeric ratio.
- Anchor description in D3 stays as *informative* (shows the model
produces ≈ 43 ns at defaults), but is no longer a test gate.
ADR-0065 second-pass review:
- D5 step 5 aligned with D6.6 invariant: explicit conflict check before
auto-binding (previously written as if auto-bind always happens).
Mirrors D6.6 wording: error if both kernel explicit `a` and prologue
primary_out are present.
- Test #7 made robust: `opt3 > 2 × opt2` gate (was ≈ 4.0×).
Default-calibration model expectation (≈ 4.0×) recorded as informative
reference, not the gate.
DDD-0065 follow-on robust thresholds:
- §1.4 success criteria: ratio condition rewritten as `opt3 > 2 × opt2`
with model-expected ≈ 4.0× as informative.
- §9 P6 phase gate: `opt3 > 2 × opt2` (was 4.0× ± 15%).
- §10 dispatch ratio test: assert `opt3 > 2 × opt2`; record observed
ratio for performance tracking but do not gate on it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ADR-0065 review fixes:
- D3 semantics clarified: position determines *phase* (pre-loop / head /
in-accumulation / per-output-tile / post-loop), scope determines
*repetition*. KERNEL = "once per CompositeCmd invocation", not "once
per kernel launch" — removes ambiguity around softmax_merge's KERNEL
scope at different positions before/after GEMM.
- D4 MATH-TCM invariant: Phase 1 limits DMA auto-insertion to GEMM
operands and head outputs; MATH operand with space="hbm" raises
validation error at TLContext emit. Catches kernel misuse early; a
MATH-with-HBM path is future work (explicit tl.load + MATH composite).
- D6 #6 NEW: auto-bind conflict — explicit GEMM `a=` + prologue
primary_out simultaneously → validation error (prevents "which value
wins" ambiguity).
- D6 #7 NEW: MATH operand TCM-only restatement.
- Consequences/Negative: strict-FIFO conservatism note — safe but may
under-expose composite-level overlap when handle-sharing composites
could in principle run in parallel.
- Test Requirements #10, #11: auto-bind conflict + MATH-HBM validation
error tests.
ADR-0064 Revision 2 clarification:
- D2 counting rule: per-op summation, no deduplication. Same handle
appearing in multiple OpSpecs or in rw_handles is counted
independently — matches HW reality (each descriptor field is a
distinct address slot, rw_handles is separate metadata block).
Example walked through (mul_bcast with O in src_a + out + rw_handles).
DDD-0065 follow-on:
- §3.2: new test file test_tl_composite_validation.py covers auto-bind
conflict, MATH-TCM, GEMM-count invariants.
- §4.3: phase vs repetition split aligned with ADR-0065 D3.
- §6 lowering pipeline: step 5 adds auto-bind conflict check; step 8
NEW — operand space validation; step 9 (size cap); step 10 (emit).
- §8 strict-FIFO conservatism note: explains why next-tile #1 cross-tile
pipelining still works under FIFO (no rw overlap), and where the
conservatism would hurt (future handle-sharing recipes).
- §10 verification: invariant-guard tests (auto-bind conflict, MATH
TCM, per-op summation accuracy).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ADR-0064 Revision 2 review fixes:
- D7 NEW: composite size cap (MAX_COMPOSITE_LOGICAL_BYTES default 1024
bytes). Oversized recipes deterministically segmented into N
CompositeCmds; each segment incurs its own dispatch cost. Models real
HW limits (descriptor queue entry, scheduler parser buffer, command
SRAM) and prevents the model from rewarding pathologically-large
fused composites.
- D2: type-aware extra-field byte counting (int/float=4, bool=1,
tuple/list=1+4N, str=1) — replaces uniform 4 bytes per extra.
- D3: recalibrated defaults to FIXED=40 cycles, R=0.0625 cycles/byte
(16 B/cycle — typical on-die descriptor queue width); anchor stays at
~43 ns for typical 1-OpSpec composite. Clarified anchor description:
DMA stages do not appear in logical_bytes (auto-inserted by
PE_SCHEDULER from operand.space per ADR-0065 D4).
- D4: removed clock_freq_ghz from pe_cost_model: override block;
conversion uses the PE node's existing clock_freq_ghz attr. Added
max_composite_logical_bytes knob.
- Context: emphasized command-count reduction (FIXED) as the primary
signal; byte term as secondary refinement.
- Open review: added large-composite scheduler-cost stress test.
- Test req: added composite-size-cap (#8) and R-sensitivity sweep (#9).
ADR-0065 + DDD-0065 follow-on updates:
- opt2 vs opt3 dispatch ratio updated 2.4× → ≈4.0× under new defaults
(FIXED-dominated, reflecting the corrected framing).
- Test req #9: decode opt2 composite fits within 1024-byte cap; no
segmentation needed for the GQA workload.
- DDD §6: TLContext lowering checks logical_bytes against cap (step 8).
- DDD §11: performance model recomputed with new defaults + sensitivity
table across R ∈ {0.25, 0.0625, 0.03125} confirming opt2 < opt3 holds.
- DDD §9 P6 gate: ratio band 2.4×±10% → 4.0×±15%; sensitivity sweep added.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three logically distinct changes, bundled for atomic test green:
1. **P3c — prefill_long tile-granular Ring KV** (ADR-0060 §5.5.1 amendment).
Convert the ring from slice-granular (one full ``(d_head, S_local)``
KV slice per step) to tile-granular (``n_tiles`` tiles of
``TILE_S_KV`` per step). Nested loop with outer tile, inner ring step:
each tile propagates through all C ring positions before the next
tile starts, so IPCQ in-flight depth stays at 1 per direction.
Bootstrap at ``(t=0, k=0)`` outside the scratch_scope establishes the
persistent ``(m, ℓ, O)``; every other iteration scope-wraps + persists
via ``copy_to``. Per-rank persistent scratch shrinks to ~1 KB; per-tile
scope bounded by TILE_S_KV regardless of S_local. Headline:
prefill_long now completes at S_kv=128K (previously overflowed).
New: ``tests/attention/test_gqa_prefill_long_tile_ring.py``
(3 tests — ceiling-lift + tile-granular ipcq_copy count +
per-CUBE distributed output regression guard).
2. **Rename ``gqa_*`` → ``gqa_attention_*``** across kernel files,
function names, and importers. The "attention" name makes the role
explicit (GQA is grouped-query attention) and matches upstream Triton
FlashAttention naming conventions. Renames:
_gqa_decode_long.py -> _gqa_attention_decode_long.py
_gqa_decode_short.py -> _gqa_attention_decode_short.py
_gqa_prefill_long.py -> _gqa_attention_prefill_long.py
_gqa_prefill_short.py -> _gqa_attention_prefill_short.py
And function names ``gqa_<phase>_<context>_kernel`` →
``gqa_attention_<phase>_<context>_kernel``. Updated 1 bench file
(milestone_gqa_headline.py) and 10 test files.
3. **ADR-0060 / 0062 / 0063 / 0064: Proposed → Accepted**.
All four are reflected in production code and covered by tests:
- ADR-0060 (GQA fused attention): 4 kernels deployed; §5.5.1
amendment added for the tile-granular Ring KV introduced by P3c
(EN + KO mirror).
- ADR-0062 (lazy tl.load): LoadFuture + _await_pending live in
tl_context.py.
- ADR-0063 (tl.scratch_scope + tl.copy_to): used in every chain
reduce + tile sweep + ring step. EN-only previously; KO
translation authored as part of this commit (CLAUDE.md
bidirectional rule).
- ADR-0064 (per-op-type CPU issue cost): cpu_issue_cost.py +
issue_cost_table wiring in tl_context.py (Phase E).
Files git mv'd from docs/adr-proposed/ to docs/adr/ (EN) and
docs/adr-ko/ (KO). ADR-0061 (tl.broadcast) stays Proposed — no
implementation; documented as optional convenience primitive in
the ADR itself.
Tests: 88/88 focused regression green
(tests/attention/ + Phase E + TL discipline).
ADR pair verification: ``python tools/verify_adr_lang_pairs.py`` OK.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Land the new GQA fused-attention kernels (ADR-0060) for prefill/decode
across long and short context, the TL discipline primitives they depend
on (ADR-0062 lazy load, ADR-0063 scratch_scope + copy_to), and the
per-op-type CPU issue cost model (ADR-0064). Remove the pre-ADR-0060
mesh-attention baseline now that the unified kernels supersede it.
ADR-0060 (long context)
- _gqa_decode.py: M-fold + 2-level chain reduce-to-root (Level-2
intra-CUBE row-then-col + Level-1 inter-CUBE) — root-only output.
- _gqa_prefill.py: head-parallel + Ring KV rotation around C CUBEs,
online-softmax merge per ring step, per-CUBE distributed output.
- Each merge stage wraps in scratch_scope() and persists running
(m, l, O) via copy_to() to lift the 1 MiB scratch ceiling.
ADR-0060 §B.split.2 (short context, kv_per_cube in {1,2,4,8})
- _gqa_decode_short.py / _gqa_prefill_short.py: no cube-SP; each CUBE
owns whole KV heads; PE-parallel heads with intra-group chain
reduce. Prefill has no Ring KV (each head fully resident).
ADR-0062 (lazy tl.load): future-bearing TensorHandle, auto-wait at
first consuming op (dot/MATH/store/send/copy_to/composite).
ADR-0063 (tl.scratch_scope + tl.copy_to): scoped per-tile arena with
copy_to writeback primitive for persistent running state.
ADR-0064 (CPU issue cost model)
- common/cpu_issue_cost.py: per-op-type table (composite=40 ns,
primitives=5 ns); ratios are load-bearing per D1.
- TLContext: issue_cost_table param; _emit_dispatch_overhead(kind)
consults table with dispatch_cycles fallback (ADR-0046 §D6
back-compat).
- Live PE_CPU paths (greenlet + legacy) construct TLContext with
DEFAULT_CPU_ISSUE_COST so saturation lever (ADR-0060 §1) is
measurable end-to-end.
P7 headline bench: milestone-gqa-headline writes per-panel
op_log_summary to 1H_milestone_output/gqa_headline/sweep.json. No
figure renderers yet (deferred).
Removals (pre-ADR-0060 baseline now superseded):
- benches: _attention_mesh_kv.py, _attention_mesh_mlo.py,
_attention_mesh_mlo_2d.py, milestone_gqa_llama70b.py
- tests: test_attention_*, test_mesh_*, test_milestone_gqa_llama70b
- topology: llama70b_4sip.yaml (only consumer was the deleted diag)
- artifacts: 1H_milestone_output/gqa/ (sweep.json + 5 PNGs)
- tests/gqa/ plot helper + test (broken on Windows Tcl/Tkinter)
- ADR-0060/0061 references to deleted file paths cleaned up
(EN + KO kept in sync).
Tests: 124/124 focused regression green (attention + Phase E + TL
discipline + triton_emu + pe_components). Full regression: 764 pass,
2 pre-existing test_bench_registry failures (stale EXPECTED_NAMES
across multiple benches, not introduced here).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Amends ADR-0026 to reflect the cube_start field added in e2fe331.
The production code shipped without an ADR update; this fills that
gap. Documentation-only change (no production code, no test code).
Changes (mirrored in both EN and KO):
- Status: Revision 5 → Revision 6
- D1: add ``cube_start: int = 0`` to the canonical DPPolicy dataclass
- D3: ``cube = policy.cube_start + cube_id`` in resolve_dp_policy
- D8: new section explaining purpose (disjoint cube sub-meshes for
GQA Llama-70B 8-KV-group headline), semantics, default-0 backward
compatibility, intra-device constraint, design rationale
(scalar vs 2D origin vs cube_ids list), and the kernel-side
cube_start subtraction needed to compensate for ADR-0022's
physical-cube-id ``program_id(axis=1)`` semantics.
tools/verify_adr_lang_pairs.py passes (EN/KO Status keyword and
title in sync).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per review:
- Placement unified to *contiguous* C×P blocks throughout (S0/S2.1/S2.2/S0.5);
round-robin demoted to the rejected alternative. Adds driver SP-enable
threshold fallback (smaller C / C=P=1 for short/early decode).
- Ring-vs-reduce cost model in S5.5: reduce ~ G*T_q*log(C*P) (O dominant;
m,l scalars), ring ~ 2*S (total K+V bytes a CUBE injects over C-1
rotations; recv_async pipelines so latency ~ max-step) -> ring wins when
T_q > 2S/(G*log(C*P)).
- opt3 'removes the bubble' -> 'hides (subject to scheduler+engine balance)'
everywhere; table 'hidden*' with footnote.
- 'rank' defined (SP participant = a PE in a CUBE, KV shard in its HBM->TCM).
- out-proj handoff contract (S0.5.4); S11 gate-type note (absolute latency
deferred to ADR-0064; structural + relative-to-baseline gates now).
- greenlet-as-contrast tightened to 'primitive-op (tl.dot) path' (S1, SB).
KO mirror synced; DDD gets the SP-enable threshold fallback. Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DDD-0060 was rewritten to the two-kernel composite-hybrid design, so SB
item 1 ('DDD not yet synced') is stale. Update EN + KO mirror to record the
DDD as synced (ADR authoritative, DDD = impl how-to). Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Supersede the old greenlet-primitive/load_async/single-kernel DDD. Align to
ADR-0060's composite-hybrid + hierarchical CUBE-Group SP with two kernels:
decode+SP (head-replicated, contiguous C×P static shard, 2-level
reduce-to-root) and prefill+SP (1 Q head/CUBE, Ring KV, no reduce). Updated
file plan (lazy tl.load, scratch_scope, broadcast-opt; evolve
_attention_mesh_mlo_2d/_kv; DPPolicy.cube_start; llama70b_4sip.yaml),
placement (contiguous shared KV), phase plan (P1..P8), verification, perf
model, risks, glossary. Defers design rationale to ADR-0060 (now
authoritative); open items point to ADR-0060 §B. ADR-0064 cost model noted.
Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Regenerate the KO mirror to match the current EN: two SP kernels
(decode=reduce / prefill=ring), TL;DR full code for the 3 decode variants
+ prefill, 'Q replicated / M-fold' and '1 Q head per CUBE' terminology,
contiguous shared KV layout, opt2 tl.wait, and all SB items. Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ywkang
mukesh