- ADR-0060: GEMMs (Q.Kt, P.V) via existing tl.composite (scheduler-managed
tiling + K/V DMA streaming); softmax merge + IPCQ tree reduction stay in
kernel. Front TL;DR pseudocode of the final composite kernel; new section
B lists open design items (DDD sync, K pre-transpose, dma_read lever,
kernel-vs-scheduler tiling, ring path).
- ADR-0062: redefined from a new load_async op to global lazy tl.load
(non-blocking + auto-wait on first use; API unchanged; goldens regenerate).
- ADR-0064 (new): per-op-type CPU issue cost model (composite ~40ns >>
primitive) so the hybrid's CPU-saturation win becomes measurable
(currently dispatch_cycles=0 hides it). Cost-model impl deferred.
- KO mirrors for ADR-0060/0062/0064 (-ko suffix, adr-proposed).
Rationale: non-blocking CompositeCmd offloads tiling to PE_SCHEDULER,
decoupling CPU issue-rate from execution so the CPU can saturate the
engines; the prior 'composite = no latency benefit' claim was an artifact
of dispatch_cycles=0. Docs only; no production code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified against sim_engine data path (memory_store, data_executor):
- GQA reuse does NOT need a broadcast op. The baseline's h_q==h_kv limit
is its head-packing reshape hack, not a missing primitive. Correct fix:
per-KV-head loop with G folded into matmul M dim (byte-conserving
reshape) — runs today, timing correct (m=G*T_q), data mode runs.
- ADR-0061 broadcast demoted from 'the blocker' to optional convenience.
- Surfaced tl.trans = reshape-not-transpose (memory_store reshapes;
data_executor np.matmul on reshaped operands) -> numeric parity is
bounded; verification is structural/timing/determinism-first (matches
SPEC perf-model purpose). Optional tl.transpose deferred.
- Reordered DDD phase plan (P1 GQA needs no new feature; P3 scratch_scope
is the key scale feature); added open decisions 10.10 (transpose) and
10.11 (GQA-via-M-fold finding).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reframes the proposed GQA FlashAttention design onto kernbench's actual
execution model (greenlet tl API + IPCQ), replacing the composite-centric
mechanism that does not match the simulator:
- Records relationship to existing baseline kernels (_attention_mesh_kv/mlo,
milestone-gqa-llama70b) and their 3 deliberate limitations.
- Mechanism is greenlet tl (per-op latency; no fusion benefit), not
composites; running (m,l,O) is Python handles; reduction is tl.send/recv.
- Tree reduction (log N) replaces baseline all-to-all fan-out (N-1).
- Pseudocode rewritten in real tl.* signatures; depends on ADR-0061/62/63.
- Rejects composite-IPCQ-push + composite-carried-state + flash-composite
with documented efficient alternatives.
- Adds verification plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Proposed prerequisites surfaced while evaluating the GQA fused-attention
ADR against the actual kernbench tl/sim_engine implementation:
- ADR-0061 tl.broadcast: data-faithful GQA head reuse (fixes the
MemoryStore nbytes check that forces h_q==h_kv==1 today).
- ADR-0062 tl.load_async: non-blocking HBM tile load for KV prefetch
(KV-load-bound decode/long-context overlap).
- ADR-0063 tl.scratch_scope: per-tile scratch recycling (removes the
1 MiB bump-allocator ceiling that caps context at S=16).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ywkang
Yangwook Kang