ywkang/kernbench2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gqa(adr): ADR-0064/0065 review fixes — composite size cap, type-aware logical_bytes, R recalibration

Browse Source 
ADR-0064 Revision 2 review fixes:
- D7 NEW: composite size cap (MAX_COMPOSITE_LOGICAL_BYTES default 1024
  bytes). Oversized recipes deterministically segmented into N
  CompositeCmds; each segment incurs its own dispatch cost. Models real
  HW limits (descriptor queue entry, scheduler parser buffer, command
  SRAM) and prevents the model from rewarding pathologically-large
  fused composites.
- D2: type-aware extra-field byte counting (int/float=4, bool=1,
  tuple/list=1+4N, str=1) — replaces uniform 4 bytes per extra.
- D3: recalibrated defaults to FIXED=40 cycles, R=0.0625 cycles/byte
  (16 B/cycle — typical on-die descriptor queue width); anchor stays at
  ~43 ns for typical 1-OpSpec composite. Clarified anchor description:
  DMA stages do not appear in logical_bytes (auto-inserted by
  PE_SCHEDULER from operand.space per ADR-0065 D4).
- D4: removed clock_freq_ghz from pe_cost_model: override block;
  conversion uses the PE node's existing clock_freq_ghz attr. Added
  max_composite_logical_bytes knob.
- Context: emphasized command-count reduction (FIXED) as the primary
  signal; byte term as secondary refinement.
- Open review: added large-composite scheduler-cost stress test.
- Test req: added composite-size-cap (#8) and R-sensitivity sweep (#9).

ADR-0065 + DDD-0065 follow-on updates:
- opt2 vs opt3 dispatch ratio updated 2.4× → ≈4.0× under new defaults
  (FIXED-dominated, reflecting the corrected framing).
- Test req #9: decode opt2 composite fits within 1024-byte cap; no
  segmentation needed for the GQA workload.
- DDD §6: TLContext lowering checks logical_bytes against cap (step 8).
- DDD §11: performance model recomputed with new defaults + sensitivity
  table across R ∈ {0.25, 0.0625, 0.03125} confirming opt2 < opt3 holds.
- DDD §9 P6 gate: ratio band 2.4×±10% → 4.0×±15%; sensitivity sweep added.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yangwook Kang
2026-06-10 16:20:58 -07:00
parent 7fad0371c5
commit 95cecccd8e
5 changed files with 296 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr-proposed/ADR-0064-perf-cpu-issue-cost-model.md
+127 -23
View File
@@ -56,6 +56,20 @@ N OpSpecs has ~N× the bytes of a primitive op with one OpSpec. The
*fixed* part (queue head update, completion register, MMIO-class
latency) is per-command. The two together compose: `FIXED + bytes × R`.
### What this model actually exposes
The **primary** signal is **command-count reduction** through the
per-command FIXED cost. The **byte** term is a secondary refinement
that prevents pathologically-large composites from looking free. With
realistic on-die queue bandwidth (16 B/cycle, D3), FIXED accounts for
≥85% of the dispatch cost differential between opt3 (≈10 cmds/tile)
and opt2 (≈2 cmds/tile) for decode opt2.
This framing also bounds the model: if a single composite were allowed
to grow unboundedly large, the byte term alone would not stop the
formula from rewarding ever-bigger fused commands beyond what real HW
supports — hence the descriptor-size cap in **D7**.
## Decision
### D1. Structural dispatch cost formula
@@ -113,39 +127,76 @@ def logical_bytes(self) -> int:
1 + 1 # opcode + scope
+ 1 + 8 * len(self.operands) # named operand handles
+ (8 if self.out is not None else 0) # out handle
+ 1 + sum(4 for _ in self.extra.values()) # extra scalars
+ 1 + sum(_extra_bytes(v) for v in self.extra.values())
)
def _extra_bytes(v) -> int:
"""Type-aware byte count for OpSpec.extra values."""
if isinstance(v, bool): return 1
if isinstance(v, (int, float)): return 4
if isinstance(v, (tuple, list)): return 1 + 4 * len(v) # shape, axes, …
if isinstance(v, str): return 1 # opcode-like tag
return 4 # default scalar
# Example types in extra:
# m, k, n int → 4 each
# reduce_axis int → 4
# shape=(64, 64) tuple → 1 + 8 = 9
# factor=1.0 float → 4
```
(Identical rule for `DmaReadCmd`, `MathCmd`, etc. — one property per
dataclass, ~3 lines each.)
### D3. Defaults — anchored on a typical composite ≈ 45 ns
### D3. Defaults — anchored on a typical composite ≈ 43 ns
Anchor: a typical 1-op DMA→GEMM→DMA composite has `logical_bytes ≈ 52`
(framing 4 + GEMM OpSpec 39 + rw_handles 9). Target dispatch = 45 ns.
On-die producer→consumer queue assumed at 4 bytes/cycle.
Anchor: a **single-OpSpec composite for a DMA-staged GEMM path** —
one `OpSpec(kind="gemm", ...)`; DMA stages are auto-inserted by
PE_SCHEDULER from operand `space` (ADR-0065 D4) and **do not appear in
`logical_bytes`** (the kernel does not issue them as separate cmds).
Breakdown:
```
clock = 1 GHz # 1 cycle = 1 ns
FIXED_PER_CMD = 32 cycles
R = 0.25 cycles/byte
framing 4
ops tuple length 1
GEMM OpSpec 40 (opcode 1 + scope 1 + 1 + 2 handles 16
+ out 8 + 1 + extra m/k/n 12)
rw_handles tuple length 1
rw_handles content 8 (one RW handle for the output)
─────────────────────────────
total ~54 bytes
```
Verification: `32 + 52 × 0.25 = 45 cycles ≈ 45 ns` ✓
Target dispatch = ~43 ns. On-die producer→consumer queue at 16 bytes/cycle
(typical on-die descriptor queue width).
```
FIXED_PER_CMD = 40 cycles
R = 0.0625 cycles/byte (= 16 bytes/cycle)
```
Verification: `40 + 54 × 0.0625 = 43.375 cycles ≈ 43 ns` ✓
Cycle→ns conversion uses the PE node's existing `clock_freq_ghz` attr
(the same one used by PE_MATH `_compute_ns`). The cost-model knobs are
**cycle-domain only** — they do not duplicate the clock setting.
### D4. Topology config override
Defaults are baked into `pe_cpu.py`. Topology yaml may override under a
`pe_cost_model:` section at the PE node attrs:
`pe_cost_model:` section at the PE node attrs (cycle-domain knobs only;
clock comes from the PE's existing `clock_freq_ghz`):
```yaml
pe:
attrs:
clock_freq_ghz: 1.0 # existing, used for cycle→ns
pe_cost_model:
fixed_per_cmd_cycles: 32
byte_cycles_recip: 0.25
clock_freq_ghz: 1.0
fixed_per_cmd_cycles: 40
byte_cycles_recip: 0.0625 # = 16 bytes/cycle
max_composite_logical_bytes: 1024 # D7 — descriptor size cap
```
Missing keys fall back to defaults. The dispatch formula reads from
@@ -170,6 +221,34 @@ latencies are **regenerated once** when this ADR lands — same posture as
ADR-0062 D3 lazy-load. After regeneration, the same calibration is in
effect for ADR-0065 opt2 measurement.
### D7. Composite size cap (deterministic segmentation)
Each `CompositeCmd`'s `logical_bytes` is capped at
**`MAX_COMPOSITE_LOGICAL_BYTES`** (default **1024 bytes**, overridable
per D4). Oversized commands are deterministically **segmented** by the
*emitter* (host-side TLContext, ADR-0065 D5) into N consecutive
`CompositeCmd`s, each ≤ cap.
- Each segment carries its own `completion: CompletionHandle`.
- Each segment incurs its own dispatch cost — `total =
sum(FIXED + bytes_i × R) = N × FIXED + total_bytes × R`. The FIXED
term is paid per segment.
- Segments share `rw_handles` where applicable; **strict FIFO**
(ADR-0065 D6.3) preserves write-after-write ordering automatically.
- The segmentation algorithm is deterministic (greedy by op index in
emit order); it is part of the host emitter, not PE_SCHEDULER.
Why this is needed. Real hardware imposes hard limits — descriptor
queue entry size, scheduler parser buffer, command SRAM, firmware
input. Without a cap, the `FIXED + bytes × R` model would reward
arbitrarily large fused composites beyond what hardware accepts (e.g.,
fusing 100 primitive ops into one composite, paying one `FIXED`).
A composite of (say) 100 ops at ~30 bytes each = ~3000 bytes >
1024 cap → segmented into 3 ≈ 3 × FIXED, restoring honest accounting.
Decode opt2's `#2` composite (10 ops, ~310 bytes) sits comfortably
inside the 1024 cap — no segmentation for the GQA workload.
## Alternatives
### A1. Keep Revision 1's op-type calibration table
@@ -218,22 +297,32 @@ calibration shows it matters.
## Open review items
1. **Calibration source for FIXED and R.** Defaults from "typical
composite = 45 ns + on-die queue 4 bytes/cycle"; reasonable for an
on-die producer→consumer queue. Revisit when a HW reference appears.
2. **Scheduler plan-gen cost.** Stays 0 — D5 keeps PE_SCHEDULER's
plan-generation outside the dispatch formula. Expose via existing
`overhead_ns` if a workload shows scheduler-bound behaviour.
composite ≈ 43 ns + on-die queue 16 bytes/cycle"; reasonable for an
on-die descriptor queue. Revisit when a HW reference appears.
2. **Scheduler plan-gen cost vs large composites.** Stays 0 — D5 keeps
PE_SCHEDULER's plan-generation outside the dispatch formula. The
D7 cap (1024 bytes ≈ 30–35 OpSpecs) bounds the worst case, but a
composite near the cap still costs the scheduler the same as a 1-op
composite under the current zero-cost model. If a stress test
(large-composite microbench) shows scheduler-bound behaviour,
expose `overhead_ns` per-op-count.
3. **Where the override lives.** `pe_cost_model:` block under PE node
attrs in topology yaml — keeps all knobs in one place, reviewable.
Clock comes from the PE's existing `clock_freq_ghz` attr, not
duplicated here.
4. **Path parity.** Both greenlet (`_execute_legacy` and `kernel_runner`)
and replay paths must read the same cost model. Verify.
5. **Sensitivity of conclusions to R.** opt2 < opt3 must hold across a
reasonable range of `R` (queue-bandwidth assumption). Sensitivity
sweep is part of Test Requirements (#9).
## Test Requirements
1. **Anchor preservation.** A typical DMA→GEMM→DMA composite (1 op,
`logical_bytes ≈ 52`) dispatches in 45 ns at the default values.
1. **Anchor preservation.** A single-OpSpec composite for a DMA-staged
GEMM path (`logical_bytes ≈ 54`) dispatches in ≈43 ns at default
values (within ±2 ns).
2. **Structural ratio.** opt3 vs opt2 dispatch (per ADR-0065 §verification):
`opt3 / opt2 ≈ 2.4×` at default calibration.
`opt3 / opt2 ≈ 4.0×` at default calibration.
3. **Override path.** Topology yaml `pe_cost_model:` block changes the
per-PE dispatch cost; default is recovered when block is missing.
4. **`PeCpuOverheadCmd` bypass.** Manual `tl.cycles(n)` issues exactly `n`
@@ -244,13 +333,28 @@ calibration shows it matters.
(SPEC §0.1).
7. **Path parity.** Greenlet and replay paths produce identical
dispatch-cycle accounting for the same kernel.
8. **Composite size cap (D7).** A recipe that would emit `logical_bytes
> MAX_COMPOSITE_LOGICAL_BYTES` is segmented into N consecutive
`CompositeCmd`s; total dispatch = sum of segment dispatches; RW
ordering across segments preserved by strict FIFO.
9. **Sensitivity sweep.** At `R ∈ {0.25, 0.0625, 0.03125}` cycles/byte
(= 4 / 16 / 32 bytes/cycle), the conclusion *opt2 per-tile dispatch
< opt3 per-tile dispatch* must hold (ratio monotonically increases
as R decreases — FIXED dominates more).
## Migration
ADR-0064 Revision 2 lands as a single PR with:
- `logical_bytes` property on each `PeCommand` dataclass
- `logical_bytes` property on each `PeCommand` dataclass (type-aware
extra-field counting per D2)
- formula application in `pe_cpu.py` dispatch path
- `pe_cost_model:` override read at PE_CPU init
- `pe_cost_model:` override read at PE_CPU init (cycle-domain knobs +
`max_composite_logical_bytes`)
- **composite size cap (D7)** — TLContext-side segmentation logic with
`MAX_COMPOSITE_LOGICAL_BYTES` default 1024; not needed for any
existing bench (largest current composite is well under 200 bytes),
but the mechanism lands so it is in place when ADR-0065's
10-op decode-opt2 composite (~310 bytes) and larger recipes appear.
- one-time goldens regeneration
After this lands, ADR-0065 builds on top with no further goldens churn