# ADR-0064: Structural CPU dispatch cost model (`logical_bytes` + FIXED + R) ## Status Accepted (Revision 2) > Supporting ADR for **ADR-0060** (AHBM GQA Fused Attention) and **ADR-0065** > (flat-ops composite + first stateful recipe). The hybrid decision there > (GEMMs via `tl.composite`, softmax merge in the kernel) wins by > **offloading tiling to PE_SCHEDULER so the CPU issues coarse descriptors > and runs ahead, keeping the engines saturated**. That win is currently > **invisible in the simulator** because per-op CPU issue cost is zero. > > Revision 2 replaces the **op-type calibration table** (the original > proposal) with a **structural formula** derived from each command's > `logical_bytes` — no per-op-type calibration needed; new op kinds are > covered automatically. ## Context ### What exists today - Every `tl.*` op calls `_emit_dispatch_overhead()` before emitting its command (`tl_context.py:196-212`), which emits `PeCpuOverheadCmd(cycles=dispatch_cycles)` **only if** `dispatch_cycles > 0`. The knob is **uniform** across op kinds and hardcoded to **0** in both live execution paths (`pe_cpu.py:101` greenlet, `:195` replay). - ⇒ issuing a command — *constructing the descriptor and pushing it to the scheduler queue* — currently costs **0 ns** on PE_CPU. - `PeCpuOverheadCmd` is consumed as `yield env.timeout(cmd.cycles)` on PE_CPU (`kernel_runner.py:131-132`). ### Why uniform-and-zero is wrong for the hybrid ADR-0060 §1's argument is that **one** composite descriptor offloads `N_tiles` worth of GEMM tiling, so the CPU issues `O(1)` coarse commands instead of `O(N_tiles × ops/tile)` fine ones. With issue cost = 0, the model cannot show: - that the primitive path may **fail to saturate** the engines when the CPU cannot push fast enough, nor - that a composite **costs more to construct** than a single primitive but far less than the many primitives it replaces. ### Why per-op-type calibration (Revision 1) was over-shaped The original proposal had a `cost_table[kind]` keyed by op kind (`composite`, `load`, `dot`, `math`, …). That required: - a value per kind (calibration cost ≥ |kinds|), - a new entry every time a new kind appears, - and yet the *ratio* it tried to capture — "composite ≫ primitive, but ≪ the primitives it replaces" — is structurally a function of **how many fields the command carries**, not of the op kind. A command's *byte footprint* is the natural proxy: a composite carrying N OpSpecs has ~N× the bytes of a primitive op with one OpSpec. The *fixed* part (queue head update, completion register, MMIO-class latency) is per-command. The two together compose: `FIXED + bytes × R`. ### What this model actually exposes The **primary** signal is **command-count reduction** through the per-command FIXED cost. The **byte** term is a secondary refinement that prevents pathologically-large composites from looking free. With realistic on-die queue bandwidth (16 B/cycle, D3), FIXED accounts for ≥85% of the dispatch cost differential between opt3 (≈10 cmds/tile) and opt2 (≈2 cmds/tile) for decode opt2. This framing also bounds the model: if a single composite were allowed to grow unboundedly large, the byte term alone would not stop the formula from rewarding ever-bigger fused commands beyond what real HW supports — hence the descriptor-size cap in **D7**. ## Decision ### D1. Structural dispatch cost formula Each PE command going to PE_SCHEDULER incurs PE_CPU dispatch cycles: ``` dispatch_cycles(cmd) = FIXED_PER_CMD + cmd.logical_bytes × R ``` where: - `FIXED_PER_CMD` (cycles per command) models queue-tail update, MMIO-class RTT, completion-event registration — fixed per command regardless of size. - `R` (cycles per byte) models the queue-write bandwidth — bytes of the command serialized into the scheduler queue. - `cmd.logical_bytes` (int) is each command's *HW-logical* byte size, computed from D2 below — not Python's `sys.getsizeof`. PE_CPU emits `PeCpuOverheadCmd(cycles=dispatch_cycles(cmd))` before dispatching, exactly as the existing hook (`tl_context.py:_emit_dispatch_ overhead`) — only the cycle value changes. ### D2. `logical_bytes` rule Each PE command dataclass exposes `logical_bytes: int` (property). The counting rule (HW-friendly, ignores Python overhead): | Field kind | Bytes | |---|---| | Command framing (cmd-type discriminator + completion id ref) | 4 | | Opcode (op kind enum) | 1 | | Enum (scope, etc.) | 1 | | `TensorHandle` reference (address only — shape/dtype assumed in descriptor table) | 8 | | Scalar (int/float) | 4 | | Tuple length marker | 1 | `CompositeCmd` recursively sums its `ops` and `rw_handles`: ```python @property def logical_bytes(self) -> int: return ( 4 # framing + 1 + sum(op.logical_bytes for op in self.ops) + 1 + 8 * len(self.rw_handles) ) ``` `OpSpec`: ```python @property def logical_bytes(self) -> int: return ( 1 + 1 # opcode + scope + 1 + 8 * len(self.operands) # named operand handles + (8 if self.out is not None else 0) # out handle + 1 + sum(_extra_bytes(v) for v in self.extra.values()) ) def _extra_bytes(v) -> int: """Type-aware byte count for OpSpec.extra values.""" if isinstance(v, bool): return 1 if isinstance(v, (int, float)): return 4 if isinstance(v, (tuple, list)): return 1 + 4 * len(v) # shape, axes, … if isinstance(v, str): return 1 # opcode-like tag return 4 # default scalar # Example types in extra: # m, k, n int → 4 each # reduce_axis int → 4 # shape=(64, 64) tuple → 1 + 8 = 9 # factor=1.0 float → 4 ``` (Identical rule for `DmaReadCmd`, `MathCmd`, etc. — one property per dataclass, ~3 lines each.) **Counting rule — per-op summation, no deduplication.** Each `TensorHandle` reference inside an `OpSpec` is counted independently, even when the same handle appears in multiple OpSpecs or in `rw_handles`. The `rw_handles` block is metadata for cross-composite hazard tracking (ADR-0065 D6.3) and is counted separately from operand references in `ops`. There is no deduplication. This matches HW reality: the descriptor encodes each operand slot as an independent address field, and the dispatcher tracks `rw_handles` as a distinct metadata block. Example: `OpSpec(kind="mul_bcast", operands={"src_a": O, "src_b": corr}, out=O)` counts handle `O` *twice* (once for `src_a`, once for `out`); if `O` is also in the enclosing CompositeCmd's `rw_handles`, it is counted a third time. ### D3. Defaults — anchored on a typical composite ≈ 43 ns Anchor: a **single-OpSpec composite for a DMA-staged GEMM path** — one `OpSpec(kind="gemm", ...)`; DMA stages are auto-inserted by PE_SCHEDULER from operand `space` (ADR-0065 D4) and **do not appear in `logical_bytes`** (the kernel does not issue them as separate cmds). Breakdown: ``` framing 4 ops tuple length 1 GEMM OpSpec 40 (opcode 1 + scope 1 + 1 + 2 handles 16 + out 8 + 1 + extra m/k/n 12) rw_handles tuple length 1 rw_handles content 8 (one RW handle for the output) ───────────────────────────── total ~54 bytes ``` Target dispatch = ~43 ns. On-die producer→consumer queue at 16 bytes/cycle (typical on-die descriptor queue width). ``` FIXED_PER_CMD = 40 cycles R = 0.0625 cycles/byte (= 16 bytes/cycle) ``` Verification: `40 + 54 × 0.0625 = 43.375 cycles ≈ 43 ns` ✓ Cycle→ns conversion uses the PE node's existing `clock_freq_ghz` attr (the same one used by PE_MATH `_compute_ns`). The cost-model knobs are **cycle-domain only** — they do not duplicate the clock setting. ### D4. Topology config override Defaults are baked into `pe_cpu.py`. Topology yaml may override under a `pe_cost_model:` section at the PE node attrs (cycle-domain knobs only; clock comes from the PE's existing `clock_freq_ghz`): ```yaml pe: attrs: clock_freq_ghz: 1.0 # existing, used for cycle→ns pe_cost_model: fixed_per_cmd_cycles: 40 byte_cycles_recip: 0.0625 # = 16 bytes/cycle max_composite_logical_bytes: 1024 # D7 — descriptor size cap ``` Missing keys fall back to defaults. The dispatch formula reads from `node.attrs["pe_cost_model"]` at PE_CPU init. ### D5. Scope — what does and does not pay | Path | Pays dispatch cost? | |---|---| | PE_CPU → PE_SCHEDULER for any `PeCommand` | **Yes** | | `PeCpuOverheadCmd` itself (already cycles-explicit) | **No** (formula bypass) | | Stages auto-generated by PE_SCHEDULER (DMA_READ/WRITE/FETCH/STORE) | **No** (PE_SCHEDULER-internal) | | Engine compute latency (DMA `drain_ns`, GEMM/MATH `_compute_ns`) | **No change** — stays on engines (SPEC §0.1) | This preserves the "latency on modelled components" invariant — dispatch cost is *additional* CPU-side time, not folded into engine times. ### D6. Configurable values; goldens regenerate Turning issue cost non-zero changes **every** bench's latency. Golden latencies are **regenerated once** when this ADR lands — same posture as ADR-0062 D3 lazy-load. After regeneration, the same calibration is in effect for ADR-0065 opt2 measurement. ### D7. Composite size cap (hard limit — validation error) Each `CompositeCmd`'s `logical_bytes` is capped at **`max_composite_logical_bytes`** (default **1024 bytes**, overridable per D4). A composite whose `logical_bytes` exceeds the cap is **rejected at emit time** by the host-side TLContext with a `ValueError` — there is **no automatic segmentation**. The kernel author must restructure the recipe (e.g., split it into multiple smaller composites explicitly) so each `CompositeCmd` fits within the descriptor capacity. **Why a cap is needed.** Real hardware imposes hard limits — descriptor queue entry size, scheduler parser buffer, command SRAM, firmware input. Without a cap, the `FIXED + bytes × R` model would reward arbitrarily large fused composites beyond what hardware accepts (e.g., fusing 100 primitive ops into one composite, paying one `FIXED`). **Why 1024 bytes specifically.** This is a **safe engineering limit**, not a measured HW number — intentionally chosen to be well above all currently known composites (decode opt2's `#2` at ~322 bytes is the largest in the kernbench codebase) while still representing a *finite* descriptor capacity that future recipes must respect. The number is overridable per topology (D4); when a real HW reference appears, the value should be recalibrated. The role of this default is to make the cap *exist as a discipline*, not to fit a specific HW. **Rationale for a hard error over auto-segmentation.** Auto-splitting an oversized composite (the original Revision 2 proposal) added emitter complexity — inter-segment ordering, shared `rw_handles`, completion chaining — to paper over what is, in practice, a kernel that asked for a descriptor larger than the hardware allows. Surfacing it as an explicit error keeps the emitter simple and makes the HW constraint visible to the kernel author, who is best placed to decide how to split the work. Decode opt2's `#2` composite (10 ops, ~322 bytes) sits comfortably inside the 1024 cap — no error for the GQA workload. ### D8. Single-op-cmd fast-path (per-cmd-type FIXED) Every **single-op** command — the DMA descriptors (`DmaReadCmd`, `DmaWriteCmd`) *and* the single-op compute dispatches (`GemmCmd`, `MathCmd`, `CopyCmd`) — carries a separate, lighter FIXED than the general control-path cost: **`fixed_per_single_op_cmd_cycles`** (default **8 cycles**) replaces `fixed_per_cmd_cycles` for all of them. **`CompositeCmd` is the only command that stays on the general control-path cost** `fixed_per_cmd_cycles` (= 40 cycles). Concretely, the D1 formula becomes ``` dispatch_cycles(cmd) = FIXED(type(cmd)) + cmd.logical_bytes × R FIXED(CompositeCmd) = fixed_per_cmd_cycles (= 40) FIXED(everything else, i.e. every single-op cmd: DmaReadCmd / DmaWriteCmd / GemmCmd / MathCmd / CopyCmd) = fixed_per_single_op_cmd_cycles (= 8) ``` **Why a separate, lighter FIXED for single-op cmds.** D1 treated every emitted command uniformly because that is the simplest defensible model. In practice an single-op command — whether a DMA descriptor or a single GEMM / elementwise / copy dispatch to an engine — is *much* cheaper than the generic control-path cost the 40-cycle FIXED was modeling. It is a single descriptor or instruction pushed to an engine queue, with no scheduler-side plan to build: - NVIDIA Hopper TMA: a single PTX instruction (`cp.async.bulk`) initiates a bulk async DMA — ~1 ISA cycle on the SM side, with the TMA engine doing fan-out and chunking itself. - NVIDIA Ampere `cp.async`: ~1–2 cycles per warp-level async copy. - A single MMA / tensor-core issue (`wgmma`, `mma.sync`): one instruction to the math pipe, not a control-path round trip. - AMD AQL packet write to a HSA queue: ~5–15 cycles. - Generic descriptor-ring-push designs (Synopsys/Xilinx-style DMA IP): ~5–20 cycles for the MMIO writes. - Tenstorrent tile descriptor emission, Habana Gaudi TPC descriptor RAM + start register: single-digit cycles per descriptor. The 40-cycle FIXED is justified for `CompositeCmd` specifically, because a composite is *not* a single dispatch: the scheduler must build a tile-feeder plan, track per-tile read/write hazards across its internal `DMA_READ → FETCH → GEMM → STORE → DMA_WRITE` stages, and wire a completion handle. An single-op `GemmCmd` is one engine issue with none of that machinery; charging it the same 40 cycles as a composite conflates a single instruction with a whole scheduled plan. **Why this matters for kernel evaluation.** A uniform 40-cycle FIXED inflates the dispatch overhead of user-orchestrated single-op-primitive kernels. A chunked-prefetching async GEMM at `K = N_K · TILE_K` emits, per K-tile, a `DmaReadCmd` (B-chunk load) **and** a `GemmCmd` (the per-chunk `tl.dot`) **and** a `MathCmd` (the running `tl.add` accumulate) — so the per-command FIXED dominates its dispatch budget. The single-op fast-path keeps the *structural* command-count signal that D1 was designed to capture (a recipe that emits `N` single-op commands *does* pay more than a recipe that emits one composite covering `N` tiles internally) without conflating it with a modeling overcharge that would be specific to a 40-cycle generic control path. **Why 8 cycles specifically.** 8 sits at the mid-range of the survey above (TMA ~1 to descriptor-ring ~15). It is intentionally a *modeling default*, not a measured HW number — the role of this default is to distinguish the single-op dispatch path from the composite control path in qualitative behaviour. The value is overridable per topology (D4). **Effect on composite numbers.** Composite tile-internal stages (`DMA_READ`, `FETCH`, `GEMM`, `STORE`, `DMA_WRITE` per HW tile) are emitted by the scheduler's tile-feeder loop, not by the host-side TLContext, so they do **not** go through `_charge_dispatch`. The single `CompositeCmd` itself still pays the 40-cycle control-path FIXED. D8 only changes the FIXED charged to user-side single-op primitives (`tl.load` / `tl.store` / `tl.dot` / `tl.add` / …). Composite measurements remain stable. ## Alternatives ### A1. Keep Revision 1's op-type calibration table Rejected: calibration cost scales with |kinds|, and the *ratio* the table tried to capture is structurally a function of cmd size. The structural formula reaches the same qualitative behaviour with two calibratable numbers instead of N. ### A2. Byte-only formula (no FIXED term) Rejected. With FIXED = 0, opt2 (Option Y per ADR-0065) does **not** win over opt3 — the total *bytes* dispatched per tile are similar (opt3 ≈ 232, opt2 ≈ 380); the win is entirely in *fewer per-cmd fixed costs*. A byte-only formula erases the very signal the model needs to expose. ### A3. Charge dispatch on PE_SCHEDULER instead of PE_CPU Rejected: the saturation question is *"can the CPU push descriptors fast enough to keep the engines busy?"* — that is a **PE_CPU** issue-bandwidth property. Charging on the scheduler would not model CPU back-pressure. ### A4. Model DMA program/setup time as a separate fixed per-descriptor cost Deferred: initially fold the descriptor-program cost into the **issuing op's** dispatch cost. Split it out to a PE_DMA fixed setup only if calibration shows it matters. ## Consequences ### Positive - Hybrid's CPU-offload / saturation win (ADR-0060 §1) becomes **measurable**, with a structurally honest model (no calibration table). - Adding new op kinds (e.g., ADR-0065's `softmax_merge` 8-step recipe) costs zero — they fit the same formula automatically. - More faithful to hardware (queue-head MMIO RTT + queue-write bandwidth). ### Negative - **All** bench goldens shift → one-time regeneration (D6); CI golden fixtures update. - Two calibration knobs (FIXED, R) need values; defaults are anchored on a documented assumption — treat absolute latencies as provisional until a reference exists; keep the **ratios** defensible. - Adds a small `logical_bytes` property to each PE command dataclass. ## Open review items 1. **Calibration source for FIXED and R.** Defaults from "typical composite ≈ 43 ns + on-die queue 16 bytes/cycle"; reasonable for an on-die descriptor queue. Revisit when a HW reference appears. 2. **Scheduler plan-gen cost vs large composites.** Stays 0 — D5 keeps PE_SCHEDULER's plan-generation outside the dispatch formula. The D7 cap (1024 bytes ≈ 30–35 OpSpecs) bounds the worst case, but a composite near the cap still costs the scheduler the same as a 1-op composite under the current zero-cost model. If a stress test (large-composite microbench) shows scheduler-bound behaviour, expose `overhead_ns` per-op-count. 3. **Where the override lives.** `pe_cost_model:` block under PE node attrs in topology yaml — keeps all knobs in one place, reviewable. Clock comes from the PE's existing `clock_freq_ghz` attr, not duplicated here. 4. **Path parity.** Both greenlet (`_execute_legacy` and `kernel_runner`) and replay paths must read the same cost model. Verify. 5. **Sensitivity of conclusions to R.** opt2 < opt3 must hold across a reasonable range of `R` (queue-bandwidth assumption). Sensitivity sweep is part of Test Requirements (#9). ## Test Requirements Tests are written against the **formula** (D1), not against specific numeric anchors, so they remain valid when calibration changes or when OpSpec/CompositeCmd fields are added. 1. **Formula preservation.** For any `CompositeCmd` `c`, PE_CPU's recorded dispatch overhead equals `FIXED_PER_CMD + c.logical_bytes × R` (within ±1 cycle for floor/round-off). Parametrized over several composites: a 1-OpSpec GEMM composite, a 5-op MATH chain, and a 10-op recipe composite. Default-calibration numbers (anchor ≈43 ns for the 1-OpSpec composite) are informative reference, not the test gate — the test gate is the formula equality. 2. **Qualitative ratio (robust).** opt3 per-tile PE_CPU dispatch strictly exceeds opt2 per-tile dispatch by at least a 2× margin — `opt3 > 2 × opt2`. The default-calibration model predicts ≈4×; the gate is the loose 2× bound so the test does not break when calibration is moved (e.g., when a HW reference replaces the default). Informative numbers — see ADR-0065 §verification and DDD-0065 §11 for the model expectation. 3. **Override path.** Topology yaml `pe_cost_model:` block changes the per-PE dispatch cost; default is recovered when block is missing. The formula identity from #1 must hold with the override values. 4. **`PeCpuOverheadCmd` bypass.** Manual `tl.cycles(n)` issues exactly `n` cycles, not `n + dispatch_cycles(...)`. 5. **No double-count.** PE_DMA `drain_ns`, PE_GEMM/MATH `_compute_ns` identical to pre-ADR values. 6. **Determinism.** Identical inputs → identical op_log + latency (SPEC §0.1). 7. **Path parity.** Greenlet and replay paths produce identical dispatch-cycle accounting for the same kernel. 8. **Composite size cap (D7).** A composite that would emit `logical_bytes > max_composite_logical_bytes` raises a `ValueError` at emit time (no segmentation). A composite within the cap emits normally. 9. **Sensitivity (qualitative).** At `R ∈ {0.25, 0.0625, 0.03125}` cycles/byte, `opt3 > opt2` at *all* three points. Direction (ratio monotonically increases as R decreases) is also asserted, but absolute ratio values are *not* required. ## Migration ADR-0064 Revision 2 lands as a single PR with: - `logical_bytes` property on each `PeCommand` dataclass (type-aware extra-field counting per D2) - formula application in `pe_cpu.py` dispatch path - `pe_cost_model:` override read at PE_CPU init (cycle-domain knobs + `max_composite_logical_bytes`) - **composite size cap (D7)** — TLContext-side hard cap with `max_composite_logical_bytes` default 1024: a composite over the cap raises a `ValueError` at emit time (no auto-segmentation). Not triggered by any existing bench (largest current composite is ~322 bytes), but the check lands so the descriptor-capacity limit is enforced as recipes grow. - one-time goldens regeneration After this lands, ADR-0065 builds on top with no further goldens churn in existing benches (ADR-0065 is a meaning-preserving refactor of `CompositeCmd` for the existing path; only opt2 is a new bench).