ADR housekeeping: category prefixes, lifecycle folders, retroactive 0034-0037
Filename + lifecycle:
- ADR rename to ADR-NNNN-<cat>-title.md with 8 3-letter category prefixes
(dev / mem / lat / prog / algo / par / api / ver). Numbers stay immutable.
- ADR Lifecycle split into 3 folders, documented in CLAUDE.md Part 2:
docs/adr/ (Accepted), docs/adr-proposed/ (Proposed/Stub/Draft),
docs/adr-history/ (Superseded/Merged). Status field gains "Draft" for
retroactive docs pending verification.
Merges (one ADR per topic, no change-history annotations):
- ADR-0017 absorbs ADR-0019 (Cube NOC + per-PE HBM connectivity, 10 D-items)
- ADR-0014 absorbs ADR-0021 (PE pipeline execution model, 8 D-items incl.
TileToken self-routing and multi-op composite epilogue scope)
- ADR-0023 absorbs docs/ipcq-dma-codesign-hw.md as new "HW Realization
Notes (Informative)" section (D16-D23 + Open HW Questions). codesign-hw.md
deleted; ADR-0019/0021 moved to adr-history with one-line stub status
Retroactive documentation (G4 closures, code-verified):
- ADR-0037 forwarding component (TransitComponent: first-flit overhead,
serial worker, path-based routing, single impl/multiple names)
- ADR-0036 IO_CPU component (target_start_ns global barrier stamping,
per-cube fan-out, response aggregation)
- ADR-0035 M_CPU & M_CPU.DMA component (3 fan-out paths, DMA Resources,
target_start_ns passthrough)
- ADR-0034 HBM controller internal design (per-PC state, address-based
selection, flit-aware per-flit commit, async finalize, command-only
fallback path)
Content updates:
- ADR-0010 expanded to full CLI surface (run/probe/web), retitled
"Command Line Interface and Execution Semantics"
- ADR-0007 D2 rewritten to current state; ADR-0015 supersession notes pruned
- ADR-0005 wrapped in Decision header with D1-D5; ADR-0022 metadata
block replaced with standard Status header
- ADR-0024 trimmed to rank=SIP launcher essentials (D1-D4);
ADR-0027 cleaned of supersession history
- ADR-0033 D6 cleanup: address-based PC selection moved out of future-work
(now documented in ADR-0034 D3); related D1/D3 wording realigned
- Cross-references back-filled in 5 ADRs (G3 gaps closed)
Onboarding docs split:
- docs/onboarding/ created
- moved: hw-architecture-overview.md, latency-model.md, di-presentation.md,
ccl-author-guide{,.en}.md
- references updated in README, ADR-0023{,.en}, src/kernbench/ccl/__init__.py
Source / test / yaml: ADR-NNNN cross-references in docstrings and YAML
comments updated after the merges (ADR-0021->0014 D6, ADR-0019->0017 D8).
No behavior change.
Tooling:
- tools/verify_adr_lang_pairs.py + tests/test_verify_adr_lang_pairs.py
(ADR EN/KO pair invariant checker)
- .claude/commands/report.md tracked (/report slash command)
- .gitignore: allow .claude/commands/*.md while keeping settings files ignored
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,146 @@
|
||||
# ADR-0009: Kernel Execution Messaging and Completion Semantics
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
Kernel execution is initiated by the host and proceeds through
|
||||
device control components:
|
||||
|
||||
Host → IO_CPU → M_CPU → PE_CPU → schedulers → engines
|
||||
|
||||
Completion propagates in reverse order.
|
||||
|
||||
To keep benchmarks simple and topology-agnostic,
|
||||
kernel execution must be endpoint-driven with deterministic aggregation.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
### D1. Kernel launch is an endpoint request
|
||||
|
||||
A kernel launch is initiated by submitting a single KernelLaunch request
|
||||
to the IO_CPU endpoint.
|
||||
|
||||
The runtime API MUST:
|
||||
|
||||
- construct the kernel launch request,
|
||||
- submit it to IO_CPU,
|
||||
- await a single completion result.
|
||||
|
||||
The runtime API MUST NOT orchestrate internal fan-out.
|
||||
|
||||
---
|
||||
|
||||
### D2. Tensor arguments are passed by metadata
|
||||
|
||||
KernelLaunch requests MUST reference tensor arguments via:
|
||||
|
||||
- host-owned tensor handles, or
|
||||
- resolved device address maps derived from those handles.
|
||||
|
||||
Bulk tensor data MUST NOT be embedded in kernel launch messages.
|
||||
|
||||
---
|
||||
|
||||
### D3. Fan-out and aggregation are component responsibilities
|
||||
|
||||
- IO_CPU fans out work to M_CPUs.
|
||||
- M_CPU fans out work to PE_CPUs.
|
||||
- PE_CPU manages kernel execution and engine dispatch.
|
||||
|
||||
Completion semantics:
|
||||
|
||||
- M_CPU completes when all targeted PEs complete or a failure policy triggers.
|
||||
- IO_CPU completes when all targeted CUBEs complete or a failure policy triggers.
|
||||
|
||||
---
|
||||
|
||||
### D4. Completion and failure propagation
|
||||
|
||||
- All messages MUST carry correlation identifiers.
|
||||
- Completion and failure MUST propagate deterministically to the host.
|
||||
- The simulation engine provides futures/handles to observe completion.
|
||||
|
||||
---
|
||||
|
||||
### D5. Launch timing is endpoint-synchronized
|
||||
|
||||
All PEs targeted by a single kernel launch MUST begin executing the kernel
|
||||
body at the same simulated time, regardless of their dispatch path length
|
||||
from the launch entry point.
|
||||
|
||||
Rationale. The dispatch tree Host → IO_CPU → M_CPU → PE_CPU has variable
|
||||
latency at every level. PEs near their M_CPU receive the launch earlier
|
||||
than PEs farther away; cubes near an IO_CPU receive it earlier than cubes
|
||||
farther away. Without synchronization, each PE's kernel begins at a
|
||||
different `env.now`, making per-PE metrics such as `pe_exec_ns` a function
|
||||
of dispatch-path geometry rather than of the kernel's behavior —
|
||||
producing measurement artifacts in benchmarks that time kernel-internal
|
||||
waits (for example `tl.recv` on cross-cube or cross-SIP hops).
|
||||
|
||||
Mechanism.
|
||||
|
||||
- `KernelLaunchMsg` carries an optional `target_start_ns: float | None`.
|
||||
- **IO_CPU** is the canonical stamper. On fan-out to M_CPUs, it
|
||||
computes `target_start_ns = env.now + max_latency` where
|
||||
`max_latency` is the maximum, over every target (sip, cube, pe)
|
||||
tuple, of the **two-leg dispatch chain**:
|
||||
|
||||
```
|
||||
max_latency(sip, cube, pe) =
|
||||
compute_path_latency_ns(find_node_path(io_cpu, m_cpu(sip, cube)))
|
||||
+ compute_path_latency_ns(find_node_path(m_cpu(sip, cube), pe_cpu))
|
||||
- io_cpu.overhead_ns
|
||||
- m_cpu.overhead_ns
|
||||
```
|
||||
|
||||
This models the actual dispatch as **two sequential Transactions**
|
||||
(IO_CPU → M_CPU, then M_CPU → PE_CPU). Each leg's
|
||||
`compute_path_latency_ns` adds its endpoints' `overhead_ns`;
|
||||
`io_cpu.overhead_ns` is subtracted because IO_CPU has already
|
||||
paid it before this method runs, and `m_cpu.overhead_ns` is
|
||||
subtracted once because it appears as endpoint of leg1 *and*
|
||||
start of leg2 but is paid only once at run time. A single
|
||||
`find_node_path(io_cpu, pe_cpu)` walk is **not** equivalent —
|
||||
it can pick a graph path that bypasses M_CPU and silently
|
||||
under-shoots the prediction for far cubes, breaking the D5
|
||||
invariant.
|
||||
|
||||
The fanned-out sub-Transactions carry **`nbytes = 0`** for
|
||||
`KernelLaunchMsg` (control message only). Without this,
|
||||
large kernel-launch payloads would occupy fabric BW on the
|
||||
shared first hop and serialize the per-cube dispatch, pushing
|
||||
far M_CPUs past `target_start_ns` and re-introducing the
|
||||
late-arrival violation.
|
||||
- **M_CPU** passes an already-stamped `target_start_ns` through
|
||||
unchanged. Only when the value is absent (e.g. a direct
|
||||
launch-to-M_CPU unit test) does M_CPU compute a per-cube barrier
|
||||
`env.now + max(local command-path latency)`.
|
||||
- **PE_CPU** yields `env.timeout(target_start_ns - env.now)` at the top
|
||||
of `_execute_kernel`, before recording `pe_exec_start` and invoking
|
||||
the kernel body.
|
||||
- When `target_start_ns is None`, PE_CPU falls through to the legacy
|
||||
unsynchronized behavior — preserving backward compatibility.
|
||||
|
||||
IO_CPU-level stamping guarantees every PE across every targeted cube
|
||||
uses the same barrier sim-time, eliminating both the within-cube
|
||||
dispatch-offset artifact *and* the cross-cube offset artifact in
|
||||
multi-cube launches. Models a real-hardware timed-broadcast launch
|
||||
(latency-equalized dispatch tree).
|
||||
|
||||
The synchronization is internal to the engine / IO_CPU / M_CPU / PE_CPU
|
||||
control plane — runtime API and application kernels are unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Links
|
||||
|
||||
- SPEC R1, R2, R7, R8
|
||||
- ADR-0007 (Runtime API boundaries)
|
||||
- ADR-0008 (Tensor deployment)
|
||||
- ADR-0013 (Verification strategy — V2 fan-out tests)
|
||||
- ADR-0015 D4 (concrete fabric path for kernel launch)
|
||||
Reference in New Issue
Block a user