mukesh
c1a5cf3a2a
The single-walk predictor (find_node_path(io_cpu, pe_cpu) +
compute_path_latency_ns) under-shot actual dispatch latency for far
cubes -- the routing graph could pick a path bypassing M_CPU, and
non-zero-nbytes launch sub-txns serialized on shared first hops.
Far PEs arrived at _execute_kernel after target_start_ns, silently
skipped the barrier yield, and started pe_exec_start late. Their
reported pe_exec_ns under-counted by exactly the late_ns amount
(63 ns observed at h4 cube4.pe0 in the IPCQ test, up to 113 ns
worst case for cubes 9-11), producing the suspicious flat region
in the h4 IPCQ curve at 8192/10240 bytes.
Fix:
- IO_CPU predictor uses the explicit two-leg chain
(IO_CPU->M_CPU + M_CPU->PE_CPU - io.overhead - m.overhead), so
every PE on every targeted cube has a barrier >= its real
dispatch arrival.
- Kernel-launch fanout sub-txns carry nbytes=0 (control-plane,
not data-plane), removing the per-cube fanout serialization
that pushed far M_CPUs past the predictor.
- Legacy io_cpu mirror updated.
ADR-0009 D5 mechanism updated to specify the two-leg formula and
the nbytes=0 requirement. New tests/test_d5_barrier_invariant.py
asserts (a) no PE enters _execute_kernel after target_start_ns and
(b) every PE in a multi-cube launch has identical pe_exec_start --
both regressions silently pass on the existing
tests/test_kernel_launch_sync.py because that test only inspects
post-aggregation max(pe_exec_ns).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
145 lines
4.9 KiB
Markdown
145 lines
4.9 KiB
Markdown
# ADR-0009: Kernel Execution Messaging and Completion Semantics
|
|
|
|
## Status
|
|
|
|
Accepted
|
|
|
|
## Context
|
|
|
|
Kernel execution is initiated by the host and proceeds through
|
|
device control components:
|
|
|
|
Host → IO_CPU → M_CPU → PE_CPU → schedulers → engines
|
|
|
|
Completion propagates in reverse order.
|
|
|
|
To keep benchmarks simple and topology-agnostic,
|
|
kernel execution must be endpoint-driven with deterministic aggregation.
|
|
|
|
---
|
|
|
|
## Decision
|
|
|
|
### D1. Kernel launch is an endpoint request
|
|
|
|
A kernel launch is initiated by submitting a single KernelLaunch request
|
|
to the IO_CPU endpoint.
|
|
|
|
The runtime API MUST:
|
|
|
|
- construct the kernel launch request,
|
|
- submit it to IO_CPU,
|
|
- await a single completion result.
|
|
|
|
The runtime API MUST NOT orchestrate internal fan-out.
|
|
|
|
---
|
|
|
|
### D2. Tensor arguments are passed by metadata
|
|
|
|
KernelLaunch requests MUST reference tensor arguments via:
|
|
|
|
- host-owned tensor handles, or
|
|
- resolved device address maps derived from those handles.
|
|
|
|
Bulk tensor data MUST NOT be embedded in kernel launch messages.
|
|
|
|
---
|
|
|
|
### D3. Fan-out and aggregation are component responsibilities
|
|
|
|
- IO_CPU fans out work to M_CPUs.
|
|
- M_CPU fans out work to PE_CPUs.
|
|
- PE_CPU manages kernel execution and engine dispatch.
|
|
|
|
Completion semantics:
|
|
|
|
- M_CPU completes when all targeted PEs complete or a failure policy triggers.
|
|
- IO_CPU completes when all targeted CUBEs complete or a failure policy triggers.
|
|
|
|
---
|
|
|
|
### D4. Completion and failure propagation
|
|
|
|
- All messages MUST carry correlation identifiers.
|
|
- Completion and failure MUST propagate deterministically to the host.
|
|
- The simulation engine provides futures/handles to observe completion.
|
|
|
|
---
|
|
|
|
### D5. Launch timing is endpoint-synchronized
|
|
|
|
All PEs targeted by a single kernel launch MUST begin executing the kernel
|
|
body at the same simulated time, regardless of their dispatch path length
|
|
from the launch entry point.
|
|
|
|
Rationale. The dispatch tree Host → IO_CPU → M_CPU → PE_CPU has variable
|
|
latency at every level. PEs near their M_CPU receive the launch earlier
|
|
than PEs farther away; cubes near an IO_CPU receive it earlier than cubes
|
|
farther away. Without synchronization, each PE's kernel begins at a
|
|
different `env.now`, making per-PE metrics such as `pe_exec_ns` a function
|
|
of dispatch-path geometry rather than of the kernel's behavior —
|
|
producing measurement artifacts in benchmarks that time kernel-internal
|
|
waits (for example `tl.recv` on cross-cube or cross-SIP hops).
|
|
|
|
Mechanism.
|
|
|
|
- `KernelLaunchMsg` carries an optional `target_start_ns: float | None`.
|
|
- **IO_CPU** is the canonical stamper. On fan-out to M_CPUs, it
|
|
computes `target_start_ns = env.now + max_latency` where
|
|
`max_latency` is the maximum, over every target (sip, cube, pe)
|
|
tuple, of the **two-leg dispatch chain**:
|
|
|
|
```
|
|
max_latency(sip, cube, pe) =
|
|
compute_path_latency_ns(find_node_path(io_cpu, m_cpu(sip, cube)))
|
|
+ compute_path_latency_ns(find_node_path(m_cpu(sip, cube), pe_cpu))
|
|
- io_cpu.overhead_ns
|
|
- m_cpu.overhead_ns
|
|
```
|
|
|
|
This models the actual dispatch as **two sequential Transactions**
|
|
(IO_CPU → M_CPU, then M_CPU → PE_CPU). Each leg's
|
|
`compute_path_latency_ns` adds its endpoints' `overhead_ns`;
|
|
`io_cpu.overhead_ns` is subtracted because IO_CPU has already
|
|
paid it before this method runs, and `m_cpu.overhead_ns` is
|
|
subtracted once because it appears as endpoint of leg1 *and*
|
|
start of leg2 but is paid only once at run time. A single
|
|
`find_node_path(io_cpu, pe_cpu)` walk is **not** equivalent —
|
|
it can pick a graph path that bypasses M_CPU and silently
|
|
under-shoots the prediction for far cubes, breaking the D5
|
|
invariant.
|
|
|
|
The fanned-out sub-Transactions carry **`nbytes = 0`** for
|
|
`KernelLaunchMsg` (control message only). Without this,
|
|
large kernel-launch payloads would occupy fabric BW on the
|
|
shared first hop and serialize the per-cube dispatch, pushing
|
|
far M_CPUs past `target_start_ns` and re-introducing the
|
|
late-arrival violation.
|
|
- **M_CPU** passes an already-stamped `target_start_ns` through
|
|
unchanged. Only when the value is absent (e.g. a direct
|
|
launch-to-M_CPU unit test) does M_CPU compute a per-cube barrier
|
|
`env.now + max(local command-path latency)`.
|
|
- **PE_CPU** yields `env.timeout(target_start_ns - env.now)` at the top
|
|
of `_execute_kernel`, before recording `pe_exec_start` and invoking
|
|
the kernel body.
|
|
- When `target_start_ns is None`, PE_CPU falls through to the legacy
|
|
unsynchronized behavior — preserving backward compatibility.
|
|
|
|
IO_CPU-level stamping guarantees every PE across every targeted cube
|
|
uses the same barrier sim-time, eliminating both the within-cube
|
|
dispatch-offset artifact *and* the cross-cube offset artifact in
|
|
multi-cube launches. Models a real-hardware timed-broadcast launch
|
|
(latency-equalized dispatch tree).
|
|
|
|
The synchronization is internal to the engine / IO_CPU / M_CPU / PE_CPU
|
|
control plane — runtime API and application kernels are unchanged.
|
|
|
|
---
|
|
|
|
## Links
|
|
|
|
- SPEC R1, R2, R7, R8
|
|
- ADR-0007 (Runtime API boundaries)
|
|
- ADR-0008 (Tensor deployment)
|