Files

T

mukesh c1a5cf3a2a ADR-0009 D5: chain-aware target_start_ns + zero-byte launch fanout

The single-walk predictor (find_node_path(io_cpu, pe_cpu) +
compute_path_latency_ns) under-shot actual dispatch latency for far
cubes -- the routing graph could pick a path bypassing M_CPU, and
non-zero-nbytes launch sub-txns serialized on shared first hops.
Far PEs arrived at _execute_kernel after target_start_ns, silently
skipped the barrier yield, and started pe_exec_start late. Their
reported pe_exec_ns under-counted by exactly the late_ns amount
(63 ns observed at h4 cube4.pe0 in the IPCQ test, up to 113 ns
worst case for cubes 9-11), producing the suspicious flat region
in the h4 IPCQ curve at 8192/10240 bytes.

Fix:
  - IO_CPU predictor uses the explicit two-leg chain
    (IO_CPU->M_CPU + M_CPU->PE_CPU - io.overhead - m.overhead), so
    every PE on every targeted cube has a barrier >= its real
    dispatch arrival.
  - Kernel-launch fanout sub-txns carry nbytes=0 (control-plane,
    not data-plane), removing the per-cube fanout serialization
    that pushed far M_CPUs past the predictor.
  - Legacy io_cpu mirror updated.

ADR-0009 D5 mechanism updated to specify the two-leg formula and
the nbytes=0 requirement. New tests/test_d5_barrier_invariant.py
asserts (a) no PE enters _execute_kernel after target_start_ns and
(b) every PE in a multi-cube launch has identical pe_exec_start --
both regressions silently pass on the existing
tests/test_kernel_launch_sync.py because that test only inspects
post-aggregation max(pe_exec_ns).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-27 15:12:58 -07:00

4.9 KiB

Raw Blame History

ADR-0009: Kernel Execution Messaging and Completion Semantics

Status

Accepted

Context

Kernel execution is initiated by the host and proceeds through device control components:

Host → IO_CPU → M_CPU → PE_CPU → schedulers → engines

Completion propagates in reverse order.

To keep benchmarks simple and topology-agnostic, kernel execution must be endpoint-driven with deterministic aggregation.

Decision

D1. Kernel launch is an endpoint request

A kernel launch is initiated by submitting a single KernelLaunch request to the IO_CPU endpoint.

The runtime API MUST:

construct the kernel launch request,
submit it to IO_CPU,
await a single completion result.

The runtime API MUST NOT orchestrate internal fan-out.

D2. Tensor arguments are passed by metadata

KernelLaunch requests MUST reference tensor arguments via:

host-owned tensor handles, or
resolved device address maps derived from those handles.

Bulk tensor data MUST NOT be embedded in kernel launch messages.

D3. Fan-out and aggregation are component responsibilities

IO_CPU fans out work to M_CPUs.
M_CPU fans out work to PE_CPUs.
PE_CPU manages kernel execution and engine dispatch.

Completion semantics:

M_CPU completes when all targeted PEs complete or a failure policy triggers.
IO_CPU completes when all targeted CUBEs complete or a failure policy triggers.

D4. Completion and failure propagation

All messages MUST carry correlation identifiers.
Completion and failure MUST propagate deterministically to the host.
The simulation engine provides futures/handles to observe completion.

D5. Launch timing is endpoint-synchronized

All PEs targeted by a single kernel launch MUST begin executing the kernel body at the same simulated time, regardless of their dispatch path length from the launch entry point.

Rationale. The dispatch tree Host → IO_CPU → M_CPU → PE_CPU has variable latency at every level. PEs near their M_CPU receive the launch earlier than PEs farther away; cubes near an IO_CPU receive it earlier than cubes farther away. Without synchronization, each PE's kernel begins at a different env.now, making per-PE metrics such as pe_exec_ns a function of dispatch-path geometry rather than of the kernel's behavior — producing measurement artifacts in benchmarks that time kernel-internal waits (for example tl.recv on cross-cube or cross-SIP hops).

Mechanism.

KernelLaunchMsg carries an optional target_start_ns: float | None.
IO_CPU is the canonical stamper. On fan-out to M_CPUs, it computes target_start_ns = env.now + max_latency where max_latency is the maximum, over every target (sip, cube, pe) tuple, of the two-leg dispatch chain:
```
max_latency(sip, cube, pe) =
    compute_path_latency_ns(find_node_path(io_cpu, m_cpu(sip, cube)))
  + compute_path_latency_ns(find_node_path(m_cpu(sip, cube), pe_cpu))
  - io_cpu.overhead_ns
  - m_cpu.overhead_ns
```
This models the actual dispatch as two sequential Transactions (IO_CPU → M_CPU, then M_CPU → PE_CPU). Each leg's compute_path_latency_ns adds its endpoints' overhead_ns; io_cpu.overhead_ns is subtracted because IO_CPU has already paid it before this method runs, and m_cpu.overhead_ns is subtracted once because it appears as endpoint of leg1 and start of leg2 but is paid only once at run time. A single find_node_path(io_cpu, pe_cpu) walk is not equivalent — it can pick a graph path that bypasses M_CPU and silently under-shoots the prediction for far cubes, breaking the D5 invariant.

The fanned-out sub-Transactions carry nbytes = 0 for KernelLaunchMsg (control message only). Without this, large kernel-launch payloads would occupy fabric BW on the shared first hop and serialize the per-cube dispatch, pushing far M_CPUs past target_start_ns and re-introducing the late-arrival violation.
M_CPU passes an already-stamped target_start_ns through unchanged. Only when the value is absent (e.g. a direct launch-to-M_CPU unit test) does M_CPU compute a per-cube barrier env.now + max(local command-path latency).
PE_CPU yields env.timeout(target_start_ns - env.now) at the top of _execute_kernel, before recording pe_exec_start and invoking the kernel body.
When target_start_ns is None, PE_CPU falls through to the legacy unsynchronized behavior — preserving backward compatibility.

IO_CPU-level stamping guarantees every PE across every targeted cube uses the same barrier sim-time, eliminating both the within-cube dispatch-offset artifact and the cross-cube offset artifact in multi-cube launches. Models a real-hardware timed-broadcast launch (latency-equalized dispatch tree).

The synchronization is internal to the engine / IO_CPU / M_CPU / PE_CPU control plane — runtime API and application kernels are unchanged.

4.9 KiB Raw Blame History