Add PE-level IPCQ collective infra + unified ccl_allreduce bench (ADR-0023)
Major changes:
PE-level IPCQ infrastructure:
- New PE_IPCQ component: ring-buffer control plane with 4-direction
neighbor mapping, head/tail pointers, backpressure (poll/sleep).
- PE_DMA extended with vc_comm channel for IPCQ outbound/inbound DMA,
including in-flight data snapshot (D9) and op_log recording at
outbound time for Phase 2 replay correctness.
- IpcqDmaToken piggyback model: data + metadata travel together,
atomic visibility at receiver (invariant I6).
- Credit return fast path: bottleneck-BW latency, no fabric vc_comm.
Phase 2 data execution (ADR-0020 integration):
- op_log extended: DmaWriteCmd now captures src_space/src_addr for
Phase 2 dma_write copy; ipcq_copy ops recorded at outbound time.
- DataExecutor replays dma_write + ipcq_copy in t_start order.
- Engine._flush_data_phase: incremental cursor-based replay after
each engine.wait() so host reads see post-Phase-2 data.
- KernelRunner Phase 1 writes disabled when op_log is active to
prevent stale data from corrupting the MemoryStore snapshot.
TLContext / kernel API:
- tl.send(dir, src=TensorHandle), tl.recv(dir, shape, dtype),
tl.recv_async, tl.wait(RecvFuture), copy_to_dst mode.
- TensorHandle operator overloading (add/sub/mul/div) via thread-local
active TLContext → MathCmd dispatch through PE_MATH.
- PE-local scratch allocator for math output handles.
- tl.load returns space="hbm" handles for correct Phase 2 addressing.
- Additional math functions: maximum, minimum, fma, clamp, softmax, cdiv.
Unified ccl_allreduce bench (PyTorch-compat host code):
- Single benches/ccl_allreduce.py with run() + worker(rank, ws, torch)
split matching real PyTorch DDP worker pattern.
- torch.distributed facade: init_process_group, get_world_size,
get_rank, get_backend, all_reduce, barrier — only real PyTorch names.
- AhbmCCLBackend: eager install_ipcq at init, all_reduce dispatches
kernel via tensor shard metadata (n_elem from shards[0].nbytes).
- world_size derived from topology spec (sips × cubes × pes_per_cube)
with optional algorithm-level override in ccl.yaml.
Tensor API (PyTorch-compat surface):
- Tensor.numpy(): gather-aware (all shards via VA-based addressing).
- Tensor.copy_(source): scatter from host tensor into sharded target.
- RuntimeContext.from_numpy(arr): host-side staging tensor.
- Tensor.data property fixed to use numpy() (was shards[0]-only).
Algorithm modules moved to src/kernbench/ccl/algorithms/:
- ring_allreduce, mesh_allreduce, tree_allreduce, hello_send.
- Each module exports kernel_args(world_size, n_elem) helper.
- ccl.yaml module paths updated to kernbench.ccl.algorithms.*.
Dead code removed:
- 7 per-variant bench files (ccl_allreduce_{tcm,hbm,sram}, etc.).
- _run_ccl_bench greenlet-per-SIP scheduler.
- benches.loader.is_ccl_bench + run_rank detection.
- benches/ccl/ directory.
Tests:
- New test_ccl_allreduce_matrix.py: 7 parametrized cases
(ring×3 buffers, ring 8/16, mesh 4, tree 7).
- New test_runtime_api_tensor.py: copy_/numpy/from_numpy unit tests.
- Existing tests updated for new import paths + world_size_override.
Docs:
- Korean ccl-author-guide.md and ADR-0023 paths updated.
- New English versions: ccl-author-guide.en.md, ADR-0023.en.md.
502 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -51,7 +51,42 @@ class DataExecutor:
|
||||
self._execute_math(op)
|
||||
|
||||
def _execute_memory(self, op: OpRecord) -> None:
|
||||
"""Memory ops are already handled by Phase 1 MemoryStore. Skip."""
|
||||
"""Replay memory copy ops in Phase 2 (ADR-0020 + ADR-0023).
|
||||
|
||||
- dma_read: no-op (handle already references HBM source).
|
||||
- dma_write: copy (src_space, src_addr) → (dst_space, dst_addr).
|
||||
Required because Phase 2 may have just produced new data at the
|
||||
source addr (e.g. PE_MATH scratch output).
|
||||
- ipcq_copy: copy across PEs — sender's source → receiver's slot.
|
||||
Required because the source may be a Phase 2 math output, and
|
||||
a downstream math op on the receiver reads from the slot.
|
||||
|
||||
Legacy entries without src/dst metadata are silently skipped.
|
||||
"""
|
||||
p = op.params
|
||||
if op.op_name == "dma_write" or op.op_name == "ipcq_copy":
|
||||
src_space = p.get("src_space")
|
||||
src_addr = p.get("src_addr")
|
||||
dst_space = p.get("dst_space")
|
||||
dst_addr = p.get("dst_addr")
|
||||
if (src_space is None or src_addr is None
|
||||
or dst_space is None or dst_addr is None):
|
||||
return
|
||||
# Prefer the Phase-1-time snapshot (captured at record_end /
|
||||
# outbound) so we don't read from a source that has since been
|
||||
# mutated by another op. Fall back to MemoryStore for sources
|
||||
# that had no Phase 1 data (e.g. math scratch outputs that
|
||||
# only get populated by Phase 2's math replay).
|
||||
data = p.get("snapshot")
|
||||
if data is None:
|
||||
try:
|
||||
data = self.store.read(
|
||||
src_space, src_addr,
|
||||
shape=p.get("shape"), dtype=p.get("dtype"),
|
||||
)
|
||||
except KeyError:
|
||||
return
|
||||
self.store.write(dst_space, dst_addr, data)
|
||||
|
||||
def _execute_gemm(self, op: OpRecord) -> None:
|
||||
"""Execute GEMM: out = a @ b."""
|
||||
@@ -77,18 +112,35 @@ class DataExecutor:
|
||||
"""Execute math op: unary, binary, or reduction."""
|
||||
p = op.params
|
||||
math_op = p.get("op", op.op_name)
|
||||
space = p.get("addr_space", "tcm")
|
||||
dtype = p.get("dtype", "f32")
|
||||
input_addrs = p.get("input_addrs", [])
|
||||
input_shapes = p.get("input_shapes", [])
|
||||
# Per-input space/dtype (ADR-0023 CCL accumulation): math ops can
|
||||
# mix inputs from different MemoryStore spaces (e.g. acc in "hbm",
|
||||
# recv slot in "tcm"). Fall back to legacy single-space mode when
|
||||
# the per-input lists are absent.
|
||||
input_spaces = p.get("input_spaces") or [p.get("addr_space", "tcm")] * len(input_addrs)
|
||||
input_dtypes = p.get("input_dtypes") or [dtype] * len(input_addrs)
|
||||
# Per-input data snapshots (ADR-0020 D6): captured at op_log
|
||||
# record time. Phase 1 has correct values for slot/HBM addrs at
|
||||
# that moment, which lets Phase 2 sidestep the slot-wraparound
|
||||
# races where a later round overwrites a slot before this op
|
||||
# runs in t_start order.
|
||||
snapshots = p.get("input_snapshots") or [None] * len(input_addrs)
|
||||
dst_space = p.get("dst_space", p.get("addr_space", "tcm"))
|
||||
|
||||
inputs = []
|
||||
for addr, shape in zip(input_addrs, input_shapes):
|
||||
inputs.append(self.store.read(space, addr, shape=shape, dtype=dtype))
|
||||
for addr, shape, space, idtype, snap in zip(
|
||||
input_addrs, input_shapes, input_spaces, input_dtypes, snapshots
|
||||
):
|
||||
if snap is not None:
|
||||
inputs.append(snap)
|
||||
else:
|
||||
inputs.append(self.store.read(space, addr, shape=shape, dtype=idtype))
|
||||
|
||||
result = _compute_math(math_op, inputs, p.get("axis"))
|
||||
if result is not None:
|
||||
self.store.write(space, p["dst_addr"], result)
|
||||
self.store.write(dst_space, p["dst_addr"], result)
|
||||
|
||||
def verify(self, expected: dict[tuple[str, int], np.ndarray],
|
||||
rtol: float = 1e-3, atol: float = 1e-3) -> dict[str, bool]:
|
||||
@@ -146,6 +198,14 @@ def _compute_math(op: str, inputs: list[np.ndarray], axis: int | None) -> np.nda
|
||||
if op == "min":
|
||||
return np.min(x, axis=axis, keepdims=True)
|
||||
|
||||
# Softmax (numerically stable)
|
||||
if op == "softmax":
|
||||
ax = axis if axis is not None else -1
|
||||
x_max = np.max(x, axis=ax, keepdims=True)
|
||||
e = np.exp(x - x_max)
|
||||
s = np.sum(e, axis=ax, keepdims=True)
|
||||
return e / s
|
||||
|
||||
# Binary
|
||||
if len(inputs) >= 2:
|
||||
y = inputs[1]
|
||||
@@ -157,9 +217,18 @@ def _compute_math(op: str, inputs: list[np.ndarray], axis: int | None) -> np.nda
|
||||
return x * y
|
||||
if op == "div":
|
||||
return x / y
|
||||
if op == "maximum":
|
||||
return np.maximum(x, y)
|
||||
if op == "minimum":
|
||||
return np.minimum(x, y)
|
||||
|
||||
# Ternary
|
||||
if op == "where" and len(inputs) >= 3:
|
||||
return np.where(inputs[0], inputs[1], inputs[2])
|
||||
if len(inputs) >= 3:
|
||||
if op == "where":
|
||||
return np.where(inputs[0], inputs[1], inputs[2])
|
||||
if op == "fma":
|
||||
return inputs[0] * inputs[1] + inputs[2]
|
||||
if op == "clamp":
|
||||
return np.minimum(np.maximum(inputs[0], inputs[1]), inputs[2])
|
||||
|
||||
return None
|
||||
|
||||
@@ -51,8 +51,12 @@ class GraphEngine:
|
||||
if enable_data:
|
||||
from kernbench.sim_engine.memory_store import MemoryStore
|
||||
from kernbench.sim_engine.op_log import OpLogger
|
||||
self._op_logger = OpLogger()
|
||||
self._memory_store = MemoryStore()
|
||||
self._op_logger = OpLogger(memory_store=self._memory_store)
|
||||
# Cursor for incremental Phase 2 replay (ADR-0020 D6).
|
||||
# SimPy env.now is monotonic so newly logged records always sort
|
||||
# to the tail; the cursor remains valid across waits.
|
||||
self._data_cursor = 0
|
||||
|
||||
ctx = ComponentContext(
|
||||
router=self._router,
|
||||
@@ -147,11 +151,60 @@ class GraphEngine:
|
||||
self._env.process(self._process(str(handle), request, event))
|
||||
return handle
|
||||
|
||||
def _flush_data_phase(self) -> None:
|
||||
"""Replay newly recorded op_log entries through DataExecutor.
|
||||
|
||||
ADR-0020 D6 Phase 2: when data tracking is enabled, run DataExecutor
|
||||
on records added since the last flush so that callers reading
|
||||
MemoryStore between launches observe correct (compute-replayed)
|
||||
tensor data.
|
||||
|
||||
Cursor-based incremental replay is necessary because Phase 2 is
|
||||
NOT idempotent across full re-runs: a math op writes a TCM scratch
|
||||
addr, a later dma_write copies that scratch into HBM[X], and an
|
||||
even-later math op may then read HBM[X]. Re-running everything
|
||||
from scratch would let the second pass's first math op read the
|
||||
already-overwritten HBM[X] instead of the original input.
|
||||
"""
|
||||
if self._op_logger is None or self._memory_store is None:
|
||||
return
|
||||
records = self._op_logger.records # sorted by t_start (stable)
|
||||
if self._data_cursor >= len(records):
|
||||
return
|
||||
new_records = records[self._data_cursor:]
|
||||
from kernbench.sim_engine.data_executor import DataExecutor
|
||||
DataExecutor(new_records, self._memory_store).run()
|
||||
self._data_cursor = len(records)
|
||||
|
||||
def wait(self, handle: RequestHandle) -> None:
|
||||
key = str(handle)
|
||||
event = self._events[key]
|
||||
if not event.triggered:
|
||||
self._env.run(until=event)
|
||||
try:
|
||||
self._env.run(until=event)
|
||||
except (simpy.core.EmptySchedule, RuntimeError) as exc:
|
||||
# SimPy raises EmptySchedule directly OR (in newer simpy)
|
||||
# wraps it as a RuntimeError("No scheduled events left ...").
|
||||
# Either case while our event is still pending → IPCQ deadlock.
|
||||
msg = str(exc)
|
||||
is_deadlock = (
|
||||
isinstance(exc, simpy.core.EmptySchedule)
|
||||
or "No scheduled events left" in msg
|
||||
)
|
||||
if not is_deadlock:
|
||||
raise
|
||||
from kernbench.ccl.diagnostics import IpcqDeadlock, pointer_dump
|
||||
dump = pointer_dump(self)
|
||||
if dump.strip():
|
||||
raise IpcqDeadlock(
|
||||
"IPCQ deadlock: simulation schedule empty while "
|
||||
f"request {handle!r} is still pending.\n"
|
||||
f"Pointer state:\n{dump}"
|
||||
) from None
|
||||
raise
|
||||
# ADR-0020: replay newly logged ops so the caller observes
|
||||
# post-Phase-2 tensor state from MemoryStore.
|
||||
self._flush_data_phase()
|
||||
|
||||
def get_completion(self, handle: RequestHandle) -> tuple[Completion, Trace | None]:
|
||||
return self._results[str(handle)]
|
||||
|
||||
@@ -29,9 +29,13 @@ class OpLogger:
|
||||
Records are maintained in t_start stable ordering (insertion order).
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
def __init__(self, memory_store: Any | None = None) -> None:
|
||||
self._records: list[OpRecord] = []
|
||||
self._pending: dict[int, dict[str, Any]] = {} # msg id → partial record
|
||||
# Optional MemoryStore reference. When set, math op records capture
|
||||
# input data snapshots at record_end time so Phase 2 replay does
|
||||
# not depend on slot/scratch addrs surviving until math runs.
|
||||
self._memory_store = memory_store
|
||||
|
||||
@property
|
||||
def records(self) -> list[OpRecord]:
|
||||
@@ -53,6 +57,38 @@ class OpLogger:
|
||||
if pending is None:
|
||||
return
|
||||
op_kind, op_name, params = _extract_op_info(msg)
|
||||
# Snapshot data at record time so Phase 2 replay sidesteps
|
||||
# downstream mutations of source addrs (e.g. a tl.store that
|
||||
# overwrites HBM after a load handle was sent, or a slot that
|
||||
# gets reused on the next ring round).
|
||||
if self._memory_store is not None:
|
||||
if op_kind == "math":
|
||||
snaps: list[Any] = []
|
||||
for addr, shape, space, idtype in zip(
|
||||
params.get("input_addrs", []),
|
||||
params.get("input_shapes", []),
|
||||
params.get("input_spaces", []),
|
||||
params.get("input_dtypes", []),
|
||||
):
|
||||
try:
|
||||
arr = self._memory_store.read(
|
||||
space, addr, shape=shape, dtype=idtype,
|
||||
)
|
||||
snaps.append(arr.copy() if hasattr(arr, "copy") else arr)
|
||||
except Exception:
|
||||
snaps.append(None)
|
||||
params["input_snapshots"] = snaps
|
||||
elif op_name == "dma_write":
|
||||
try:
|
||||
arr = self._memory_store.read(
|
||||
params["src_space"], params["src_addr"],
|
||||
shape=params.get("shape"), dtype=params.get("dtype"),
|
||||
)
|
||||
params["snapshot"] = (
|
||||
arr.copy() if hasattr(arr, "copy") else arr
|
||||
)
|
||||
except Exception:
|
||||
params["snapshot"] = None
|
||||
self._records.append(OpRecord(
|
||||
t_start=pending["t_start"],
|
||||
t_end=t,
|
||||
@@ -62,6 +98,45 @@ class OpLogger:
|
||||
params=params,
|
||||
))
|
||||
|
||||
def record_copy(
|
||||
self, t_start: float, t_end: float, component_id: str,
|
||||
src_space: str, src_addr: int,
|
||||
dst_space: str, dst_addr: int,
|
||||
shape: tuple[int, ...], dtype: str, nbytes: int,
|
||||
) -> None:
|
||||
"""Record a memory copy op for Phase 2 replay (ADR-0023 + ADR-0020).
|
||||
|
||||
Used by PE_DMA at outbound (sender) time: the snapshot captures
|
||||
the source data at the moment the send was issued, so Phase 2
|
||||
replay does not see later mutations of the source addr (e.g. a
|
||||
tl.store that runs after the recv at the sender).
|
||||
|
||||
For sources whose data is not yet materialized in Phase 1 (math
|
||||
scratch outputs), the snapshot is None and Phase 2 falls back to
|
||||
reading from MemoryStore — by which point the corresponding math
|
||||
op has been replayed and the scratch addr is populated.
|
||||
"""
|
||||
snap = None
|
||||
if self._memory_store is not None:
|
||||
try:
|
||||
arr = self._memory_store.read(
|
||||
src_space, src_addr, shape=shape, dtype=dtype,
|
||||
)
|
||||
snap = arr.copy() if hasattr(arr, "copy") else arr
|
||||
except Exception:
|
||||
snap = None
|
||||
self._records.append(OpRecord(
|
||||
t_start=t_start, t_end=t_end,
|
||||
component_id=component_id,
|
||||
op_kind="memory", op_name="ipcq_copy",
|
||||
params={
|
||||
"src_space": src_space, "src_addr": src_addr,
|
||||
"dst_space": dst_space, "dst_addr": dst_addr,
|
||||
"shape": shape, "dtype": dtype, "nbytes": nbytes,
|
||||
"snapshot": snap,
|
||||
},
|
||||
))
|
||||
|
||||
|
||||
def _extract_op_info(msg: Any) -> tuple[str, str, dict[str, Any]]:
|
||||
"""Extract op_kind, op_name, params from a data_op message."""
|
||||
@@ -76,6 +151,11 @@ def _extract_op_info(msg: Any) -> tuple[str, str, dict[str, Any]]:
|
||||
}
|
||||
if isinstance(msg, DmaWriteCmd):
|
||||
return "memory", "dma_write", {
|
||||
"src_space": getattr(msg.handle, "space", "tcm"),
|
||||
"src_addr": msg.handle.addr,
|
||||
"shape": msg.handle.shape,
|
||||
"dtype": msg.handle.dtype,
|
||||
"dst_space": "hbm",
|
||||
"dst_addr": msg.dst_addr,
|
||||
"nbytes": msg.nbytes,
|
||||
"handle_id": msg.handle.id,
|
||||
@@ -96,7 +176,10 @@ def _extract_op_info(msg: Any) -> tuple[str, str, dict[str, Any]]:
|
||||
return "math", msg.op, {
|
||||
"input_addrs": [h.addr for h in msg.inputs],
|
||||
"input_shapes": [h.shape for h in msg.inputs],
|
||||
"input_spaces": [getattr(h, "space", "tcm") for h in msg.inputs],
|
||||
"input_dtypes": [h.dtype for h in msg.inputs],
|
||||
"dst_addr": msg.out.addr,
|
||||
"dst_space": getattr(msg.out, "space", "tcm"),
|
||||
"shape_out": msg.out.shape,
|
||||
"dtype": msg.out.dtype,
|
||||
"axis": msg.axis,
|
||||
|
||||
Reference in New Issue
Block a user