Add PE-level IPCQ collective infra + unified ccl_allreduce bench (ADR-0023)
Major changes:
PE-level IPCQ infrastructure:
- New PE_IPCQ component: ring-buffer control plane with 4-direction
neighbor mapping, head/tail pointers, backpressure (poll/sleep).
- PE_DMA extended with vc_comm channel for IPCQ outbound/inbound DMA,
including in-flight data snapshot (D9) and op_log recording at
outbound time for Phase 2 replay correctness.
- IpcqDmaToken piggyback model: data + metadata travel together,
atomic visibility at receiver (invariant I6).
- Credit return fast path: bottleneck-BW latency, no fabric vc_comm.
Phase 2 data execution (ADR-0020 integration):
- op_log extended: DmaWriteCmd now captures src_space/src_addr for
Phase 2 dma_write copy; ipcq_copy ops recorded at outbound time.
- DataExecutor replays dma_write + ipcq_copy in t_start order.
- Engine._flush_data_phase: incremental cursor-based replay after
each engine.wait() so host reads see post-Phase-2 data.
- KernelRunner Phase 1 writes disabled when op_log is active to
prevent stale data from corrupting the MemoryStore snapshot.
TLContext / kernel API:
- tl.send(dir, src=TensorHandle), tl.recv(dir, shape, dtype),
tl.recv_async, tl.wait(RecvFuture), copy_to_dst mode.
- TensorHandle operator overloading (add/sub/mul/div) via thread-local
active TLContext → MathCmd dispatch through PE_MATH.
- PE-local scratch allocator for math output handles.
- tl.load returns space="hbm" handles for correct Phase 2 addressing.
- Additional math functions: maximum, minimum, fma, clamp, softmax, cdiv.
Unified ccl_allreduce bench (PyTorch-compat host code):
- Single benches/ccl_allreduce.py with run() + worker(rank, ws, torch)
split matching real PyTorch DDP worker pattern.
- torch.distributed facade: init_process_group, get_world_size,
get_rank, get_backend, all_reduce, barrier — only real PyTorch names.
- AhbmCCLBackend: eager install_ipcq at init, all_reduce dispatches
kernel via tensor shard metadata (n_elem from shards[0].nbytes).
- world_size derived from topology spec (sips × cubes × pes_per_cube)
with optional algorithm-level override in ccl.yaml.
Tensor API (PyTorch-compat surface):
- Tensor.numpy(): gather-aware (all shards via VA-based addressing).
- Tensor.copy_(source): scatter from host tensor into sharded target.
- RuntimeContext.from_numpy(arr): host-side staging tensor.
- Tensor.data property fixed to use numpy() (was shards[0]-only).
Algorithm modules moved to src/kernbench/ccl/algorithms/:
- ring_allreduce, mesh_allreduce, tree_allreduce, hello_send.
- Each module exports kernel_args(world_size, n_elem) helper.
- ccl.yaml module paths updated to kernbench.ccl.algorithms.*.
Dead code removed:
- 7 per-variant bench files (ccl_allreduce_{tcm,hbm,sram}, etc.).
- _run_ccl_bench greenlet-per-SIP scheduler.
- benches.loader.is_ccl_bench + run_rank detection.
- benches/ccl/ directory.
Tests:
- New test_ccl_allreduce_matrix.py: 7 parametrized cases
(ring×3 buffers, ring 8/16, mesh 4, tree 7).
- New test_runtime_api_tensor.py: copy_/numpy/from_numpy unit tests.
- Existing tests updated for new import paths + world_size_override.
Docs:
- Korean ccl-author-guide.md and ADR-0023 paths updated.
- New English versions: ccl-author-guide.en.md, ADR-0023.en.md.
502 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -29,11 +29,10 @@ def run_bench(
|
||||
correlation_id: str = "bench0",
|
||||
completion_policy: CompletionPolicy = CompletionPolicy.LAST_SUBMITTED,
|
||||
) -> BenchResult:
|
||||
"""
|
||||
Minimal bench runner.
|
||||
"""Minimal bench runner.
|
||||
|
||||
- topology: compiled topology object (opaque to runtime here)
|
||||
- bench_fn: callable that receives RuntimeContext and submits requests
|
||||
- bench_fn: callable ``run(torch)`` receiving a RuntimeContext
|
||||
- device: DeviceSelector ("all" or "sip:<N>")
|
||||
- engine_factory: builds sim_engine for given topology & device
|
||||
- completion_policy: how to determine overall completion/result
|
||||
@@ -48,7 +47,6 @@ def run_bench(
|
||||
)
|
||||
|
||||
bench_fn(ctx)
|
||||
|
||||
ctx.wait_all()
|
||||
|
||||
collected_traces = ctx._traces or None
|
||||
|
||||
@@ -9,6 +9,39 @@ from kernbench.common.types import Completion, RequestHandle, SimEngine
|
||||
from .types import DeviceSelector
|
||||
|
||||
|
||||
def _world_size_from_spec(spec: dict | None) -> int:
|
||||
"""Derive world_size from topology spec: sips × cubes × pes_per_cube."""
|
||||
spec = spec or {}
|
||||
sips = int(spec.get("system", {}).get("sips", {}).get("count", 1))
|
||||
cm = spec.get("sip", {}).get("cube_mesh", {})
|
||||
cubes_per_sip = int(cm.get("w", 1)) * int(cm.get("h", 1))
|
||||
pl = spec.get("cube", {}).get("pe_layout", {})
|
||||
corners = pl.get("corners", [])
|
||||
pe_per_corner = int(pl.get("pe_per_corner", 1))
|
||||
pes_per_cube = pe_per_corner * max(len(corners), 1)
|
||||
return sips * cubes_per_sip * pes_per_cube
|
||||
|
||||
|
||||
def _numpy_to_dtype_str(np_dtype) -> str:
|
||||
"""Map numpy dtype → kernbench dtype string used by Tensor."""
|
||||
import numpy as np
|
||||
|
||||
kind_map = {
|
||||
np.float16: "f16",
|
||||
np.float32: "f32",
|
||||
np.int8: "i8",
|
||||
np.int16: "i16",
|
||||
np.int32: "i32",
|
||||
np.uint8: "u8",
|
||||
np.uint16: "u16",
|
||||
np.uint32: "u32",
|
||||
}
|
||||
for np_type, s in kind_map.items():
|
||||
if np.dtype(np_dtype) == np.dtype(np_type):
|
||||
return s
|
||||
raise ValueError(f"unsupported numpy dtype: {np_dtype!r}")
|
||||
|
||||
|
||||
@dataclass
|
||||
class RuntimeContext:
|
||||
engine: SimEngine
|
||||
@@ -23,6 +56,66 @@ class RuntimeContext:
|
||||
_tensor_counter: int = field(default=0, init=False)
|
||||
_traces: list[dict] = field(default_factory=list, init=False)
|
||||
_tensors: list[Any] = field(default_factory=list, init=False)
|
||||
distributed: Any = field(default=None, init=False) # DistributedContext for CCL benches
|
||||
_ipcq_plan: dict = field(default_factory=dict, init=False) # ADR-0023 install plan
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Eagerly attach a DistributedContext so bench code can do
|
||||
# ``dist = torch.distributed`` + ``dist.init_process_group(...)``
|
||||
# without needing a separate launcher to install it.
|
||||
from kernbench.runtime_api.distributed import DistributedContext
|
||||
dc = DistributedContext()
|
||||
dc._ctx_ref = self # back-reference for AhbmCCLBackend to reach ctx.launch etc.
|
||||
self.distributed = dc
|
||||
|
||||
def install_ipcq(
|
||||
self,
|
||||
algorithm: str | None = None,
|
||||
ccl_yaml: str | None = None,
|
||||
world_size_override: int | None = None,
|
||||
rank_to_pe: list[tuple[int, int, int]] | None = None,
|
||||
) -> dict:
|
||||
"""Install IPCQ neighbor tables on all participating PEs (ADR-0023 D10).
|
||||
|
||||
Loads ``ccl.yaml`` (or the path provided), resolves the chosen
|
||||
algorithm (or ``defaults.algorithm`` if None), and pushes per-PE
|
||||
IpcqInitMsg into every PE_IPCQ component via the engine.
|
||||
|
||||
Args:
|
||||
algorithm: name of the algorithm in ccl.yaml (or use defaults).
|
||||
ccl_yaml: optional path to ccl.yaml.
|
||||
world_size_override: if set, replace the algorithm's world_size.
|
||||
|
||||
Returns the install plan dict (rank → (sip,cube,pe), neighbor table).
|
||||
"""
|
||||
import importlib
|
||||
from kernbench.ccl.install import (
|
||||
install_ipcq as _install,
|
||||
load_ccl_config,
|
||||
resolve_algorithm_config,
|
||||
)
|
||||
|
||||
cfg = load_ccl_config(ccl_yaml)
|
||||
merged = resolve_algorithm_config(cfg, algorithm)
|
||||
if world_size_override is not None:
|
||||
merged["world_size"] = world_size_override
|
||||
elif "world_size" not in merged:
|
||||
# Derive from topology.yaml when neither the algorithm entry
|
||||
# nor ``defaults`` carries ``world_size`` (matches pytorch DDP
|
||||
# where env vars determine ranks, not the ccl config file).
|
||||
merged["world_size"] = _world_size_from_spec(self.spec)
|
||||
algo_module = None
|
||||
try:
|
||||
algo_module = importlib.import_module(merged["module"])
|
||||
except ModuleNotFoundError:
|
||||
pass
|
||||
plan = _install(
|
||||
self.engine, self.spec, merged,
|
||||
algo_module=algo_module, rank_to_pe=rank_to_pe,
|
||||
)
|
||||
self._ipcq_plan = plan
|
||||
self._ipcq_config = merged
|
||||
return plan
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
@@ -258,6 +351,24 @@ class RuntimeContext:
|
||||
"""Allocate a tensor in HBM without initialization (like torch.empty)."""
|
||||
return self._create_tensor(shape, dtype, name, pattern=None, dp=dp)
|
||||
|
||||
def from_numpy(self, arr: Any):
|
||||
"""Create a host-side tensor wrapping a numpy array.
|
||||
|
||||
Mirrors ``torch.from_numpy``. The returned tensor is NOT deployed
|
||||
to any PE — it lives in an in-memory host staging buffer. Use
|
||||
``target.copy_(host_tensor)`` to scatter its contents into a
|
||||
sharded, deployed tensor.
|
||||
"""
|
||||
import numpy as np
|
||||
from kernbench.runtime_api.tensor import Tensor
|
||||
|
||||
arr_c = np.ascontiguousarray(arr)
|
||||
dtype_str = _numpy_to_dtype_str(arr_c.dtype)
|
||||
t = Tensor(shape=tuple(arr_c.shape), dtype=dtype_str, name="host")
|
||||
t._host_buffer = arr_c
|
||||
t._memory_store = getattr(self.engine, "_memory_store", None)
|
||||
return t
|
||||
|
||||
def _create_tensor(
|
||||
self,
|
||||
shape: tuple[int, ...],
|
||||
@@ -418,13 +529,12 @@ class RuntimeContext:
|
||||
TensorArgShard,
|
||||
)
|
||||
from kernbench.runtime_api.tensor import Tensor
|
||||
from kernbench.triton_emu.registry import register_kernel
|
||||
from kernbench.triton_emu.registry import _kernels, register_kernel
|
||||
|
||||
# Register kernel (idempotent)
|
||||
try:
|
||||
register_kernel(kernel_name, kernel_fn)
|
||||
except ValueError:
|
||||
pass
|
||||
# Register kernel (idempotent overwrite — last call wins).
|
||||
# Tests can re-register the same kernel_name with a different
|
||||
# function; the user's most recent launch must use the latest fn.
|
||||
_kernels[kernel_name] = kernel_fn
|
||||
|
||||
# Collect tensors and scalars
|
||||
tensor_args: list[Tensor] = []
|
||||
@@ -506,6 +616,7 @@ class RuntimeContext:
|
||||
|
||||
# Per-SIP kernel launch: each SIP gets TensorArgs with local va_base
|
||||
last_handle = None
|
||||
_pending_handles: list[tuple[Any, int]] = []
|
||||
for sip_id in sorted(sip_set):
|
||||
sip_kernel_args: list = []
|
||||
sip_cube_set: set[int] = set()
|
||||
@@ -566,10 +677,17 @@ class RuntimeContext:
|
||||
target_cubes=target_cubes,
|
||||
target_pe=target_pe,
|
||||
))
|
||||
# Defer wait until all SIPs are submitted (multi-SIP CCL needs
|
||||
# all participating PEs to be live concurrently — waiting
|
||||
# per-SIP would deadlock when ranks span SIP boundaries).
|
||||
_pending_handles.append((h, sip_id))
|
||||
last_handle = h
|
||||
|
||||
# Drain pending handles now that every SIP has a launch posted.
|
||||
for h, sip_id in _pending_handles:
|
||||
self.wait(h, _meta={
|
||||
"phase": "kernel", "name": kernel_name,
|
||||
"sip": sip_id, "target_pe": target_pe,
|
||||
})
|
||||
last_handle = h
|
||||
|
||||
return last_handle
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
"""PyTorch-compatible distributed communication shim (ADR-0023 D11).
|
||||
|
||||
Provides a ``torch.distributed``-like API whose public surface matches
|
||||
real PyTorch so that bench code looks identical to a DDP training script.
|
||||
|
||||
Only the ``ahbm`` backend is implemented. It:
|
||||
|
||||
1. Reads ``ccl.yaml`` to decide which collective algorithm to run.
|
||||
2. Derives world_size from the algorithm entry, the defaults section, or
|
||||
from the topology spec (``system.sips.count × sip.cube_mesh × pe_layout``).
|
||||
3. At ``init_process_group`` time, eagerly installs the IPCQ neighbor
|
||||
table once (one-time comm setup — mirrors NCCL communicator creation).
|
||||
4. On each ``all_reduce(tensor)`` call, reads per-shard metadata from the
|
||||
tensor handle and dispatches ``torch.launch`` with the registered
|
||||
kernel. The kernel performs intra-PE ring/tree/mesh CCL via IPCQ,
|
||||
and Phase 2 DataExecutor replays math + copies from op_log so
|
||||
MemoryStore is correct when ``all_reduce`` returns.
|
||||
|
||||
Host bench code uses only real-PyTorch names:
|
||||
dist.init_process_group, dist.is_initialized, dist.get_world_size,
|
||||
dist.get_rank, dist.get_backend, dist.all_reduce, dist.barrier
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
from typing import Any
|
||||
|
||||
|
||||
class AhbmCCLBackend:
|
||||
"""Ahbm CCL backend — drives kernel-level collectives via IPCQ."""
|
||||
|
||||
def __init__(self, torch_ctx: Any) -> None:
|
||||
from kernbench.ccl.install import (
|
||||
load_ccl_config,
|
||||
resolve_algorithm_config,
|
||||
)
|
||||
|
||||
self.ctx = torch_ctx
|
||||
self._cfg_all = load_ccl_config()
|
||||
self._merged = resolve_algorithm_config(self._cfg_all)
|
||||
self._algo_module = importlib.import_module(self._merged["module"])
|
||||
self._world_size = self._resolve_world_size()
|
||||
|
||||
# Eager IPCQ install — ``init_process_group`` time. Mirrors NCCL
|
||||
# communicator creation: done once, reused across every subsequent
|
||||
# collective call on the same process group.
|
||||
self.ctx.install_ipcq(
|
||||
algorithm=self._merged["algorithm"],
|
||||
world_size_override=self._world_size,
|
||||
)
|
||||
|
||||
def _resolve_world_size(self) -> int:
|
||||
"""Derive world_size (priority: algorithm override > defaults > topology).
|
||||
|
||||
Topology derivation:
|
||||
sips × cubes_per_sip × pes_per_cube
|
||||
"""
|
||||
if "world_size" in self._merged:
|
||||
return int(self._merged["world_size"])
|
||||
defaults = self._cfg_all.get("defaults", {})
|
||||
if "world_size" in defaults:
|
||||
return int(defaults["world_size"])
|
||||
spec = self.ctx.spec or {}
|
||||
sips = int(spec.get("system", {}).get("sips", {}).get("count", 1))
|
||||
cm = spec.get("sip", {}).get("cube_mesh", {})
|
||||
cubes_per_sip = int(cm.get("w", 1)) * int(cm.get("h", 1))
|
||||
pl = spec.get("cube", {}).get("pe_layout", {})
|
||||
corners = pl.get("corners", [])
|
||||
pe_per_corner = int(pl.get("pe_per_corner", 1))
|
||||
pes_per_cube = pe_per_corner * max(len(corners), 1)
|
||||
return sips * cubes_per_sip * pes_per_cube
|
||||
|
||||
@property
|
||||
def world_size(self) -> int:
|
||||
return self._world_size
|
||||
|
||||
def all_reduce(self, tensor: Any, op: str = "sum") -> None:
|
||||
"""Dispatch the configured CCL algorithm as a single kernel launch.
|
||||
|
||||
Raises if ``op != "sum"`` (current kernels only implement add
|
||||
reduction) or if the tensor's shard count disagrees with the
|
||||
world_size that was installed into PE_IPCQ.
|
||||
"""
|
||||
if op != "sum":
|
||||
raise NotImplementedError(f"all_reduce op={op!r} not supported")
|
||||
if tensor._handle is None:
|
||||
raise RuntimeError(
|
||||
f"Tensor '{tensor.name}' is not deployed (call torch.zeros "
|
||||
"with a DPPolicy first)"
|
||||
)
|
||||
shards = tensor._handle.shards
|
||||
if len(shards) != self._world_size:
|
||||
raise RuntimeError(
|
||||
f"all_reduce tensor has {len(shards)} shards but the "
|
||||
f"ahbm backend was installed with world_size="
|
||||
f"{self._world_size}; adjust the tensor's DPPolicy or "
|
||||
"restart the process group"
|
||||
)
|
||||
n_elem = shards[0].nbytes // tensor.itemsize
|
||||
kernel_fn = self._algo_module.kernel
|
||||
kernel_args = self._algo_module.kernel_args(self._world_size, n_elem)
|
||||
self.ctx.launch(
|
||||
self._merged["algorithm"], kernel_fn, tensor, *kernel_args,
|
||||
)
|
||||
|
||||
def barrier(self) -> None:
|
||||
# Single-driver model → no cross-process sync needed. Keeping the
|
||||
# method so ``dist.barrier()`` is callable (pytorch-compat surface).
|
||||
return None
|
||||
|
||||
|
||||
class DistributedContext:
|
||||
"""torch.distributed-compat facade.
|
||||
|
||||
Public surface matches real PyTorch so bench code reads identically
|
||||
to a DDP training script. Single-driver semantics: ``get_rank()``
|
||||
always returns 0 because kernbench runs as one Python process;
|
||||
``get_world_size()`` returns the CCL group size (number of PEs
|
||||
participating in the collective).
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._backend: AhbmCCLBackend | None = None
|
||||
|
||||
def init_process_group(
|
||||
self,
|
||||
backend: str = "ahbm",
|
||||
world_size: int | None = None,
|
||||
rank: int | None = None,
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
"""Create the default process group.
|
||||
|
||||
``world_size`` and ``rank`` are accepted for API parity with
|
||||
``torch.distributed.init_process_group`` but ignored — the ahbm
|
||||
backend derives both from ``ccl.yaml`` + topology automatically
|
||||
(like reading ``RANK``/``WORLD_SIZE`` env vars in real DDP).
|
||||
"""
|
||||
if backend != "ahbm":
|
||||
raise ValueError(
|
||||
f"Unsupported backend '{backend}'. Only 'ahbm' is supported."
|
||||
)
|
||||
ctx = getattr(self, "_ctx_ref", None)
|
||||
if ctx is None:
|
||||
raise RuntimeError(
|
||||
"DistributedContext not bound to a RuntimeContext"
|
||||
)
|
||||
self._backend = AhbmCCLBackend(torch_ctx=ctx)
|
||||
|
||||
def is_initialized(self) -> bool:
|
||||
return self._backend is not None
|
||||
|
||||
def get_world_size(self) -> int:
|
||||
self._ensure_initialized()
|
||||
return self._backend.world_size
|
||||
|
||||
def get_rank(self) -> int:
|
||||
# Single-driver kernbench: there is only one host rank.
|
||||
self._ensure_initialized()
|
||||
return 0
|
||||
|
||||
def get_backend(self) -> str:
|
||||
self._ensure_initialized()
|
||||
return "ahbm"
|
||||
|
||||
def all_reduce(self, tensor: Any, op: str = "sum") -> None:
|
||||
self._ensure_initialized()
|
||||
self._backend.all_reduce(tensor, op=op)
|
||||
|
||||
def barrier(self) -> None:
|
||||
self._ensure_initialized()
|
||||
self._backend.barrier()
|
||||
|
||||
def _ensure_initialized(self) -> None:
|
||||
if self._backend is None:
|
||||
raise RuntimeError(
|
||||
"Default process group has not been initialized. "
|
||||
"Call init_process_group(backend='ahbm') first."
|
||||
)
|
||||
@@ -152,3 +152,30 @@ class MmuUnmapMsg:
|
||||
target_cubes: tuple[int, ...] | Literal["all"] = "all"
|
||||
target_pe: int | Literal["all"] = "all"
|
||||
msg_type: Literal["mmu_unmap"] = "mmu_unmap"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IpcqInitMsg:
|
||||
"""IPCQ neighbor table install (sideband fan-out, ADR-0023 D10/D12).
|
||||
|
||||
Backend issues this at ``init_process_group`` time to install per-PE
|
||||
IPCQ neighbor tables. Each entry covers one direction (N/S/E/W) and
|
||||
carries the peer's IpcqEndpoint plus this PE's own rx_buffer base
|
||||
and a pre-wired SimPy Store for credit return fast path (D9).
|
||||
|
||||
Routing is similar to MmuMapMsg.
|
||||
"""
|
||||
|
||||
correlation_id: str
|
||||
request_id: str
|
||||
target_sips: tuple[int, ...] | Literal["all"] = "all"
|
||||
target_cubes: tuple[int, ...] | Literal["all"] = "all"
|
||||
target_pe: int | tuple[int, ...] | Literal["all"] = "all"
|
||||
# entries: tuple[IpcqInitEntry, ...] — kept as tuple of plain objects to
|
||||
# avoid a runtime import cycle (IpcqInitEntry lives in
|
||||
# kernbench.common.ipcq_types).
|
||||
entries: tuple = ()
|
||||
backpressure_mode: str = "sleep" # "poll" | "sleep"
|
||||
buffer_kind: str = "tcm" # "tcm" | "hbm" | "sram"
|
||||
credit_size_bytes: int = 16
|
||||
msg_type: Literal["ipcq_init"] = "ipcq_init"
|
||||
|
||||
@@ -146,6 +146,11 @@ class Tensor:
|
||||
self._handle: TensorHandle | None = None
|
||||
self._ctx_ref: weakref.ref | None = None # set by RuntimeContext
|
||||
self._memory_store = None # set by RuntimeContext when enable_data=True
|
||||
# Host-side staging buffer for torch.from_numpy() results. A tensor
|
||||
# with a non-None _host_buffer is NOT deployed to any PE — it lives
|
||||
# only on the host. Use `target.copy_(host_tensor)` to scatter the
|
||||
# data into a deployed, sharded target tensor.
|
||||
self._host_buffer: np.ndarray | None = None
|
||||
|
||||
def __del__(self) -> None:
|
||||
if self._ctx_ref is None or self._handle is None:
|
||||
@@ -166,15 +171,85 @@ class Tensor:
|
||||
|
||||
@property
|
||||
def data(self) -> np.ndarray:
|
||||
"""Tensor data as numpy array. Returns actual values when enable_data=True,
|
||||
zeros placeholder otherwise (like an uninitialized tensor)."""
|
||||
if self._memory_store is not None and self._handle is not None:
|
||||
shard = self._handle.shards[0]
|
||||
"""Tensor data as numpy array.
|
||||
|
||||
Gathers all shards into a single full-shape array. Returns actual
|
||||
values when enable_data=True, zeros placeholder otherwise (like an
|
||||
uninitialized tensor). Alias of ``numpy()``.
|
||||
"""
|
||||
return self.numpy()
|
||||
|
||||
def _shard_store_addr(self, shard: TensorShard) -> int:
|
||||
"""MemoryStore key for a shard.
|
||||
|
||||
Kernels read tensors via VA (translated to PA by PE_DMA's MMU when
|
||||
a mapping exists, otherwise the addr is treated as a PA-equivalent
|
||||
key). Tensor I/O therefore writes/reads at ``va_base + offset_bytes``
|
||||
when ``va_base`` is set, falling back to ``shard.pa`` for the
|
||||
VA-less mode used by some legacy paths.
|
||||
"""
|
||||
if self._handle and self._handle.va_base:
|
||||
return self._handle.va_base + shard.offset_bytes
|
||||
return shard.pa
|
||||
|
||||
def numpy(self) -> np.ndarray:
|
||||
"""Return a single numpy array gathered from all shards.
|
||||
|
||||
Mirrors ``torch.Tensor.numpy()``. In kernbench, sharded tensors are
|
||||
gathered into a single full-shape ndarray according to each shard's
|
||||
``offset_bytes`` / ``nbytes`` range.
|
||||
"""
|
||||
np_dtype = _numpy_dtype(self.dtype)
|
||||
# Host-side tensor (created via torch.from_numpy) has no shards.
|
||||
if self._host_buffer is not None:
|
||||
return self._host_buffer.copy()
|
||||
if self._handle is None or self._memory_store is None:
|
||||
return np.zeros(self.shape, dtype=np_dtype)
|
||||
flat = np.zeros(math.prod(self.shape), dtype=np_dtype)
|
||||
for shard in self._handle.shards:
|
||||
start = shard.offset_bytes // self.itemsize
|
||||
count = shard.nbytes // self.itemsize
|
||||
try:
|
||||
return self._memory_store.read("hbm", shard.pa, shape=self.shape, dtype=self.dtype)
|
||||
piece = self._memory_store.read(
|
||||
"hbm", self._shard_store_addr(shard),
|
||||
)
|
||||
except KeyError:
|
||||
pass
|
||||
return np.zeros(self.shape, dtype=_numpy_dtype(self.dtype))
|
||||
continue
|
||||
flat[start : start + count] = (
|
||||
np.asarray(piece, dtype=np_dtype).reshape(-1)[:count]
|
||||
)
|
||||
return flat.reshape(self.shape)
|
||||
|
||||
def copy_(self, source: "Tensor") -> "Tensor":
|
||||
"""In-place copy from another tensor into self.
|
||||
|
||||
Mirrors ``torch.Tensor.copy_()``. If ``source`` is a host tensor
|
||||
(from ``torch.from_numpy``), its ndarray is split across self's
|
||||
shards using each shard's byte range. If ``source`` is a deployed
|
||||
(sharded) tensor, its contents are gathered first and then
|
||||
re-scattered into self's shard layout.
|
||||
|
||||
Shapes must match. Returns self.
|
||||
"""
|
||||
if self._handle is None or self._memory_store is None:
|
||||
raise RuntimeError(
|
||||
f"Tensor '{self.name}' must be deployed before copy_()"
|
||||
)
|
||||
if source.shape != self.shape:
|
||||
raise ValueError(
|
||||
f"copy_ shape mismatch: self={self.shape} source={source.shape}"
|
||||
)
|
||||
np_dtype = _numpy_dtype(self.dtype)
|
||||
arr = source.numpy().astype(np_dtype, copy=False)
|
||||
flat = np.ascontiguousarray(arr).reshape(-1)
|
||||
for shard in self._handle.shards:
|
||||
start = shard.offset_bytes // self.itemsize
|
||||
count = shard.nbytes // self.itemsize
|
||||
piece = flat[start : start + count].copy()
|
||||
self._memory_store.write(
|
||||
"hbm", self._shard_store_addr(shard), piece,
|
||||
)
|
||||
return self
|
||||
|
||||
@property
|
||||
def itemsize(self) -> int:
|
||||
|
||||
Reference in New Issue
Block a user