kernbench2

ywkang/kernbench2

Fork 0

Commit Graph

Author	SHA1	Message	Date
ywkang	cfc2d74ec4	Refactor ccl_allreduce bench: rank=SIP only, remove rank=PE legacy path The unified ccl_allreduce bench previously carried two execution models in one worker with ``if world_size == n_sips:`` branching: - TP mode (rank = SIP, ADR-0024/0027): proper ProcessGroup semantics. - Legacy rank = PE mode: single-driver worker allocating one big tensor distributed across all PEs via _derive_dp, with kernel-level SPMD via program_id. The second model is unnecessary — intra-SIP PE-level collectives are expressed inside the kernel (tl.send/tl.recv with program_id, IPCQ) and do not need a host-side ProcessGroup. Removing it lets the bench be a clean reference implementation of the TP launcher. benches/ccl_allreduce.py: - Config resolved once in run() via _resolve_cfg -> _BenchCfg dataclass. - rank != n_sips now raises RuntimeError explicitly. - _worker / _allocate_rank_tile / _init_with_rank_value / _report each have one concern; duplicated init + verification paths collapsed. - _derive_dp and the second verify+print block deleted. - 166 lines -> 91 lines. ccl.yaml: - mesh_allreduce_4 (world_size: 4) and tree_allreduce_7 (world_size: 7) algorithm entries removed (rank = PE only). - Algorithm kernel files (kernbench.ccl.algorithms.mesh_allreduce, tree_allreduce) kept as-is for direct-dispatch future use. tests/test_ccl_allreduce_matrix.py: - Matrix shrinks from 7 cases to 3: ring × {tcm, hbm, sram} at ws = topology SIP count (= 2). mesh_2x2, tree_binary_7, ring_multi_cube, and the three ring_*_8 cases removed. tests/test_ccl_performance.py: - _run_8rank renamed to _run_ring; world_size: 8 override dropped; now exercises rank = SIP ring all-reduce. tests/test_mp_spawn.py, tests/test_ccl_ddp_launcher.py: - Monkeypatch target updated from bench.worker to bench._worker (signature now takes BenchCfg instead of (rank, world_size)). 555 passed, 1 intentional skip. Tests that directly call install_ipcq(world_size_override=N) for kernel-level sanity (test_ccl_hello_world_guide, test_recv_copy_to_dst, test_tl_recv_async, test_ccl_deadlock_detection) are unchanged — they never went through the bench and still exercise the kernel-only path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-14 16:45:27 -07:00
ywkang	4ba0a83e71	Implement ADR-0024 Phase A: SIP-level TP launcher MVP Scope (Phase A): - D1: world_size fallback = SIP count (rank = SIP, TP boundary) - D9: greenlet-local get_rank + _bind_rank (single-driver fallback = 0) - D10: torch.ahbm.set_device + torch.accelerator.set_device_index alias - D11: tensor placement scoped to current-device SIP (post-hoc pe_index shift — ADR-0026 replaces with structural coords) - D12/D13: multi-greenlet run() with simple round-robin scheduler; hybrid dispatch (ws == SIP count → multi-greenlet, else legacy single-worker for ccl.yaml override compat) - D7 partial: backend.all_reduce submit + yield + wait via launch()'s new _defer_wait flag; parent-less greenlets skip yield - Relaxed shard-count check (len(shards) > 0 instead of == world_size) - rank_to_pe = SIP-representative [(r, 0, 0)] when ws <= n_sips Deferred to Phase B: - Engine-routed install (D2) — keeps sideband - install_plan.py module (D6) — keeps install.py - Epoch barrier (D7 full) — simple yield is sufficient for ring ws=2 mock - Validator registry (D8) - Cross-SIP multi-greenlet + real kernel integration — matrix ring_default_ws hangs in SimPy drain despite ADR-0025 direction fix; marked xfail(run=False) pending Phase B diagnosis (suspected per-rank kernel_args / program_id mismatch) Tests: - test_ccl_ddp_launcher.py (6 new tests) — D1/D9/D10/D11/D12/D13 - test_ccl_allreduce_matrix.py — ring_default_ws xfail'd, override cases (ring_tcm_8 / hbm_8 / sram_8 / multi_cube / mesh_2x2 / tree_binary_7) all pass via legacy path 514 tests pass, 1 xfail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-14 09:00:28 -07:00

Author

SHA1

Message

Date

ywkang

cfc2d74ec4

Refactor ccl_allreduce bench: rank=SIP only, remove rank=PE legacy path

The unified ccl_allreduce bench previously carried two execution models
in one worker with ``if world_size == n_sips:`` branching:
  - TP mode (rank = SIP, ADR-0024/0027): proper ProcessGroup semantics.
  - Legacy rank = PE mode: single-driver worker allocating one big tensor
    distributed across all PEs via _derive_dp, with kernel-level SPMD via
    program_id.

The second model is unnecessary — intra-SIP PE-level collectives are
expressed inside the kernel (tl.send/tl.recv with program_id, IPCQ) and
do not need a host-side ProcessGroup. Removing it lets the bench be a
clean reference implementation of the TP launcher.

benches/ccl_allreduce.py:
- Config resolved once in run() via _resolve_cfg -> _BenchCfg dataclass.
- rank != n_sips now raises RuntimeError explicitly.
- _worker / _allocate_rank_tile / _init_with_rank_value / _report each
  have one concern; duplicated init + verification paths collapsed.
- _derive_dp and the second verify+print block deleted.
- 166 lines -> 91 lines.

ccl.yaml:
- mesh_allreduce_4 (world_size: 4) and tree_allreduce_7 (world_size: 7)
  algorithm entries removed (rank = PE only).
- Algorithm kernel files (kernbench.ccl.algorithms.mesh_allreduce,
  tree_allreduce) kept as-is for direct-dispatch future use.

tests/test_ccl_allreduce_matrix.py:
- Matrix shrinks from 7 cases to 3: ring × {tcm, hbm, sram} at ws =
  topology SIP count (= 2). mesh_2x2, tree_binary_7, ring_multi_cube,
  and the three ring_*_8 cases removed.

tests/test_ccl_performance.py:
- _run_8rank renamed to _run_ring; world_size: 8 override dropped; now
  exercises rank = SIP ring all-reduce.

tests/test_mp_spawn.py, tests/test_ccl_ddp_launcher.py:
- Monkeypatch target updated from bench.worker to bench._worker
  (signature now takes BenchCfg instead of (rank, world_size)).

555 passed, 1 intentional skip. Tests that directly call
install_ipcq(world_size_override=N) for kernel-level sanity
(test_ccl_hello_world_guide, test_recv_copy_to_dst, test_tl_recv_async,
test_ccl_deadlock_detection) are unchanged — they never went through
the bench and still exercise the kernel-only path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-14 16:45:27 -07:00

ywkang

4ba0a83e71

Implement ADR-0024 Phase A: SIP-level TP launcher MVP

Scope (Phase A):
- D1: world_size fallback = SIP count (rank = SIP, TP boundary)
- D9: greenlet-local get_rank + _bind_rank (single-driver fallback = 0)
- D10: torch.ahbm.set_device + torch.accelerator.set_device_index alias
- D11: tensor placement scoped to current-device SIP (post-hoc pe_index
  shift — ADR-0026 replaces with structural coords)
- D12/D13: multi-greenlet run() with simple round-robin scheduler;
  hybrid dispatch (ws == SIP count → multi-greenlet, else legacy
  single-worker for ccl.yaml override compat)
- D7 partial: backend.all_reduce submit + yield + wait via launch()'s
  new _defer_wait flag; parent-less greenlets skip yield
- Relaxed shard-count check (len(shards) > 0 instead of == world_size)
- rank_to_pe = SIP-representative [(r, 0, 0)] when ws <= n_sips

Deferred to Phase B:
- Engine-routed install (D2) — keeps sideband
- install_plan.py module (D6) — keeps install.py
- Epoch barrier (D7 full) — simple yield is sufficient for ring ws=2 mock
- Validator registry (D8)
- Cross-SIP multi-greenlet + real kernel integration — matrix
  ring_default_ws hangs in SimPy drain despite ADR-0025 direction fix;
  marked xfail(run=False) pending Phase B diagnosis (suspected per-rank
  kernel_args / program_id mismatch)

Tests:
- test_ccl_ddp_launcher.py (6 new tests) — D1/D9/D10/D11/D12/D13
- test_ccl_allreduce_matrix.py — ring_default_ws xfail'd, override
  cases (ring_tcm_8 / hbm_8 / sram_8 / multi_cube / mesh_2x2 /
  tree_binary_7) all pass via legacy path

514 tests pass, 1 xfail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-14 09:00:28 -07:00

2 Commits