mukesh
cc1bbd0ab7
Move the GEMM + allreduce sweep/render logic out of scripts/ and tests/
into two self-contained eval benches so a user can regenerate every
result + figure with one command:
kernbench run --bench milestone-1h-gemm (MILESTONE_FAST=1 reuses JSON)
kernbench run --bench milestone-1h-ccl
- benches/milestone_1h_{gemm,ccl}.py: single home for each domain; the
run(torch) entry drives the sweeps and writes figures into
benches/1H_milestone_output/{gemm,ccl}/ (gitignored), then submits a
sentinel tensor to satisfy the run_bench contract.
- tests/gemm + tests/sccl helpers and scripts/gemm_sweep.py become thin
re-export/wrapper shims over the benches (single source preserved); the
pytest-only param builders + _run_distributed wrapper stay in the shim.
- eval-bench pattern: a bench may drive many configs + build its own
per-config engines (extends ADR-0045 D5; reverses ADR-0044 D1/D2).
ADR-0054 (EN+KO) records the design; ADR-0043/0044/0045 + CLAUDE.md CLI
Semantics amended; ADR INDEX regenerated. Verified: milestone benches run
clean (ok=True, all artifacts), full suite 67 passed, lang-pairs OK.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
136 lines
6.1 KiB
Markdown
136 lines
6.1 KiB
Markdown
# ADR-0043: Allreduce Evaluation Harness — `tests/sccl/`
|
||
|
||
## Status
|
||
|
||
Accepted
|
||
|
||
Documents the `tests/sccl/` evaluation harness; verified against the
|
||
implementation (constants, file set, and sweep dimensions cross-checked).
|
||
|
||
**Amended by ADR-0054**: the driver core, sweeps, and renderers moved into
|
||
the `milestone-1h-ccl` bench (single home); `tests/sccl/_allreduce_helpers.py`
|
||
now re-exports from it (keeping the pytest-only param builders +
|
||
`_run_distributed` wrapper local). The figure tests are unchanged.
|
||
|
||
## Context
|
||
|
||
ADR-0032 defines the intercube all-reduce *algorithm*; ADR-0023/0024/0027
|
||
define the IPCQ backend, the rank=SIP launcher, and `mp.spawn`. None of
|
||
them describe **how the allreduce is exercised and characterized** — the
|
||
correctness tests, the latency/buffer-kind sweeps, and the derived plots.
|
||
ADR-0013 (verification strategy) is the general policy; this ADR pins the
|
||
concrete allreduce harness so the "evaluation" half of the work is
|
||
documented, not just the implementation.
|
||
|
||
The harness lives under `tests/sccl/` (the package created when the
|
||
allreduce tests were consolidated). It supersedes the earlier flat
|
||
`tests/test_allreduce_multidevice.py` + `tests/test_distributed_*` layout.
|
||
|
||
## Decision
|
||
|
||
### D1. Drive evaluation through the public `torch.distributed` path
|
||
|
||
Correctness and the sweeps run the collective through the real DDP-shaped
|
||
path — `init_process_group(backend="ahbm") → mp.spawn → dist.all_reduce`
|
||
(ADR-0024/0027) — not the lower-level `ctx.launch`. A shared helper
|
||
`_run_distributed(tmp_path, monkeypatch, topo_path, corr_id, n_elem)` in
|
||
`tests/sccl/_allreduce_helpers.py` builds the engine, runs the workers, and
|
||
returns `(engine, n_cubes)`. `monkeypatch.chdir` points the backend's
|
||
`load_ccl_config()` (cwd lookup) at a per-case temp `ccl.yaml`.
|
||
|
||
A direct-launch reference (`run_allreduce`) is retained in the same helper
|
||
module — not for the distributed tests, but because the IPCQ buffer-kind /
|
||
root-center micro-tests under `tests/` import it.
|
||
|
||
### D2. One file per evaluation concern
|
||
|
||
| File | Concern | `torch.distributed`? |
|
||
|---|---|---|
|
||
| `test_allreduce_ring_torus_mesh.py` | correctness across ring_1d / torus_2d (2×3) / mesh_2d_no_wrap (2×3) | yes |
|
||
| `test_distributed_default_topology.py` | full path on `topology.yaml` as-is | yes |
|
||
| `test_plot_latency_sweep.py` | latency sweep rows (n_elem × topology) | yes |
|
||
| `test_plot_buffer_kind_sweep.py` | TCM/SRAM/HBM sweep rows | yes |
|
||
| `test_plot_topology_diagram.py` | topology.png (pure matplotlib) | no |
|
||
| `test_plot_comparison_fsim.py` | broken-axis model-vs-FSIM comparison | no |
|
||
| `test_intercube_root_center.py` | ADR-0032 center-root latency guard (direct path) | no |
|
||
|
||
`_allreduce_helpers.py` holds the shared plumbing (driver, config writers,
|
||
sweep/buffer-kind constants, plot aggregators, topology-diagram + FSIM
|
||
comparison emitters). It is not collected (no `test_` prefix).
|
||
|
||
### D3. Latency metric — critical-path `pe_exec_ns`
|
||
|
||
The reported latency per config is `crit_ns = max(pe_exec_ns)` over
|
||
`engine._results` — the slowest rank's PE execution time. This is the
|
||
number plotted on every latency chart and recorded in `summary.csv`.
|
||
|
||
### D4. Sweep dimensions
|
||
|
||
- **Latency sweep**: `n_elem ∈ {8, 32, 64, 128, 512, 1024, 2048, 4096,
|
||
8192, 16384, 32768, 49152}` (16 excluded — collides with `n_cubes`) ×
|
||
topology ∈ {ring_1d (6), torus_2d 2×3 (6), mesh_2d_no_wrap 2×3 (6)}.
|
||
- **Buffer-kind sweep**: `buffer_kind ∈ {tcm, sram, hbm}` × a smaller
|
||
`n_elem` grid, on torus_2d 6-SIP (3×2). buffer_kind is set in the temp
|
||
`ccl.yaml` (read by the backend at `init_process_group`, ADR-0023 D6).
|
||
|
||
The 2×3 / 3×2 grids exercise the explicit-`w/h` SIP resolution
|
||
(ADR-0024 D5).
|
||
|
||
### D5. Derived plots via `pytest_sessionfinish` aggregators
|
||
|
||
Sweep tests are xdist-friendly: each parametrized case writes one JSON row
|
||
to a staging dir. The conftest `pytest_sessionfinish` hook (controller node
|
||
only) calls the aggregators in `_allreduce_helpers.py`:
|
||
|
||
- `_aggregate_sweep_plots()` → per-topology PNGs + `summary.csv`
|
||
- `aggregate_buffer_kind_plot()` → the TCM/SRAM/HBM comparison PNG + csv
|
||
|
||
The topology-diagram and FSIM-comparison figures are emitted directly by
|
||
their own `test_plot_*` tests (no row staging — they are pure functions of
|
||
`topology.yaml` and `summary.csv` respectively). All outputs land in
|
||
`docs/diagrams/allreduce_latency_plots/` and are **derived artifacts** per
|
||
CLAUDE.md (consistent-with-ADRs, no Phase-2 gate).
|
||
|
||
### D6. The FSIM comparison reference is a hardcoded constant
|
||
|
||
`emit_comparison_fsim_plot()` overlays the model curves against a single
|
||
external FSIM single-device reference (`366 µs`), held as a literal — there
|
||
is no external data file. The "measured" series comes from the simulator
|
||
(`op_log` GEMM count, `composite_window_ns`); the "theoretical" series is a
|
||
hand-derived analytical model (the same one ADR-0044 D5 flags as
|
||
ADR-unverified).
|
||
|
||
## Consequences
|
||
|
||
### Positive
|
||
|
||
- The allreduce is evaluated through the same API a real DDP script uses,
|
||
so the harness doubles as an integration test of ADR-0024/0027.
|
||
- Figures regenerate on every `pytest` run from committed data; no manual
|
||
plot step.
|
||
- Rectangular-grid sweeps gave the regression coverage that surfaced the
|
||
ADR-0024 D5 `w/h` fix.
|
||
|
||
### Negative / limitations
|
||
|
||
- The full latency sweep runs in the default `pytest` (~minutes); it is not
|
||
marked `slow`. (Contrast ADR-0044, where the GEMM sweep is `slow`.)
|
||
- `test_intercube_root_center.py` carries a latency *threshold* assertion
|
||
(ADR-0032 center-root guard) — the only absolute-latency assertion in the
|
||
suite; it is sensitive to latency-model changes (ADR-0033).
|
||
|
||
## Dependencies
|
||
|
||
- **ADR-0013**: verification strategy (general policy this specializes).
|
||
- **ADR-0023 / ADR-0024 / ADR-0027**: IPCQ backend, rank=SIP launcher,
|
||
`mp.spawn` — the path D1 drives.
|
||
- **ADR-0032**: the algorithm under evaluation; D4 grids exercise its
|
||
topology branches.
|
||
- **ADR-0044**: the sibling GEMM evaluation harness.
|
||
|
||
## Open questions
|
||
|
||
- Should the latency sweep be marked `slow` for parity with the GEMM sweep?
|
||
- Should the FSIM reference move from a hardcoded constant to a versioned
|
||
data file?
|