# ADR-0043: Allreduce Evaluation Harness — `tests/sccl/`

## Status

Accepted

Documents the `tests/sccl/` evaluation harness; verified against the
implementation (constants, file set, and sweep dimensions cross-checked).

## Context

ADR-0032 defines the intercube all-reduce *algorithm*; ADR-0023/0024/0027
define the IPCQ backend, the rank=SIP launcher, and `mp.spawn`. None of
them describe **how the allreduce is exercised and characterized** — the
correctness tests, the latency/buffer-kind sweeps, and the derived plots.
ADR-0013 (verification strategy) is the general policy; this ADR pins the
concrete allreduce harness so the "evaluation" half of the work is
documented, not just the implementation.

The harness lives under `tests/sccl/` (the package created when the
allreduce tests were consolidated). It supersedes the earlier flat
`tests/test_allreduce_multidevice.py` + `tests/test_distributed_*` layout.

## Decision

### D1. Drive evaluation through the public `torch.distributed` path

Correctness and the sweeps run the collective through the real DDP-shaped
path — `init_process_group(backend="ahbm") → mp.spawn → dist.all_reduce`
(ADR-0024/0027) — not the lower-level `ctx.launch`. A shared helper
`_run_distributed(tmp_path, monkeypatch, topo_path, corr_id, n_elem)` in
`tests/sccl/_allreduce_helpers.py` builds the engine, runs the workers, and
returns `(engine, n_cubes)`. `monkeypatch.chdir` points the backend's
`load_ccl_config()` (cwd lookup) at a per-case temp `ccl.yaml`.

A direct-launch reference (`run_allreduce`) is retained in the same helper
module — not for the distributed tests, but because the IPCQ buffer-kind /
root-center micro-tests under `tests/` import it.

### D2. One file per evaluation concern

| File | Concern | `torch.distributed`? |
|---|---|---|
| `test_allreduce_ring_torus_mesh.py` | correctness across ring_1d / torus_2d (2×3) / mesh_2d_no_wrap (2×3) | yes |
| `test_distributed_default_topology.py` | full path on `topology.yaml` as-is | yes |
| `test_plot_latency_sweep.py` | latency sweep rows (n_elem × topology) | yes |
| `test_plot_buffer_kind_sweep.py` | TCM/SRAM/HBM sweep rows | yes |
| `test_plot_topology_diagram.py` | topology.png (pure matplotlib) | no |
| `test_plot_comparison_fsim.py` | broken-axis model-vs-FSIM comparison | no |
| `test_intercube_root_center.py` | ADR-0032 center-root latency guard (direct path) | no |

`_allreduce_helpers.py` holds the shared plumbing (driver, config writers,
sweep/buffer-kind constants, plot aggregators, topology-diagram + FSIM
comparison emitters). It is not collected (no `test_` prefix).

### D3. Latency metric — critical-path `pe_exec_ns`

The reported latency per config is `crit_ns = max(pe_exec_ns)` over
`engine._results` — the slowest rank's PE execution time. This is the
number plotted on every latency chart and recorded in `summary.csv`.

### D4. Sweep dimensions

- **Latency sweep**: `n_elem ∈ {8, 32, 64, 128, 512, 1024, 2048, 4096,
  8192, 16384, 32768, 49152}` (16 excluded — collides with `n_cubes`) ×
  topology ∈ {ring_1d (6), torus_2d 2×3 (6), mesh_2d_no_wrap 2×3 (6)}.
- **Buffer-kind sweep**: `buffer_kind ∈ {tcm, sram, hbm}` × a smaller
  `n_elem` grid, on torus_2d 6-SIP (3×2). buffer_kind is set in the temp
  `ccl.yaml` (read by the backend at `init_process_group`, ADR-0023 D6).

The 2×3 / 3×2 grids exercise the explicit-`w/h` SIP resolution
(ADR-0024 D5).

### D5. Derived plots via `pytest_sessionfinish` aggregators

Sweep tests are xdist-friendly: each parametrized case writes one JSON row
to a staging dir. The conftest `pytest_sessionfinish` hook (controller node
only) calls the aggregators in `_allreduce_helpers.py`:

- `_aggregate_sweep_plots()` → per-topology PNGs + `summary.csv`
- `aggregate_buffer_kind_plot()` → the TCM/SRAM/HBM comparison PNG + csv

The topology-diagram and FSIM-comparison figures are emitted directly by
their own `test_plot_*` tests (no row staging — they are pure functions of
`topology.yaml` and `summary.csv` respectively). All outputs land in
`docs/diagrams/allreduce_latency_plots/` and are **derived artifacts** per
CLAUDE.md (consistent-with-ADRs, no Phase-2 gate).

### D6. The FSIM comparison reference is a hardcoded constant

`emit_comparison_fsim_plot()` overlays the model curves against a single
external FSIM single-device reference (`366 µs`), held as a literal — there
is no external data file. The "measured" series comes from the simulator
(`op_log` GEMM count, `composite_window_ns`); the "theoretical" series is a
hand-derived analytical model (the same one ADR-0044 D5 flags as
ADR-unverified).

## Consequences

### Positive

- The allreduce is evaluated through the same API a real DDP script uses,
  so the harness doubles as an integration test of ADR-0024/0027.
- Figures regenerate on every `pytest` run from committed data; no manual
  plot step.
- Rectangular-grid sweeps gave the regression coverage that surfaced the
  ADR-0024 D5 `w/h` fix.

### Negative / limitations

- The full latency sweep runs in the default `pytest` (~minutes); it is not
  marked `slow`. (Contrast ADR-0044, where the GEMM sweep is `slow`.)
- `test_intercube_root_center.py` carries a latency *threshold* assertion
  (ADR-0032 center-root guard) — the only absolute-latency assertion in the
  suite; it is sensitive to latency-model changes (ADR-0033).

## Dependencies

- **ADR-0013**: verification strategy (general policy this specializes).
- **ADR-0023 / ADR-0024 / ADR-0027**: IPCQ backend, rank=SIP launcher,
  `mp.spawn` — the path D1 drives.
- **ADR-0032**: the algorithm under evaluation; D4 grids exercise its
  topology branches.
- **ADR-0044**: the sibling GEMM evaluation harness.

## Open questions

- Should the latency sweep be marked `slow` for parity with the GEMM sweep?
- Should the FSIM reference move from a hardcoded constant to a versioned
  data file?