kernbench2/docs/adr/ADR-0049-ver-probe-subcommand.md

# ADR-0049: `kernbench probe` Subcommand — Traffic-Pattern Verification Harness

## Status

Accepted (2026-05-22).

Pins down the traffic-pattern catalog, formula-vs-actual comparison, and
invariant checks (monotonicity, D2H ≥ H2D, etc.) exposed by
`probes/probe.py::run_probe(...)`. ADR-0010 (CLI surface) enumerates the
`kernbench probe` subcommand, but **what probe actually measures** and
**which invariants it judges PASS/FAIL** had no ADR-level coverage.

## First action

`run_probe(topology_path, case_filter=None)` performs four startup steps:

1. `Path(topology_path).expanduser().resolve()` → absolute path.
2. `load_topology(path)` → `TopologyGraph` (graph + spec).
3. `_build_edge_map(graph)` → a `{(src, dst): Edge}` lookup table.
4. Instantiate `AddressResolver(graph)` + `PathRouter(graph)`.

Then it sets `nbytes = 32768` (= 32 KiB, the summary-table reference
size) and `show_all = (case_filter is None or case_filter == "all")`.

In short, **probe's first act is "load the topology once and prepare
edge map / resolver / router, plus pin 32 KiB as the standard measurement
size"**. After that, the H2D → D2H → PE DMA categories execute in
separate `GraphEngine` instances (no cross-talk between cases).

## Context

`kernbench probe` was introduced as a verification tool for these
purposes:

- **Manual ground truth**: when a real-simulation result (`kernbench run
  --bench ...`) shows abnormal latency, derive the answer for a simple
  traffic pattern in isolation and compare.
- **Formula vs actual**: check whether the analytical model
  (wire latency + overhead + drain) matches the simulator's
  `total_ns`. A mismatch points to which simplifying assumption in
  ADR-0033 is missing.
- **Monotonicity check**: latency should grow monotonically with hop
  count.
- **Utilization sweep**: a BW-utilization table across data sizes
  (4 KiB ~ 1 MiB).

Without an ADR for this tool:

- Adding a new traffic-pattern category (e.g., MCpuDma, IPCQ) is hard
  because the table format / measurement units of existing categories
  aren't documented at the ADR level.
- The basis for the monotonicity check (hop count? cube distance? wire
  length?) is ambiguous.
- The reference size 32 KiB and the sweep `[4 KiB, 16 KiB, 64 KiB, 256
  KiB, 1 MiB]` are only discoverable by reading source.

## Decision

### D1. Three case categories — H2D / D2H / PE DMA

Each category has a distinct data path in the topology and gets its own
summary table + sweep table + route-detail block.

- **H2D (Host → Device Write)**: `MemoryWriteMsg(dst_sip=0, dst_cube,
  dst_pe=0, pattern="zero")` flows along `pcie_ep → io_cpu → m_cpu →
  hbm_ctrl`. The cube index varies the hop count:
  - h2d-1hop: cube=0, hops=1
  - h2d-2hop: cube=4, hops=2
  - h2d-3hop: cube=8, hops=3
  - h2d-4hop: cube=12, hops=4
- **D2H (Device → Host Read)**: `MemoryReadMsg(src_sip=0, src_cube,
  src_pe=0)`. Total latency = forward command path + reverse data path.
  Same 4-hops category as H2D.
- **PE DMA (PE-initiated)**: `PeDmaMsg(src_sip, src_cube, src_pe,
  dst_pa)`. Five cases cover varying cube/PE positions:
  - pe-local-hbm: same cube, same PE
  - pe-same-half-hbm: same cube, different PE (PE 1)
  - pe-cross-half-hbm: same cube, far PE (PE 4)
  - pe-cross-cube-hbm-best: adjacent cube (cube 1)
  - pe-cross-cube-hbm-worst: diagonal far cube (cube 15)

The cube indices 4/8/12 (H2D) and 1/4/15 (PE DMA) are meaningful for a
4 × 4 cube mesh (`sip.cube_mesh.w=4, h=4`); changes to the mesh size
require these to be updated in lockstep.

### D2. Standard measurement size — `nbytes = 32768` (32 KiB)

Every case in the summary table runs once with `nbytes=32768`. 32 KiB
was chosen because:

- DMA overhead and BW drain are balanced — neither dominates.
- It compares cleanly against the one-shot transfer size of several
  sub-units (TCM, register file).

Per-size utilization variations are shown in a separate sweep table
(D3).

### D3. Utilization sweep — `[4 KiB, 16 KiB, 64 KiB, 256 KiB, 1 MiB]`

`SWEEP_SIZES = [4096, 16384, 65536, 262144, 1048576]`,
`SWEEP_LABELS = ["4KB", "16KB", "64KB", "256KB", "1MB"]`. Per size:

```
drain   = nbytes / bottleneck_bw
total   = overhead + wire + drain
eff_bw  = nbytes / total
util%   = eff_bw / bottleneck_bw × 100
```

When `bn_bw is None or <= 0`, the column shows 0.0 %. The intent: the
table shows in one view how small transfers become overhead-bound and
large transfers become drain-bound as hop count rises.

### D4. Measured columns — actual / formula / breakdown

Per-case columns:

- `Actual` (total_ns): the SimPy run's `trace["total_ns"]`.
- `Ovhd`: sum of `node.attrs["overhead_ns"]` along the path (formula).
- `Drain`: `nbytes / min(edge.bw_gbs over path)` (formula).
- `Wire`: `Σ edge.distance_mm * (ns_per_mm from spec)`.
- `Ovhd%` / `Drain%`: each portion as a percentage of Actual. Wire is
  usually too small to display.
- `Eff.BW`: `nbytes / total_ns` (measured BW).
- `BN.BW`: bottleneck bandwidth (formula). The minimum edge BW along
  the path. Missing edge BW shows "-".
- `Util%`: `Eff.BW / BN.BW × 100`. 100 % means the single-stream BW
  upper bound is reached.

A large gap between the formula sum (`wire + ovhd + drain`) and Actual
signals a factor the simplified model misses (a place to inspect
ADR-0033's assumptions).

### D5. Automatic invariant checks — PASS/FAIL

The following invariants are reported with `[v] PASS` / `[x] FAIL`:

- **H2D / D2H monotonic increase**: as hop count rises, actual latency
  must grow monotonically. `all(lats[i] < lats[i+1] for ...)`.
- **D2H ≥ H2D**: for the same hop index, D2H ≥ H2D (D2H has both
  forward command and reverse data legs). `all(d2h[i].total >=
  h2d[i].total)`.
- **PE DMA best < worst**: cross-cube best (adjacent) latency must be
  less than cross-cube worst (diagonal).
- **PE DMA local vs remote**: prints the local BN BW vs remote BN BW
  side-by-side (informational, not PASS/FAIL).

When a check fails, a single clear line surfaces the regression for
human review.

### D6. Route detail — per-hop timestamp trace

After the summary and sweep tables, each case's path and cumulative
per-hop timestamps (`_hop_timestamps`) appear in a separate section:

- H2D: leg1 (`pcie_ep → io_cpu`) + leg2 (`io_cpu → m_cpu`) + leg3
  (`m_cpu → hbm_ctrl`) + per-hop trace.
- D2H: forward (cmd, no data) and reverse (data) traces shown
  separately.
- PE DMA: `pe_dma → router → hbm_ctrl` path + per-hop trace.

Each hop's timestamp is cumulative `wire_ns + overhead_ns`. The
terminal hop's annotation appends `drain:Xns`. Bottleneck edges are
marked `<BN:XXGB/s>` so they are visually identifiable.

### D7. Semantics of the `case_filter` argument

- `None` or `"all"`: run all cases (default).
- Other strings: run only the case whose name matches exactly. Example:
  `kernbench probe --case h2d-2hop`.

Within a category, cases with `name != case_filter` are skipped; if
only one data point remains, the category's monotonicity / D2H ≥ H2D
comparisons are naturally skipped.

The CLI parser's `--case` default is `"all"`, so omitting it runs
everything.

### D8. Fresh GraphEngine per case

Each of the 4 H2D, 4 D2H, and 5 PE DMA cases runs in **its own
GraphEngine** (`engine = GraphEngine(graph)`). Reasons:

- Isolate accumulated state (op_log, completion tracking, allocators)
  so cases do not cross-talk.
- Guarantee one case's traffic does not perturb another case's BW
  measurement.

This isolation lets probe results be interpreted as **single-flow**
per-case latency. Multi-flow contention measurement is handled by
separate tooling (e.g., the `pe2pe_overview` plot or ADR-0033's
multi-flow merging model).

### D9. Output-format stability

probe's stdout is meant for humans; precise column widths, separators,
and whitespace are **not** a machine-readable contract. Automated tools
that wish to parse probe output should use a separate JSON-output mode
(not yet implemented).

The `[v]` / `[x]` prefix on PASS/FAIL lines is a stable CI grep anchor.

## Alternatives Considered

### A1. Register probe as another bench (`@bench(name="probe")`)

Rejected. probe is a verification tool, not a bench — multi-engine
execution for sweeps/analysis and PASS/FAIL invariant output are
essential, none of which fits ADR-0045's "single device + single
RuntimeContext" bench model.

### A2. Exit code 1 on monotonicity violation

Rejected (currently). probe is positioned as a human inspection tool —
PASS/FAIL is printed and exit is 0. A wrapper can `grep "\[x\]"` to
decide. A future `--strict` flag could opt into non-zero exits.

### A3. Externalize the case catalog to YAML

Rejected (currently). The 8 cases (4 H2D + 4 D2H + 5 PE DMA = 13 total)
are hardcoded and their semantics are tightly bound to the mesh
topology. Moving cube-index meaning (4, 8, 12 / 1, 4, 15) into YAML
would require separate documentation and lose cohesion. Externalize
only when case additions become frequent.

### A4. Add multi-flow contention measurement

Rejected (out of probe scope). D8's single-flow isolation is probe's
core intent. Multi-flow contention belongs in a different area of the
ADR-0033 latency model — either a separate tool or a new case
category.

## Consequences

- probe's case catalog (D1) and measurement units (D2/D3) are pinned at
  ADR level, so new traffic categories know which table format to
  follow.
- The semantics of the formula-vs-actual columns (D4) are locked in, so
  questions like "why is Drain% 5 % or 70 %?" can quickly be linked to
  ADR-0033 assumption checks.
- Automatic invariant checks (D5) are pinned, so latency-model changes
  immediately catch monotonicity / D2H ≥ H2D regressions.
- D8's case-isolation is explicit, so probe results are safe to read as
  single-flow measurements. If multi-flow is needed, a separate tool
  track is clearly required.
- A2's strict-mode flag is recorded as a follow-up so CI integration
  has a minimal change path when requested.