kernbench2/docs/adr/ADR-0044-eval-gemm-harness.md

ADR-0044: GEMM Evaluation Harness — scripts/gemm_sweep.py + tests/gemm/

Status

Accepted

Documents the GEMM evaluation/characterization harness; verified against the implementation (constants, tile sizes, figure set, and the script↔test split cross-checked). The D5/D6 caveats are recorded limitations, not inaccuracies.

Amended by ADR-0054: the sweep + renderers moved into the milestone-1h-gemm bench (single home); scripts/gemm_sweep.py and tests/gemm/ now re-export from it. D1/D2's "data generation stays a manual script / heavy work is opt-in" is superseded by the eval-bench pattern (one bench regenerates everything; MILESTONE_FAST=1 reuses the committed JSON).

Context

ADR-0014 (PE pipeline) and ADR-0042 (tile-plan generators) define the GEMM implementation; ADR-0033 defines the latency model. None of them describe how GEMM performance is swept and characterized — the shape/variant sweep that produces the timing data, and the figures that interpret it. This ADR pins that harness.

Unlike the allreduce harness (ADR-0043), the GEMM sweep is heavy (24 sim runs: 8 shapes × 3 operand-staging variants; the 512 shape alone is 2048 tiles). That weight drives the split below.

Decision

D1. Two-layer split — heavy data generation (script) vs. fast figures (tests)

• Data generation stays a manual script: scripts/gemm_sweep.py runs matmul-composite (ADR-0042 plans) across shapes × variants via the same run_bench path the CLI uses, harvests result.engine.op_log, and writes docs/diagrams/gemm_sweep.json (per-stage / per-engine wall-clock
  • occupancy + record counts + pe/composite windows).
• Figure rendering is test-generated: tests/gemm/ reads the committed gemm_sweep.json and renders matplotlib PNGs into docs/diagrams/gemm_plots/. These tests are fast and run by default.

Rationale: a slide-deck-scale sim sweep does not belong in every pytest run, but the figures (cheap, deterministic) should regenerate freely and be guarded by CI. This mirrors CLAUDE.md's script-vs-test split (scripts for heavy/manual generation; tests for fast assertions).

D2. Slow regenerator test wraps the script

tests/gemm/test_gemm_sweep.py is marked @pytest.mark.slow (excluded by the default addopts: -m "not slow"). It invokes scripts/gemm_sweep.py via subprocess to regenerate gemm_sweep.json on demand (pytest -m slow tests/gemm/test_gemm_sweep.py). The sweep logic has a single home (the script); the test only wraps it, so there is no duplicated sim-driving code.

D3. Figure set (3 charts, load_ref variant)

Test	PNG	Content
test_plot_gemm_stage_breakdown.py	gemm_stage_breakdown.png	per-stage engine wall-clock (DMA in / Fetch / GEMM / DMA out)
test_plot_gemm_mac_utilization.py	gemm_mac_utilization_measured.png	GEMM util % + useful eff %
test_plot_gemm_mac_utilization.py	gemm_mac_utilization_theoretical_vs_measured.png	theoretical vs simulator-measured util/eff

tests/gemm/_gemm_plot_helpers.py holds the shared renderers (series logic mirrors the GEMM _render_* functions in scripts/build_overview_slides.py, which still draws these natively in the PPTX). Not collected (no test_ prefix). Each test_plot_* skips if gemm_sweep.json is absent.

D4. Tile sizes are data-driven; under-tile shapes are flagged

Tile sizes are read from gemm_sweep.json (tile_sizes), which the sweep records from PeSchedulerComponent.TILE_M/K/N = 32/64/32 — the authoritative source. Shapes with M<TILE_M ∨ K<TILE_K ∨ N<TILE_N are flagged ("under-tile") on the charts. The 512³ shape is excluded from the figures (EXCLUDED_SHAPES).

D5. Theoretical model — inherited constants, NOT yet ADR-verified

The "theoretical" curves use an analytical ideal-pipeline model with constants copied verbatim from scripts/build_overview_slides.py:

HBM_GBS = 256.0   # GB/s        T_STAGE = 16.0 ns
D_STAGES = 3                    BPE = 2

These are not yet sourced against the ADRs. Notably ADR-0033's 256 is burst_bytes (256 B), a different quantity than this 256 GB/s, and ADR-0033 derives bandwidth as pc_bw_gbs = hbm_to_router_bw_gbs / num_pcs. T_STAGE/stage-count are not traced to ADR-0014 here. The model is therefore consistent with the existing deck script, not verified against the ADRs, and the constants are duplicated (deck + helper). Reconciling them (source from topology/ADR-0033/0014, de-duplicate) is deferred — see Open questions.

D6. Known naming caveat — _measured chart

gemm_mac_utilization_measured.png currently plots the theoretical ideal-pipeline numbers (its footnote says so), only the filename says "measured". This is a known misnomer pending a decision to either repoint its content to the simulator-measured series or retitle it.

Consequences

Positive

• GEMM figures are test-generated and CI-guarded, like allreduce.
• The heavy sweep stays opt-in, keeping the default test run fast.
• Single source for the sweep logic (the script), reused by the slow test.

Negative / limitations

• The theoretical-model constants (D5) are unverified and duplicated.
• The _measured figure is a misnomer (D6).
• build_overview_slides.py still renders the GEMM bars natively from gemm_sweep.json rather than embedding these PNGs — the deck rewiring to consume the test artifacts is not done.

Dependencies

• ADR-0013: verification strategy.
• ADR-0014 / ADR-0042: PE pipeline + tile-plan generators — the GEMM implementation the sweep measures; D4's stage record counts come from ADR-0042 D2/D3.
• ADR-0033: latency model — the source the D5 constants should (but do not yet) trace to.
• ADR-0043: the sibling allreduce evaluation harness.

Open questions

• Reconcile D5 constants against topology.yaml / ADR-0033 / ADR-0014 and de-duplicate (one source for the model parameters)?
• Resolve the D6 _measured naming (repoint content vs. retitle)?
• Rewire build_overview_slides.py to embed the gemm_plots/ PNGs instead of native bar-drawing?