bench(milestone-gqa-headline): drop misleading single_user_/multi_user_ panels
The legacy panel names suggested batched serving semantics they never had — all four modeled a single user with KV sharded differently (C=1 single-cube; C=4 multi-cube Cube-SP), at toy dims (T_q=4, S_kv≤128). The single-KV-group C=8 panel + the new milestone-gqa-decode-4cases bench cover the meaningful comparisons; pytest regression already covers C=1/C=4 configurations end-to-end at richer scale. Changes: - milestone_gqa_headline.py: drop the 4 legacy panels; _PANELS now contains only single_kv_group_prefill_gqa_c8_p8. Update docstring. - tests/attention/test_milestone_gqa_headline.py: drop the 3 legacy- panel architectural tests (Ring-KV traffic, root-only decode write, per-CUBE distributed output) and test_decode_panels_use_real_gqa (no decode panels in this bench anymore). Equivalent properties are asserted in test_milestone_gqa_single_kv_group_prefill_panel.py (64 dma_writes, 896 ipcq_copy) and test_milestone_gqa_decode_4cases.py (1 dma_write at cube 6, 189 ipcq_copy). - tests/attention/test_milestone_gqa_single_kv_group_prefill_panel.py: drop test_existing_prefill_panel_runner_backward_compat (it exercised multi_user_prefill_gqa which no longer exists). - scripts/paper/paper_plot_gqa.py: replace the 4 legacy _LABELS entries with the single single_kv_group_prefill_gqa_c8_p8 label. - Regenerate 1H_milestone_output/gqa_headline/sweep.json from the new panel set. Verification: 9/9 tests pass across the 3 affected test files. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -1,25 +1,14 @@
|
||||
"""Phase 1 spec test for P7: headline milestone-gqa bench with real GQA.
|
||||
"""Tests for the milestone-gqa-headline bench.
|
||||
|
||||
P7 wires the new ``_gqa_decode`` and ``_gqa_prefill`` kernels into a
|
||||
new 4-panel milestone bench (independent from the existing
|
||||
``milestone-gqa-llama70b`` which still covers the baseline kernels).
|
||||
Real GQA (``h_q > h_kv`` with G=8) runs end-to-end through a
|
||||
milestone-style sweep + sweep.json output.
|
||||
The bench wires the new ``_gqa_attention_prefill_long`` kernel into a
|
||||
single-KV-group milestone panel (LLaMA-3.1-70B target, C=8 P=8). The
|
||||
4 legacy C=1 / C=4 panels (single_user_*, multi_user_*) were removed
|
||||
once pytest regression covered those configurations comprehensively
|
||||
and the comparative decode work moved into the
|
||||
``milestone-gqa-decode-4cases`` bench.
|
||||
|
||||
Restriction in P7 first cut:
|
||||
- C ≤ 4 (single-row inter-CUBE ring SFR; ADR-0060 §B leaves
|
||||
multi-row rings for follow-on)
|
||||
- Single SIP (default ``topology.yaml`` 4×4 cube mesh; 4-SIP
|
||||
headline left to follow-on)
|
||||
- No figure renderers (defer to a separate cycle)
|
||||
|
||||
Panels (4 total):
|
||||
single_user_prefill_gqa: C=1, T_q=4, S_kv=16 (no ring)
|
||||
multi_user_prefill_gqa : C=4, T_q=4, S_kv=16 (Ring KV, 3 steps)
|
||||
single_user_decode_gqa : C=1, P=8, h_q=8, h_kv=1, S_kv=64 (M-fold + intra-cube chain)
|
||||
multi_user_decode_gqa : C=4, P=8, h_q=8, h_kv=1, S_kv=128 (M-fold + 2-level chain)
|
||||
|
||||
Phase 1 (this commit): tests only — bench module lands in Phase 2.
|
||||
Single SIP scope (default ``topology.yaml`` 4×4 cube mesh; 4-SIP
|
||||
headline deferred per ADR-0060 §B-item-1).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -35,11 +24,7 @@ from kernbench.topology.builder import resolve_topology
|
||||
BENCH_NAME = "milestone-gqa-headline"
|
||||
|
||||
PANELS = (
|
||||
"single_user_prefill_gqa",
|
||||
"multi_user_prefill_gqa",
|
||||
"single_kv_group_prefill_gqa_c8_p8",
|
||||
"single_user_decode_gqa",
|
||||
"multi_user_decode_gqa",
|
||||
)
|
||||
|
||||
|
||||
@@ -98,52 +83,7 @@ def test_sweep_json_has_expected_panels(monkeypatch):
|
||||
assert {r["panel"] for r in data["rows"]} == set(PANELS)
|
||||
|
||||
|
||||
def test_decode_panels_use_real_gqa(monkeypatch):
|
||||
"""ADR-0060 §A.1 headline: decode panels must use h_q = G·h_kv with G>1."""
|
||||
data = _sweep_json(monkeypatch)
|
||||
cfg = data["config"]
|
||||
assert cfg["h_q_decode"] > cfg["h_kv_decode"], (
|
||||
f"decode must use real GQA (h_q > h_kv); got "
|
||||
f"h_q={cfg['h_q_decode']}, h_kv={cfg['h_kv_decode']}"
|
||||
)
|
||||
|
||||
|
||||
# ── Per-panel architectural assertions ────────────────────────────────
|
||||
|
||||
|
||||
def _row(rows, panel: str) -> dict:
|
||||
for r in rows:
|
||||
if r["panel"] == panel:
|
||||
return r
|
||||
raise AssertionError(f"missing row for panel {panel!r}")
|
||||
|
||||
|
||||
def test_prefill_ring_panel_has_ipcq_traffic(monkeypatch):
|
||||
"""multi_user_prefill_gqa uses Ring KV → IPCQ traffic > 0."""
|
||||
data = _sweep_json(monkeypatch)
|
||||
row = _row(data["rows"], "multi_user_prefill_gqa")
|
||||
n_copy = row["op_log_summary"].get("ipcq_copy_count", 0)
|
||||
assert n_copy > 0, (
|
||||
f"multi_user_prefill_gqa must have Ring KV traffic; got "
|
||||
f"ipcq_copy_count={n_copy}"
|
||||
)
|
||||
|
||||
|
||||
def test_decode_reduce_panel_writes_once(monkeypatch):
|
||||
"""multi_user_decode_gqa: chain reduce-to-root → exactly 1 dma_write."""
|
||||
data = _sweep_json(monkeypatch)
|
||||
row = _row(data["rows"], "multi_user_decode_gqa")
|
||||
n_writes = row["op_log_summary"]["dma_write_count"]
|
||||
assert n_writes == 1, (
|
||||
f"multi_user_decode_gqa root-only write: expected 1; got {n_writes}"
|
||||
)
|
||||
|
||||
|
||||
def test_prefill_panel_distributes_output(monkeypatch):
|
||||
"""multi_user_prefill_gqa: per-CUBE distributed output → dma_write_count == C."""
|
||||
data = _sweep_json(monkeypatch)
|
||||
row = _row(data["rows"], "multi_user_prefill_gqa")
|
||||
n_writes = row["op_log_summary"]["dma_write_count"]
|
||||
assert n_writes == 4, (
|
||||
f"multi_user_prefill_gqa per-CUBE distributed: expected 4; got {n_writes}"
|
||||
)
|
||||
# Per-panel architectural assertions for the surviving single-KV-group
|
||||
# panel live in tests/attention/test_milestone_gqa_single_kv_group_prefill_panel.py.
|
||||
# Decode architectural assertions live in
|
||||
# tests/attention/test_milestone_gqa_decode_4cases.py.
|
||||
|
||||
@@ -116,34 +116,6 @@ def test_single_kv_group_prefill_panel_runner_smoke():
|
||||
)
|
||||
|
||||
|
||||
# ── T3: existing panel calls still work (backward compat) ────────────
|
||||
|
||||
|
||||
def test_existing_prefill_panel_runner_backward_compat():
|
||||
"""Existing callers of ``_run_prefill_panel`` (no ``P``/``T_q``/
|
||||
``d_head`` overrides) must continue to work after the signature
|
||||
extension. Exercises the ``multi_user_prefill_gqa`` panel dims
|
||||
from ``_PANEL_DISPATCH``.
|
||||
|
||||
Passes today AND after Phase 2.
|
||||
"""
|
||||
panel_name = "multi_user_prefill_gqa"
|
||||
_kind, params = _PANEL_DISPATCH[panel_name]
|
||||
topo = resolve_topology(str(TOPOLOGY_DEFAULT))
|
||||
|
||||
def _bench_fn(ctx):
|
||||
_run_prefill_panel(
|
||||
ctx,
|
||||
panel=panel_name,
|
||||
C=params["C"], S_kv=params["S_kv"],
|
||||
)
|
||||
|
||||
result = run_bench(
|
||||
topology=topo, bench_fn=_bench_fn,
|
||||
device=resolve_device(None),
|
||||
engine_factory=_engine_factory,
|
||||
)
|
||||
assert result.completion.ok, (
|
||||
f"existing prefill panel {panel_name!r} must still run with "
|
||||
f"the original kwargs; got {result.completion}"
|
||||
)
|
||||
# Backward-compat test removed when the legacy multi_user_prefill_gqa
|
||||
# panel was dropped; signature-extension contract is now exercised
|
||||
# implicitly by the live single_kv_group_prefill_gqa_c8_p8 panel.
|
||||
|
||||
Reference in New Issue
Block a user