Both auto tabs now accept a scope choice at run time:
- "Run sweep — Attention" → include_ffn=False
- "Run sweep — Attn + FFN/MoE" → include_ffn=True
Each button runs an independent sweep and caches its result under its
own session_state key. The most recently clicked button determines the
displayed view; both caches persist so users can flip between the two
scopes without re-running.
Tabs:
- Renamed "Auto Explore" → "Auto Suggest Parallelism"
(accurately reflects that it only varies parallelism knobs; HW is
held at the sidebar values).
- "Auto Hardware" tab unchanged.
- Still 6 top-level tabs; no additional tabs added.
Core changes:
auto_explore.py:
- New include_ffn: bool = True parameter on _sum_visible_latency,
_efficiency, score_config, run_auto_explore, compute_parallelism_
sensitivity. False drops all FFN stages from the summed latency.
auto_hardware.py:
- New include_ffn: bool = True parameter on joint_explore,
compute_sensitivity, _best_parallelism_for_hw, _best_parallelism_
two_stage. Forwards to score_config.
Both defaults keep existing tests byte-identical.
Verified:
- 23 pytest tests pass (added 3 new: attn-only latency lower, attn-only
Pareto non-empty, joint HW attn-only faster than full).
- Smoke: Llama 70B decode 128K:
* Attn+FFN best latency: 12.85 ms (unchanged)
* Attention-only best: 7.35 ms (~57% of full)
* Both sensitivities top-rank bw_hbm_gbs (physics preserved).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds compute_parallelism_sensitivity() + ParallelismSensitivityRow to
auto_explore.py. For a baseline ConfigScore (picked from the Pareto set),
sweeps each parallelism knob (CP, TP, PP, DP, EP) individually while
holding others fixed. Reports latency + memory-fit per value.
Sweep values start at 1 and step by 2 (multiples of 2 rather than only
powers of 2), giving finer granularity for the visual than the enumerator's
sparser set. CP goes up to 256 (per real-deployment scale), TP to 64,
PP to 32.
New UI panel in Auto Explore tab: 5 subplots (log/log), one per knob:
- Solid line = latency where the config fits memory
- Red X markers = infeasible (out of budget)
- Dotted vertical line = baseline value
User picks which Pareto row is the "baseline" via a number input; the
sensitivity chart re-computes around it.
Behaviour caveat noted during verification: for large PP the FFN AR can
cross a SIP boundary (uses stage_latencies.py:730 sips_used tier
selection), producing a step-up in latency. This is the existing model's
choice, honestly reflected in the chart. Whether the FFN AR should span
only one PP stage's ranks (thus not cross SIPs) is a separate discussion
about stage_latencies.py.
Verified:
- 11 pytest tests pass (added 2 new for parallelism sensitivity)
- Smoke: at Llama 70B decode 128K with baseline CP=8/TP=16/PP=1/DP=1:
* CP sweep: 4 fits, 8 = baseline optimum, 16+ overshoots memory
* TP sweep: 8/16 fit, 16 = baseline optimum, 32 slower (spans SIPs)
* PP sweep: 1 = baseline, 2+ slower due to sips_used tier drop for FFN AR
* DP sweep: same shape as PP
* EP sweep: monotone decreasing (bigger EP = smaller per-PE FFN)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extends the memory-only autosuggest to search the full 9-dimensional
parallelism space (CP, TP, PP, DP, kv_shard_mode, ffn_shard_scope,
tp_placement, cp_placement, cp_ring_variant) and rank feasible configs
on a 3D Pareto frontier: (latency ↓, pes_used ↓, efficiency ↑).
Throughput is stored on ConfigScore for display but is deliberately NOT
a Pareto axis because for a single-request analysis it collapses to
1 / latency, which would collapse the frontier.
Reuses existing physics (stage_latencies.all_stages + all_ffn_stages,
memory_layout.compute_memory) — no new formulas.
Single-request latency formula fix: PP does NOT reduce single-request
decode/prefill latency because the request has to traverse every layer
sequentially regardless of pipeline depth. The initial version had
latency ~ per_layer × layers_per_stage, which incorrectly rewarded high
PP. Corrected to latency ~ per_layer × model.layers.
Enumerator prunes:
- PP > model.layers
- TP > 4 × h_q
- ffn_shard_scope contains 'DP' when dp=1 (redundant)
- cp_ring_variant='qoml' when cp=1 (no-op)
Full sweep on Llama 3.1 70B: ~28,800 configs enumerated in ~7s, ~7k-10k
feasible (varies with S_kv), 2-7 unique Pareto configs. Faster context
lengths produce richer frontiers; at 1M, memory forces a single
dominant config (128 PEs, HBM 85%).
Verified:
- 9 pytest tests pass (enumerate, score, Pareto, subset invariants)
- Manual: Llama 70B decode at 8K/64K/128K/1M produces physically
sensible Pareto (CP=8/TP=16 wins latency; smaller-PE options
appear at longer context up to memory limits)
Next: Streamlit tab UI in app.py + verification against more presets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mukesh