Adds compute_parallelism_sensitivity() + ParallelismSensitivityRow to
auto_explore.py. For a baseline ConfigScore (picked from the Pareto set),
sweeps each parallelism knob (CP, TP, PP, DP, EP) individually while
holding others fixed. Reports latency + memory-fit per value.
Sweep values start at 1 and step by 2 (multiples of 2 rather than only
powers of 2), giving finer granularity for the visual than the enumerator's
sparser set. CP goes up to 256 (per real-deployment scale), TP to 64,
PP to 32.
New UI panel in Auto Explore tab: 5 subplots (log/log), one per knob:
- Solid line = latency where the config fits memory
- Red X markers = infeasible (out of budget)
- Dotted vertical line = baseline value
User picks which Pareto row is the "baseline" via a number input; the
sensitivity chart re-computes around it.
Behaviour caveat noted during verification: for large PP the FFN AR can
cross a SIP boundary (uses stage_latencies.py:730 sips_used tier
selection), producing a step-up in latency. This is the existing model's
choice, honestly reflected in the chart. Whether the FFN AR should span
only one PP stage's ranks (thus not cross SIPs) is a separate discussion
about stage_latencies.py.
Verified:
- 11 pytest tests pass (added 2 new for parallelism sensitivity)
- Smoke: at Llama 70B decode 128K with baseline CP=8/TP=16/PP=1/DP=1:
* CP sweep: 4 fits, 8 = baseline optimum, 16+ overshoots memory
* TP sweep: 8/16 fit, 16 = baseline optimum, 32 slower (spans SIPs)
* PP sweep: 1 = baseline, 2+ slower due to sips_used tier drop for FFN AR
* DP sweep: same shape as PP
* EP sweep: monotone decreasing (bigger EP = smaller per-PE FFN)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extends the memory-only autosuggest to search the full 9-dimensional
parallelism space (CP, TP, PP, DP, kv_shard_mode, ffn_shard_scope,
tp_placement, cp_placement, cp_ring_variant) and rank feasible configs
on a 3D Pareto frontier: (latency ↓, pes_used ↓, efficiency ↑).
Throughput is stored on ConfigScore for display but is deliberately NOT
a Pareto axis because for a single-request analysis it collapses to
1 / latency, which would collapse the frontier.
Reuses existing physics (stage_latencies.all_stages + all_ffn_stages,
memory_layout.compute_memory) — no new formulas.
Single-request latency formula fix: PP does NOT reduce single-request
decode/prefill latency because the request has to traverse every layer
sequentially regardless of pipeline depth. The initial version had
latency ~ per_layer × layers_per_stage, which incorrectly rewarded high
PP. Corrected to latency ~ per_layer × model.layers.
Enumerator prunes:
- PP > model.layers
- TP > 4 × h_q
- ffn_shard_scope contains 'DP' when dp=1 (redundant)
- cp_ring_variant='qoml' when cp=1 (no-op)
Full sweep on Llama 3.1 70B: ~28,800 configs enumerated in ~7s, ~7k-10k
feasible (varies with S_kv), 2-7 unique Pareto configs. Faster context
lengths produce richer frontiers; at 1M, memory forces a single
dominant config (128 PEs, HBM 85%).
Verified:
- 9 pytest tests pass (enumerate, score, Pareto, subset invariants)
- Manual: Llama 70B decode at 8K/64K/128K/1M produces physically
sensible Pareto (CP=8/TP=16 wins latency; smaller-PE options
appear at longer context up to memory limits)
Next: Streamlit tab UI in app.py + verification against more presets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mukesh