New module extends auto_explore into the hardware co-design space. For a
fixed model + workload, sweeps hardware knobs (pe_hbm_gb, bw_hbm_gbs,
peak_tflops_f16, bw_intra_gbs, bw_inter_gbs, bw_intersip_gbs) and, for
each hardware candidate, searches parallelism for the latency-minimum
that fits memory. Returns:
- all_scores: every (hw, parallelism) pair that fits, sorted by latency
- pareto_scores: 2D Pareto frontier on (latency ↓, cost_score ↓)
- sensitivity: per-knob rel_speedup when doubled from the best-fast HW
baseline. Ranks which HW knob gives the biggest speedup — a co-design
signal.
Three sweep depths trade coverage for time:
- two_stage: 1 HW candidate (defaults) × autosuggest's memory-min
parallelism. Fast (~1s), useful for the sensitivity ranking alone.
- balanced: 64 HW × ~2k reduced-parallelism configs = ~130k joint evals,
~10-20s. Default UI setting.
- coarse: 729 HW × ~2k configs = ~1.4M joint evals, ~2-5 min.
Reduced parallelism sweep for the inner loop: CP × TP × PP × DP ×
kv_shard_mode (1,920 configs), other 4 knobs held at latency-friendly
defaults (ffn_shard_scope='TP+CP', tp_placement='cube', cp_placement='pe',
cp_ring_variant='qoml' for decode, 'kv' for prefill). Full 28,800-config
auto_explore per HW would take 6+ minutes — too slow.
Cost proxy: sum of (knob / knob_default). 6.0 at defaults. Not dollars —
a rough capability score where higher = "more spec'd hardware".
Verified:
- 9 pytest tests pass:
* enumeration counts match expected (1, 64, 729)
* default cost_score = 6.0
* Pareto non-dominated + subset of all_scores
* every knob is monotone-non-worsening when doubled
* for Llama 70B decode, bw_hbm_gbs tops the sensitivity ranking
(physically correct: memory-bound workload)
- Smoke: Llama 70B decode 128K balanced sweep in ~20s produces
5 Pareto configs; best 7.57 ms with 1024 GB/s HBM BW. Doubling
HBM BW gives 33% additional speedup; every other knob < 1.5%.
Next: Streamlit UI tab consuming this in Commit 5.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mukesh
6ef09dd6f5