Previously only reloaded auto_explore / auto_hardware / autosuggest /
stage_latencies / memory_layout. When we edit pe_weight_layout,
tensor_sharding, topology_map, pipeline_diagram, or optimization_report,
Streamlit was still holding the pre-edit versions until a full restart.
Add all five to the reload list so any visualization-module edit lands
on the next Streamlit rerun without needing Ctrl+C.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When cp_placement=pe packs multiple CP ranks intra-cube, the PE-level
layout now:
1. Colors each PE's background by its CP rank (8-color palette wraps
for cp > 8). Cp_placement=cube keeps the historical light-blue
styling (only one CP rank per cube tile).
2. Adds "TP=N | CP=M" to the per-PE header so users can read the
(cp_rank, tp_rank) pair without inferring from position.
3. Fixes a head-assignment bug: _q_heads_for_pe / _kv_heads_for_pe /
_per_pe_bytes used to be called with the raw PE-in-group index,
which treated every PE as a distinct TP rank. Under cp_placement=pe
this gave wrong Q/KV head lists for PEs beyond the first TP group
(PE 2..7 with tp=2, cp=4 would ask for head slots 32..127 in a
32-head model). Now called with tp_rank = pe_id % tp, so all CP
ranks in the cube share the correct head split.
Verified via a headless matplotlib smoke test with CP=4/TP=2 on Qwen 3
8B: 8 PE patches + 1 cube patch → 5 distinct facecolors (cube + 4 CP
ranks), no errors.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
An "Apply memory-min" button was removed when the 3 latency-optimal
buttons were added. Consequence: after clicking any latency-optimal
button, the sidebar sliders drift from the memory-min caption and
there was no way to sync them back without restarting the app or
manually adjusting each slider.
Restore the button, positioned below the caption and above the 3
latency-optimal buttons. Clicking it snaps cp/tp/pp/dp/cp_placement
to what the caption shows and clears _pl_active_scope to "full" so
the Physical Layout tab stops filtering.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Was 512 GB/s in the analytical viz MachineParams default. Bringing the
default down to 128 GB/s to match what the modelled physical link now
represents (parity with inter-cube D2D and topology.yaml).
Files touched (all three needed to keep defaults coherent):
- model_config.py: MachineParams.bw_intra_gbs = 128.0 (was 512.0)
- app.py: sidebar selectbox default index = 0 (128 GB/s) instead of 2 (512)
- auto_hardware.py: _HW_KNOB_DEFAULTS + BALANCED + COARSE bw_intra_gbs
baselines shifted so cost_score at defaults remains 6.0 and the
sensitivity sweep starts from the new baseline.
Verified: 24 pytest tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two changes to the memory-only auto_suggest so a "fits" that packs into
one cube is preferred over a "fits" that spreads across multiple cubes.
Suggestion dataclass gains cubes_used and cp_placement fields.
_score_candidate: for each (CP,TP,PP) triple, try cp_placement="cube"
(historical default: CP spans cubes) and, when CP·TP fits in one cube's
PE count, also cp_placement="pe" (pack CP into intra-cube PEs). Keep
the placement with fewer cubes; break ties toward "cube".
auto_suggest: switch sort key from (pes_used ↑, pp ↑, tp ↑, cp ↑) to
(cubes_used ↑, pes_used ↑, pp ↑, tp ↑, cp ↑). Fewer cubes wins first
because a cube is the physical die-level unit; PE count is the
tiebreaker.
Sidebar caption now also displays cubes_used + cp_placement so the
user can see the packed layout at a glance. Preset-change auto-reset
also applies the picked cp_placement.
Observed effect (verified):
Qwen 3 8B / 128K decode: CP=4 TP=2, "pe" → 1 cube, 8 PEs
(was 4 cubes with default "cube")
Llama 3.1 70B / 128K: CP=4 TP=16, "cube" → 8 cubes (unchanged;
TP=16 > 8 PEs/cube, can't pack)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New tab order:
Physical layout / Auto Suggest Parallelism / Memory breakdown /
Per-stage latency / Save & compare / Auto Hardware
Physical layout is what a user typically wants to see first when
loading the app, so it takes the leftmost slot again. Auto Suggest
Parallelism moves to second — still prominent, still usable as a
first step, but doesn't push the layout view behind it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the single memory-only "Apply auto-suggest" button in the
sidebar Parallelism expander with three latency-optimal buttons:
"Attention", "FFN/MoE", "Attn+FFN".
Each clicked button runs run_auto_explore for its scope, picks the
latency-minimum Pareto config, snaps the parallelism knobs to the
sidebar's selectbox option sets (CP/TP/PP/DP), and loads the other
knobs directly (tp_placement, cp_placement, cp_ring_variant, kv_mode,
ffn_scope_label). Also sets _pl_active_scope so the Physical Layout
tab's stage-table filter follows the scope automatically.
The caption above the buttons still shows the memory-only autosuggest
values as a reference — separately labeled "(memory-min)" to avoid
confusion with the latency-optimal buttons below.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Clicking one of the three Physical Layout tab buttons now persists the
chosen scope in session_state["_pl_active_scope"] and filters the
per-stage latency tables accordingly:
- attn scope → show only "Attention" stage table
- ffn scope → show only "FFN" stage table
- full scope → show both (default; also matches a fresh sidebar-driven
config with no button click yet)
Added a "Layout scope: {label}" header so the user can tell at a glance
which scope's config is loaded.
The pipeline diagram + topology map + weight/tensor sharding + PE
layout continue to show the full model layout — those diagrams give
context that stays useful even when the user is focusing on attn or
FFN. Deeper filtering (e.g., attention-only pipeline stripe) can come
later if needed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Streamlit's hot-reload re-executes app.py on every interaction, but
sub-modules imported from app.py stay cached in sys.modules across
reruns. When we edit auto_explore.py or auto_hardware.py while
Streamlit is alive, the app keeps using the pre-edit version until a
full Ctrl+C + restart — leading to confusing "unexpected keyword
argument" errors after a signature change.
Force importlib.reload() on our own modules at the top of app.py so
future signature changes land without a full restart. Only reloads if
the module is already in sys.modules (first run just imports normally).
Applied to:
- tests.analytical_visualization.auto_explore
- tests.analytical_visualization.auto_hardware
- tests.analytical_visualization.autosuggest
- tests.analytical_visualization.stage_latencies
- tests.analytical_visualization.memory_layout
Third-party modules (streamlit, matplotlib, pandas, numpy) NOT reloaded
— unnecessary and slow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Shortcut: click any of {Attention, FFN/MoE, Attn+FFN/MoE} → runs
run_auto_explore for that scope, picks the latency-minimum Pareto config,
loads it into the sidebar's session_state (cp, tp, pp, dp, tp_placement,
cp_placement, cp_ring_variant, kv_mode, ffn_scope_label), then st.rerun().
Because the Physical Layout tab reads model + machine + cfg from the
sidebar, the sidebar update automatically redraws the pipeline diagram,
per-stage table, and everything below. No preview / parallel-display
state — WYSIWYG.
Ffn_scope_label reconstruction handles the dynamic "(div=…)" suffix the
sidebar shows (same pattern as auto_explore + auto_hardware tabs' "Load
into sidebar" widgets).
Verified: app.py parses; 24 pytest tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both auto tabs now offer three sweep scopes via three buttons instead
of two:
- Run sweep — Attention → include_attention=T, include_ffn=F
- Run sweep — FFN/MoE → include_attention=F, include_ffn=T
- Run sweep — Attn + FFN/MoE → include_attention=T, include_ffn=T
Each button caches its result under its own session_state key; the most
recently clicked button drives the display. All three caches persist so
users can flip between scopes without re-running.
Core changes:
auto_explore.py + auto_hardware.py:
- New include_attention: bool = True param alongside include_ffn
- _sum_visible_latency, _efficiency, score_config, run_auto_explore,
compute_parallelism_sensitivity, joint_explore, compute_sensitivity,
_best_parallelism_for_hw, _best_parallelism_two_stage all wired.
- Attention and FFN are additive with no overlap: measured 7.35 ms
(attn) + 5.50 ms (ffn) = 12.85 ms (full) for Llama 70B decode 128K.
Bug fix (drive-by): the display block in both tab render functions was
incorrectly nested inside the "cached ctx is stale" warning branch, so
metrics/scatter/table/sensitivity/load only rendered when the cache
was stale. Un-indented and removed the dead trailing else-info block.
Verified:
- 24 pytest tests pass (added 1 new for FFN-only scope invariants)
- Smoke: Llama 70B decode 128K all three scopes produce sensible Pareto:
* Attention only: 7.35 ms (14 pareto configs)
* FFN / MoE only: 5.50 ms (34 pareto configs)
* Attn + FFN/MoE: 12.85 ms (6 pareto configs)
Sums add up exactly, confirming no overlap in stage summation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both auto tabs now accept a scope choice at run time:
- "Run sweep — Attention" → include_ffn=False
- "Run sweep — Attn + FFN/MoE" → include_ffn=True
Each button runs an independent sweep and caches its result under its
own session_state key. The most recently clicked button determines the
displayed view; both caches persist so users can flip between the two
scopes without re-running.
Tabs:
- Renamed "Auto Explore" → "Auto Suggest Parallelism"
(accurately reflects that it only varies parallelism knobs; HW is
held at the sidebar values).
- "Auto Hardware" tab unchanged.
- Still 6 top-level tabs; no additional tabs added.
Core changes:
auto_explore.py:
- New include_ffn: bool = True parameter on _sum_visible_latency,
_efficiency, score_config, run_auto_explore, compute_parallelism_
sensitivity. False drops all FFN stages from the summed latency.
auto_hardware.py:
- New include_ffn: bool = True parameter on joint_explore,
compute_sensitivity, _best_parallelism_for_hw, _best_parallelism_
two_stage. Forwards to score_config.
Both defaults keep existing tests byte-identical.
Verified:
- 23 pytest tests pass (added 3 new: attn-only latency lower, attn-only
Pareto non-empty, joint HW attn-only faster than full).
- Smoke: Llama 70B decode 128K:
* Attn+FFN best latency: 12.85 ms (unchanged)
* Attention-only best: 7.35 ms (~57% of full)
* Both sensitivities top-rank bw_hbm_gbs (physics preserved).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds compute_parallelism_sensitivity() + ParallelismSensitivityRow to
auto_explore.py. For a baseline ConfigScore (picked from the Pareto set),
sweeps each parallelism knob (CP, TP, PP, DP, EP) individually while
holding others fixed. Reports latency + memory-fit per value.
Sweep values start at 1 and step by 2 (multiples of 2 rather than only
powers of 2), giving finer granularity for the visual than the enumerator's
sparser set. CP goes up to 256 (per real-deployment scale), TP to 64,
PP to 32.
New UI panel in Auto Explore tab: 5 subplots (log/log), one per knob:
- Solid line = latency where the config fits memory
- Red X markers = infeasible (out of budget)
- Dotted vertical line = baseline value
User picks which Pareto row is the "baseline" via a number input; the
sensitivity chart re-computes around it.
Behaviour caveat noted during verification: for large PP the FFN AR can
cross a SIP boundary (uses stage_latencies.py:730 sips_used tier
selection), producing a step-up in latency. This is the existing model's
choice, honestly reflected in the chart. Whether the FFN AR should span
only one PP stage's ranks (thus not cross SIPs) is a separate discussion
about stage_latencies.py.
Verified:
- 11 pytest tests pass (added 2 new for parallelism sensitivity)
- Smoke: at Llama 70B decode 128K with baseline CP=8/TP=16/PP=1/DP=1:
* CP sweep: 4 fits, 8 = baseline optimum, 16+ overshoots memory
* TP sweep: 8/16 fit, 16 = baseline optimum, 32 slower (spans SIPs)
* PP sweep: 1 = baseline, 2+ slower due to sips_used tier drop for FFN AR
* DP sweep: same shape as PP
* EP sweep: monotone decreasing (bigger EP = smaller per-PE FFN)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Consumes auto_hardware.joint_explore(). UI:
- Sweep-depth radio: two_stage / balanced (default) / coarse
- Run joint sweep button + spinner
- 4 metric cards: HW candidates, feasible joint, Pareto count,
best latency
- Panel 1 (Pareto scatter): latency vs hardware cost proxy, feasible
in grey, Pareto colored by PE count, dashed line connects Pareto in
cost order — this is the "optimal HW config for the model" plot
- Panel 2 (sensitivity bar chart): per-knob relative speedup when
doubled from the baseline. Green bars, sorted biggest first;
annotated with the baseline → doubled values. Answers "where to
invest next?"
- Panel 3 (table): sortable Pareto joint configs with parallelism +
HW spec columns (PE HBM, HBM BW, TFLOPs, PE↔PE, D2D, C2C)
- Load into sidebar: picks a Pareto row and syncs both the HW
selectboxes (Per-PE + Interconnect sub-tabs) AND the parallelism
sliders. Values only get loaded when the target is one of the
selectbox options; otherwise the sidebar keeps its current value.
Session-state cache under _hw_result keyed by (model, s_kv, mode, depth);
if any of these drift, a warning suggests refreshing.
Verified: app.py parses; auto_hardware smoke run on Llama 70B decode
128K in ~20s produces sensible HW co-design signal (HBM BW 33% speedup,
everything else <1.5%).
Next: verification across Qwen 3 8B, Mixtral 8x7B (Commit 6).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New module extends auto_explore into the hardware co-design space. For a
fixed model + workload, sweeps hardware knobs (pe_hbm_gb, bw_hbm_gbs,
peak_tflops_f16, bw_intra_gbs, bw_inter_gbs, bw_intersip_gbs) and, for
each hardware candidate, searches parallelism for the latency-minimum
that fits memory. Returns:
- all_scores: every (hw, parallelism) pair that fits, sorted by latency
- pareto_scores: 2D Pareto frontier on (latency ↓, cost_score ↓)
- sensitivity: per-knob rel_speedup when doubled from the best-fast HW
baseline. Ranks which HW knob gives the biggest speedup — a co-design
signal.
Three sweep depths trade coverage for time:
- two_stage: 1 HW candidate (defaults) × autosuggest's memory-min
parallelism. Fast (~1s), useful for the sensitivity ranking alone.
- balanced: 64 HW × ~2k reduced-parallelism configs = ~130k joint evals,
~10-20s. Default UI setting.
- coarse: 729 HW × ~2k configs = ~1.4M joint evals, ~2-5 min.
Reduced parallelism sweep for the inner loop: CP × TP × PP × DP ×
kv_shard_mode (1,920 configs), other 4 knobs held at latency-friendly
defaults (ffn_shard_scope='TP+CP', tp_placement='cube', cp_placement='pe',
cp_ring_variant='qoml' for decode, 'kv' for prefill). Full 28,800-config
auto_explore per HW would take 6+ minutes — too slow.
Cost proxy: sum of (knob / knob_default). 6.0 at defaults. Not dollars —
a rough capability score where higher = "more spec'd hardware".
Verified:
- 9 pytest tests pass:
* enumeration counts match expected (1, 64, 729)
* default cost_score = 6.0
* Pareto non-dominated + subset of all_scores
* every knob is monotone-non-worsening when doubled
* for Llama 70B decode, bw_hbm_gbs tops the sensitivity ranking
(physically correct: memory-bound workload)
- Smoke: Llama 70B decode 128K balanced sweep in ~20s produces
5 Pareto configs; best 7.57 ms with 1024 GB/s HBM BW. Doubling
HBM BW gives 33% additional speedup; every other knob < 1.5%.
Next: Streamlit UI tab consuming this in Commit 5.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New tab consumes auto_explore.run_auto_explore(). UI:
- Header showing current model + workload + per-PE HBM budget
- Run sweep button (spinner while ~28k configs run in ~5-10s)
- 4 metric cards: enumerated, feasible, pareto count, best latency
- 2-panel scatter:
* latency vs PEs (feasible in grey, Pareto coloured by efficiency,
dashed line connects Pareto in PE order to show the trade-off curve)
* HBM utilization vs latency for Pareto, coloured by PE count
- Sortable Pareto table
- "Load into sidebar" widget: pick a row, sets the sidebar
session_state keys (cp, tp, pp, dp, tp_placement, cp_placement,
cp_ring_variant, kv_mode, ffn_scope_label) and st.rerun()s so the
user can flip to another tab and see the full breakdown.
Session-state caches the last sweep result under _auto_explore_result;
if the model/workload/HBM change without re-running, a warning suggests
refreshing.
Ffn_scope_label mapping is dynamic (contains substituted divisors), so
the Load button reconstructs the exact label using the target
cp/tp/dp values before assigning to session_state["ffn_scope_label"].
Verified:
- app.py parses cleanly
- All 9 pytest tests in test_auto_explore.py still pass
- Smoke: matplotlib + pandas + auto_explore imports round-trip
Next: verification pass across Qwen 3 8B and Mixtral 8x7B; polish.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extends the memory-only autosuggest to search the full 9-dimensional
parallelism space (CP, TP, PP, DP, kv_shard_mode, ffn_shard_scope,
tp_placement, cp_placement, cp_ring_variant) and rank feasible configs
on a 3D Pareto frontier: (latency ↓, pes_used ↓, efficiency ↑).
Throughput is stored on ConfigScore for display but is deliberately NOT
a Pareto axis because for a single-request analysis it collapses to
1 / latency, which would collapse the frontier.
Reuses existing physics (stage_latencies.all_stages + all_ffn_stages,
memory_layout.compute_memory) — no new formulas.
Single-request latency formula fix: PP does NOT reduce single-request
decode/prefill latency because the request has to traverse every layer
sequentially regardless of pipeline depth. The initial version had
latency ~ per_layer × layers_per_stage, which incorrectly rewarded high
PP. Corrected to latency ~ per_layer × model.layers.
Enumerator prunes:
- PP > model.layers
- TP > 4 × h_q
- ffn_shard_scope contains 'DP' when dp=1 (redundant)
- cp_ring_variant='qoml' when cp=1 (no-op)
Full sweep on Llama 3.1 70B: ~28,800 configs enumerated in ~7s, ~7k-10k
feasible (varies with S_kv), 2-7 unique Pareto configs. Faster context
lengths produce richer frontiers; at 1M, memory forces a single
dominant config (128 PEs, HBM 85%).
Verified:
- 9 pytest tests pass (enumerate, score, Pareto, subset invariants)
- Manual: Llama 70B decode at 8K/64K/128K/1M produces physically
sensible Pareto (CP=8/TP=16 wins latency; smaller-PE options
appear at longer context up to memory limits)
Next: Streamlit tab UI in app.py + verification against more presets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New tests/analytical_visualization/ module - an interactive dashboard
for exploring memory / latency tradeoffs of transformer inference on
the SIP architecture.
Highlights:
- 30+ model presets (Qwen, Llama 2/3/3.1, Mistral, Gemma 2, Phi 3,
DeepSeek MLA, Mixtral/Qwen 3 MoE, Grok-1, ...)
- Placement toggles: TP and CP each on PE-level vs cube-level
- CP ring variant: K/V ring vs Q+O/m/l ring (prefill); in decode the
O/m/l all-reduce is folded into S8 (no separate C1 row)
- SIP interconnect: ring / mesh2d / torus2d with matching link drawing
- Per-stage latency table with compute + memory + comm formulas,
auto-scaled ns/us/ms, colored by dominant bound
- Ring attention loop indicator on the pipeline diagram (purple arc
over S5-S8 with 'xN hops' badge)
- Tensor sharding view with optional physical PE/cube annotations
- Replication-waste + optimization-hints panel
- Save & compare configurations (config1, config2, ...): summary table
plus side-by-side per-stage attention and FFN latency, best-in-row
highlighting
- Symbol glossary with current values for every symbol used in formulas
Not tied to production sim_engine or runtime API; purely analytical
tooling for design-space exploration.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mukesh