Replace the three memory-optimal Pareto-search buttons in the Parallelism
sidebar with three scope-aware memory-min autosuggest buttons. Each
button sizes for a different per-PE memory footprint:
- Attn -> attention weights + KV cache (no FFN weights)
- FFN/MoE -> FFN weights only (no attention, no KV)
- Attn+FFN -> everything (traditional autosuggest)
memory_layout: per_pe_weight_bytes and compute_memory take
include_attention / include_ffn flags; KV cache is zeroed when attention
scope is excluded.
autosuggest: _score_candidate and auto_suggest forward the flags into
compute_memory, so the smallest-fit search now respects the chosen
scope.
app.py: single "Apply memory-min" caption + 3 column buttons. Each
button runs auto_suggest with its scope filter, snaps the sliders to
the picked (CP, TP, PP), and sets the Physical Layout scope filter to
match. Removes the standalone Apply memory-min button (was duplicating
the Attn+FFN case) and the Pareto-search import.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two related fixes.
1) Formula strings show B alongside every other symbol. Previously the
numeric FLOPs/mem_bytes/comm_bytes were scaled by B, but the printed
formula strings still read like B=1 amounts. That made the per-stage
table confusing at high B: numbers moved but formulas didn't. Updated
every attention + FFN stage's `formula` / `flops_formula` /
`mem_formula` / `comm_formula` to include B explicitly. Weight-bytes
lines now say "(shared across batch)" / "(weight, B-invariant)" so it's
obvious what does and doesn't scale.
Stages touched: S1..S10 (attention), C1/C2/C3 (attn comm), F1..F5
(FFN), CF1 (FFN AR).
2) Renamed the button/spinner/help copy from "latency-optimal" to
"memory-optimal" to match the smallest-fit pick semantics I put in
place earlier. The buttons already picked smallest-fit; the labels
just still said "latency-optimal" from the previous iteration.
Left the Auto Hardware sensitivity chart's "latency-optimal baseline"
label alone — that panel is a HW co-design view where "how fast can
each HW go?" is the intended question.
Verified: 24 pytest tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New sidebar widget in the Workload expander:
Batch B (concurrent requests) — selectbox 1/2/4/…/256, default 1.
Wired end-to-end:
- app.py sidebar → forwards b_batch when constructing the main topo
- auto_suggest(..., b=1) — passes b to _score_candidate, which builds
TopologyConfig with b=b
- run_auto_explore(..., b=1) — sets topo.b = b for every enumerated
candidate before scoring
- joint_explore(..., b=1) — forwards b to _best_parallelism_for_hw
and _best_parallelism_two_stage; both set topo.b = b before scoring
All button handlers (sidebar Apply-scope, Physical Layout scope,
Auto Suggest tab sweep, Auto Hardware tab sweep) now pass b_batch.
Combined with the earlier partial-scaffold commits (memory_layout
scales KV by B; stage_rmsnorm / stage_wq / stage_wkv / stage_kv_append /
_per_hop_qkT_pv scale flops + activation memory by B), changing B in
the sidebar now affects reported latency and per-PE memory footprint
in the visible parts of the pipeline. The remaining FFN + comm-AR
stages still ignore B (they'll be next); their contribution is small
for the memory-bound decode case that matters most, but latency for
FFN-heavy configs at high B will be slightly under-reported until
those are scaled too.
Verified: 24 pytest tests pass at default B=1 (backward compat).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The reload guard didn't include model_config.py. When the batch (B)
field was added to TopologyConfig, Streamlit kept the stale
pre-B dataclass in sys.modules and every downstream module hit
"AttributeError: 'TopologyConfig' object has no attribute 'b'".
Fix: add model_config and model_presets to the reload list, and order
model_config FIRST so downstream modules that reload after it pick up
the new dataclass definition. Also reordered the rest so upstream
dependencies (autosuggest, memory_layout, stage_latencies) reload
before their consumers (auto_explore, auto_hardware).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a second table beneath the Pareto configurations table, sorted by
(hbm_utilization ↑, pes_used ↑, latency ↑) — top 10 smallest-memory
Pareto configs first.
Columns highlight the memory story:
HBM % | weights (GB) | KV (GB) | PEs | SIPs | lat (ms) | CP TP PP DP | kv | ffn | tp_place | cp_place
Useful when memory pressure is the real binding constraint of a
deployment — e.g., picking a config for a per-PE HBM-limited SIP,
or spotting configs that trade small latency headroom for large
memory savings.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sidebar Attention/FFN/Attn+FFN buttons and Physical Layout tab buttons
now select the SMALLEST-FIT config from the scope's Pareto set — key
changed from (latency ↑, pes_used ↑, hbm_utilization ↑) to (pes_used ↑,
hbm_utilization ↑, latency ↑). Answers "smallest deployment that's
still Pareto-optimal for this scope" instead of "fastest deployment
achievable".
Left unchanged:
- "Best latency" metric displays in Auto Suggest / Auto Hardware tabs
still show the fastest number (informational; the metric labels say
"Best latency").
- auto_hardware._best_parallelism_for_hw stays latency-min: it's
inside HW co-design, where "how fast can each HW candidate go" is
the primary question.
Verified: 24 pytest tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Every place that picks a single "best" config from the Pareto set now
uses a compound sort key: (latency_ns ↑, pes_used ↑, hbm_utilization ↑).
Primary = latency; when configs tie on latency (common — cp_ring
variants, some placement variants often produce identical numbers),
prefer smaller-footprint picks.
Places updated:
- app.py: sidebar Apply Attn/FFN/Attn+FFN button
- app.py: Physical Layout tab Attn/FFN/Attn+FFN button
- app.py: Auto Suggest tab "Best latency" metric
- app.py: Auto Hardware tab "Best latency" metric (uses parallelism.pes_used
+ parallelism.hbm_utilization since JointScore wraps ConfigScore)
- auto_hardware.py: _best_parallelism_for_hw iteration key
No behavior change when there's a strict latency winner. When there are
ties, the picked config uses fewer PEs and lower HBM utilization.
Verified: all 24 pytest tests pass (default include_attention=True and
include_ffn=True paths unchanged).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously only reloaded auto_explore / auto_hardware / autosuggest /
stage_latencies / memory_layout. When we edit pe_weight_layout,
tensor_sharding, topology_map, pipeline_diagram, or optimization_report,
Streamlit was still holding the pre-edit versions until a full restart.
Add all five to the reload list so any visualization-module edit lands
on the next Streamlit rerun without needing Ctrl+C.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
An "Apply memory-min" button was removed when the 3 latency-optimal
buttons were added. Consequence: after clicking any latency-optimal
button, the sidebar sliders drift from the memory-min caption and
there was no way to sync them back without restarting the app or
manually adjusting each slider.
Restore the button, positioned below the caption and above the 3
latency-optimal buttons. Clicking it snaps cp/tp/pp/dp/cp_placement
to what the caption shows and clears _pl_active_scope to "full" so
the Physical Layout tab stops filtering.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Was 512 GB/s in the analytical viz MachineParams default. Bringing the
default down to 128 GB/s to match what the modelled physical link now
represents (parity with inter-cube D2D and topology.yaml).
Files touched (all three needed to keep defaults coherent):
- model_config.py: MachineParams.bw_intra_gbs = 128.0 (was 512.0)
- app.py: sidebar selectbox default index = 0 (128 GB/s) instead of 2 (512)
- auto_hardware.py: _HW_KNOB_DEFAULTS + BALANCED + COARSE bw_intra_gbs
baselines shifted so cost_score at defaults remains 6.0 and the
sensitivity sweep starts from the new baseline.
Verified: 24 pytest tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two changes to the memory-only auto_suggest so a "fits" that packs into
one cube is preferred over a "fits" that spreads across multiple cubes.
Suggestion dataclass gains cubes_used and cp_placement fields.
_score_candidate: for each (CP,TP,PP) triple, try cp_placement="cube"
(historical default: CP spans cubes) and, when CP·TP fits in one cube's
PE count, also cp_placement="pe" (pack CP into intra-cube PEs). Keep
the placement with fewer cubes; break ties toward "cube".
auto_suggest: switch sort key from (pes_used ↑, pp ↑, tp ↑, cp ↑) to
(cubes_used ↑, pes_used ↑, pp ↑, tp ↑, cp ↑). Fewer cubes wins first
because a cube is the physical die-level unit; PE count is the
tiebreaker.
Sidebar caption now also displays cubes_used + cp_placement so the
user can see the packed layout at a glance. Preset-change auto-reset
also applies the picked cp_placement.
Observed effect (verified):
Qwen 3 8B / 128K decode: CP=4 TP=2, "pe" → 1 cube, 8 PEs
(was 4 cubes with default "cube")
Llama 3.1 70B / 128K: CP=4 TP=16, "cube" → 8 cubes (unchanged;
TP=16 > 8 PEs/cube, can't pack)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New tab order:
Physical layout / Auto Suggest Parallelism / Memory breakdown /
Per-stage latency / Save & compare / Auto Hardware
Physical layout is what a user typically wants to see first when
loading the app, so it takes the leftmost slot again. Auto Suggest
Parallelism moves to second — still prominent, still usable as a
first step, but doesn't push the layout view behind it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the single memory-only "Apply auto-suggest" button in the
sidebar Parallelism expander with three latency-optimal buttons:
"Attention", "FFN/MoE", "Attn+FFN".
Each clicked button runs run_auto_explore for its scope, picks the
latency-minimum Pareto config, snaps the parallelism knobs to the
sidebar's selectbox option sets (CP/TP/PP/DP), and loads the other
knobs directly (tp_placement, cp_placement, cp_ring_variant, kv_mode,
ffn_scope_label). Also sets _pl_active_scope so the Physical Layout
tab's stage-table filter follows the scope automatically.
The caption above the buttons still shows the memory-only autosuggest
values as a reference — separately labeled "(memory-min)" to avoid
confusion with the latency-optimal buttons below.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Clicking one of the three Physical Layout tab buttons now persists the
chosen scope in session_state["_pl_active_scope"] and filters the
per-stage latency tables accordingly:
- attn scope → show only "Attention" stage table
- ffn scope → show only "FFN" stage table
- full scope → show both (default; also matches a fresh sidebar-driven
config with no button click yet)
Added a "Layout scope: {label}" header so the user can tell at a glance
which scope's config is loaded.
The pipeline diagram + topology map + weight/tensor sharding + PE
layout continue to show the full model layout — those diagrams give
context that stays useful even when the user is focusing on attn or
FFN. Deeper filtering (e.g., attention-only pipeline stripe) can come
later if needed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Streamlit's hot-reload re-executes app.py on every interaction, but
sub-modules imported from app.py stay cached in sys.modules across
reruns. When we edit auto_explore.py or auto_hardware.py while
Streamlit is alive, the app keeps using the pre-edit version until a
full Ctrl+C + restart — leading to confusing "unexpected keyword
argument" errors after a signature change.
Force importlib.reload() on our own modules at the top of app.py so
future signature changes land without a full restart. Only reloads if
the module is already in sys.modules (first run just imports normally).
Applied to:
- tests.analytical_visualization.auto_explore
- tests.analytical_visualization.auto_hardware
- tests.analytical_visualization.autosuggest
- tests.analytical_visualization.stage_latencies
- tests.analytical_visualization.memory_layout
Third-party modules (streamlit, matplotlib, pandas, numpy) NOT reloaded
— unnecessary and slow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Shortcut: click any of {Attention, FFN/MoE, Attn+FFN/MoE} → runs
run_auto_explore for that scope, picks the latency-minimum Pareto config,
loads it into the sidebar's session_state (cp, tp, pp, dp, tp_placement,
cp_placement, cp_ring_variant, kv_mode, ffn_scope_label), then st.rerun().
Because the Physical Layout tab reads model + machine + cfg from the
sidebar, the sidebar update automatically redraws the pipeline diagram,
per-stage table, and everything below. No preview / parallel-display
state — WYSIWYG.
Ffn_scope_label reconstruction handles the dynamic "(div=…)" suffix the
sidebar shows (same pattern as auto_explore + auto_hardware tabs' "Load
into sidebar" widgets).
Verified: app.py parses; 24 pytest tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both auto tabs now offer three sweep scopes via three buttons instead
of two:
- Run sweep — Attention → include_attention=T, include_ffn=F
- Run sweep — FFN/MoE → include_attention=F, include_ffn=T
- Run sweep — Attn + FFN/MoE → include_attention=T, include_ffn=T
Each button caches its result under its own session_state key; the most
recently clicked button drives the display. All three caches persist so
users can flip between scopes without re-running.
Core changes:
auto_explore.py + auto_hardware.py:
- New include_attention: bool = True param alongside include_ffn
- _sum_visible_latency, _efficiency, score_config, run_auto_explore,
compute_parallelism_sensitivity, joint_explore, compute_sensitivity,
_best_parallelism_for_hw, _best_parallelism_two_stage all wired.
- Attention and FFN are additive with no overlap: measured 7.35 ms
(attn) + 5.50 ms (ffn) = 12.85 ms (full) for Llama 70B decode 128K.
Bug fix (drive-by): the display block in both tab render functions was
incorrectly nested inside the "cached ctx is stale" warning branch, so
metrics/scatter/table/sensitivity/load only rendered when the cache
was stale. Un-indented and removed the dead trailing else-info block.
Verified:
- 24 pytest tests pass (added 1 new for FFN-only scope invariants)
- Smoke: Llama 70B decode 128K all three scopes produce sensible Pareto:
* Attention only: 7.35 ms (14 pareto configs)
* FFN / MoE only: 5.50 ms (34 pareto configs)
* Attn + FFN/MoE: 12.85 ms (6 pareto configs)
Sums add up exactly, confirming no overlap in stage summation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both auto tabs now accept a scope choice at run time:
- "Run sweep — Attention" → include_ffn=False
- "Run sweep — Attn + FFN/MoE" → include_ffn=True
Each button runs an independent sweep and caches its result under its
own session_state key. The most recently clicked button determines the
displayed view; both caches persist so users can flip between the two
scopes without re-running.
Tabs:
- Renamed "Auto Explore" → "Auto Suggest Parallelism"
(accurately reflects that it only varies parallelism knobs; HW is
held at the sidebar values).
- "Auto Hardware" tab unchanged.
- Still 6 top-level tabs; no additional tabs added.
Core changes:
auto_explore.py:
- New include_ffn: bool = True parameter on _sum_visible_latency,
_efficiency, score_config, run_auto_explore, compute_parallelism_
sensitivity. False drops all FFN stages from the summed latency.
auto_hardware.py:
- New include_ffn: bool = True parameter on joint_explore,
compute_sensitivity, _best_parallelism_for_hw, _best_parallelism_
two_stage. Forwards to score_config.
Both defaults keep existing tests byte-identical.
Verified:
- 23 pytest tests pass (added 3 new: attn-only latency lower, attn-only
Pareto non-empty, joint HW attn-only faster than full).
- Smoke: Llama 70B decode 128K:
* Attn+FFN best latency: 12.85 ms (unchanged)
* Attention-only best: 7.35 ms (~57% of full)
* Both sensitivities top-rank bw_hbm_gbs (physics preserved).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds compute_parallelism_sensitivity() + ParallelismSensitivityRow to
auto_explore.py. For a baseline ConfigScore (picked from the Pareto set),
sweeps each parallelism knob (CP, TP, PP, DP, EP) individually while
holding others fixed. Reports latency + memory-fit per value.
Sweep values start at 1 and step by 2 (multiples of 2 rather than only
powers of 2), giving finer granularity for the visual than the enumerator's
sparser set. CP goes up to 256 (per real-deployment scale), TP to 64,
PP to 32.
New UI panel in Auto Explore tab: 5 subplots (log/log), one per knob:
- Solid line = latency where the config fits memory
- Red X markers = infeasible (out of budget)
- Dotted vertical line = baseline value
User picks which Pareto row is the "baseline" via a number input; the
sensitivity chart re-computes around it.
Behaviour caveat noted during verification: for large PP the FFN AR can
cross a SIP boundary (uses stage_latencies.py:730 sips_used tier
selection), producing a step-up in latency. This is the existing model's
choice, honestly reflected in the chart. Whether the FFN AR should span
only one PP stage's ranks (thus not cross SIPs) is a separate discussion
about stage_latencies.py.
Verified:
- 11 pytest tests pass (added 2 new for parallelism sensitivity)
- Smoke: at Llama 70B decode 128K with baseline CP=8/TP=16/PP=1/DP=1:
* CP sweep: 4 fits, 8 = baseline optimum, 16+ overshoots memory
* TP sweep: 8/16 fit, 16 = baseline optimum, 32 slower (spans SIPs)
* PP sweep: 1 = baseline, 2+ slower due to sips_used tier drop for FFN AR
* DP sweep: same shape as PP
* EP sweep: monotone decreasing (bigger EP = smaller per-PE FFN)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Consumes auto_hardware.joint_explore(). UI:
- Sweep-depth radio: two_stage / balanced (default) / coarse
- Run joint sweep button + spinner
- 4 metric cards: HW candidates, feasible joint, Pareto count,
best latency
- Panel 1 (Pareto scatter): latency vs hardware cost proxy, feasible
in grey, Pareto colored by PE count, dashed line connects Pareto in
cost order — this is the "optimal HW config for the model" plot
- Panel 2 (sensitivity bar chart): per-knob relative speedup when
doubled from the baseline. Green bars, sorted biggest first;
annotated with the baseline → doubled values. Answers "where to
invest next?"
- Panel 3 (table): sortable Pareto joint configs with parallelism +
HW spec columns (PE HBM, HBM BW, TFLOPs, PE↔PE, D2D, C2C)
- Load into sidebar: picks a Pareto row and syncs both the HW
selectboxes (Per-PE + Interconnect sub-tabs) AND the parallelism
sliders. Values only get loaded when the target is one of the
selectbox options; otherwise the sidebar keeps its current value.
Session-state cache under _hw_result keyed by (model, s_kv, mode, depth);
if any of these drift, a warning suggests refreshing.
Verified: app.py parses; auto_hardware smoke run on Llama 70B decode
128K in ~20s produces sensible HW co-design signal (HBM BW 33% speedup,
everything else <1.5%).
Next: verification across Qwen 3 8B, Mixtral 8x7B (Commit 6).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New tab consumes auto_explore.run_auto_explore(). UI:
- Header showing current model + workload + per-PE HBM budget
- Run sweep button (spinner while ~28k configs run in ~5-10s)
- 4 metric cards: enumerated, feasible, pareto count, best latency
- 2-panel scatter:
* latency vs PEs (feasible in grey, Pareto coloured by efficiency,
dashed line connects Pareto in PE order to show the trade-off curve)
* HBM utilization vs latency for Pareto, coloured by PE count
- Sortable Pareto table
- "Load into sidebar" widget: pick a row, sets the sidebar
session_state keys (cp, tp, pp, dp, tp_placement, cp_placement,
cp_ring_variant, kv_mode, ffn_scope_label) and st.rerun()s so the
user can flip to another tab and see the full breakdown.
Session-state caches the last sweep result under _auto_explore_result;
if the model/workload/HBM change without re-running, a warning suggests
refreshing.
Ffn_scope_label mapping is dynamic (contains substituted divisors), so
the Load button reconstructs the exact label using the target
cp/tp/dp values before assigning to session_state["ffn_scope_label"].
Verified:
- app.py parses cleanly
- All 9 pytest tests in test_auto_explore.py still pass
- Smoke: matplotlib + pandas + auto_explore imports round-trip
Next: verification pass across Qwen 3 8B and Mixtral 8x7B; polish.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New tests/analytical_visualization/ module - an interactive dashboard
for exploring memory / latency tradeoffs of transformer inference on
the SIP architecture.
Highlights:
- 30+ model presets (Qwen, Llama 2/3/3.1, Mistral, Gemma 2, Phi 3,
DeepSeek MLA, Mixtral/Qwen 3 MoE, Grok-1, ...)
- Placement toggles: TP and CP each on PE-level vs cube-level
- CP ring variant: K/V ring vs Q+O/m/l ring (prefill); in decode the
O/m/l all-reduce is folded into S8 (no separate C1 row)
- SIP interconnect: ring / mesh2d / torus2d with matching link drawing
- Per-stage latency table with compute + memory + comm formulas,
auto-scaled ns/us/ms, colored by dominant bound
- Ring attention loop indicator on the pipeline diagram (purple arc
over S5-S8 with 'xN hops' badge)
- Tensor sharding view with optional physical PE/cube annotations
- Replication-waste + optimization-hints panel
- Save & compare configurations (config1, config2, ...): summary table
plus side-by-side per-stage attention and FFN latency, best-in-row
highlighting
- Symbol glossary with current values for every symbol used in formulas
Not tied to production sim_engine or runtime API; purely analytical
tooling for design-space exploration.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mukesh