Move the per-stage shape tables (attention + FFN) off the Per-stage
latency tab and combine them with the weight + KV shape tables into a
single 'All tensor shapes (per PE)' expander at the bottom of the
Physical Layout tab. Five subsections in one place:
- Attention weights (global + per-PE shape, per layer)
- FFN weights (same schema)
- KV cache (same schema)
- Per-stage attention shapes (input / weight / output)
- Per-stage FFN shapes (same)
Rationale: the shape catalog belongs next to the physical-layout view
that shows how the tensors are placed on hardware. The Per-stage
latency tab now focuses purely on time-per-stage. Weight/KV tables
stay on the Memory Breakdown tab too because the summary metrics
underneath still consume the same rows.
Guard test verifies the section string appears exactly once, is
positioned inside tab_layout (before tab_memory / tab_stages), and
that neither attn_stage_shape_rows nor ffn_stage_shape_rows is called
inside the tab_stages block anymore.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New stage_shapes module reports per-PE input / weight / output tensor
shapes for every attention (S1..S10, C1..C3) and FFN (F1..F5, CF1)
stage. Streamlit's Stages tab picks these up below the existing
latency table + bar chart as two separate dataframes ('Attention
stages' and 'FFN stages'), so anyone who wants to see the exact
sharded tensor dims per stage can, without cluttering the latency
view.
Rows mirror the conditional structure of all_stages / all_ffn_stages:
S8 merge only when CP>1; C1 CP ring only in prefill with CP>1; C3
Score AR only when kv_shard_mode='split' and TP>H_kv; C2 TP AR only
when TP>1; CF1 FFN AR only when the FFN divisor > 1. Batch B flows
into every activation dim; weights stay B-invariant.
test_stage_shapes covers column presence, conditional row insertion
(decode omits C1; CP=1 omits S8; prefill+CP>1 inserts C1), batch
scaling of the leading dim, and that S5 output references both T_q
and S_local for the current cfg.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
auto_suggest only explores (cp, tp, pp, cp_placement); every other
TopologyConfig field is left at its dataclass default when the score
is computed. The sidebar's memory-min apply used to update only
cp/tp/pp/dp/cp_placement, so any stale session_state value from a
previous user interaction (most damaging: tp_placement="cube") would
carry over into the rerun and produce a topology the memory-min search
never scored — visibly, cubes_used could balloon (e.g. Llama 3 70B
Attn scope: auto_suggest picks 1 cube; stale tp_placement="cube" made
the applied config span 8 cubes).
Fix: after picking a Suggestion, reset tp_placement, kv_mode,
cp_ring_variant, ep back to their TopologyConfig defaults, and drop
the ffn_scope_label key (dynamic string; falls back to the TP+CP
default index on re-render). Now the sliders + placement + mode state
after apply exactly reproduce the topology auto_suggest scored.
test_auto_suggest_cubes_match_default_topology asserts, across four
model/skv combinations and all three scopes, that a TopologyConfig
built from (cp, tp, pp, cp_placement) plus dataclass defaults has the
same cubes_used the Suggestion reports.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the three memory-optimal Pareto-search buttons in the Parallelism
sidebar with three scope-aware memory-min autosuggest buttons. Each
button sizes for a different per-PE memory footprint:
- Attn -> attention weights + KV cache (no FFN weights)
- FFN/MoE -> FFN weights only (no attention, no KV)
- Attn+FFN -> everything (traditional autosuggest)
memory_layout: per_pe_weight_bytes and compute_memory take
include_attention / include_ffn flags; KV cache is zeroed when attention
scope is excluded.
autosuggest: _score_candidate and auto_suggest forward the flags into
compute_memory, so the smallest-fit search now respects the chosen
scope.
app.py: single "Apply memory-min" caption + 3 column buttons. Each
button runs auto_suggest with its scope filter, snaps the sliders to
the picked (CP, TP, PP), and sets the Physical Layout scope filter to
match. Removes the standalone Apply memory-min button (was duplicating
the Attn+FFN case) and the Pareto-search import.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two related fixes.
1) Formula strings show B alongside every other symbol. Previously the
numeric FLOPs/mem_bytes/comm_bytes were scaled by B, but the printed
formula strings still read like B=1 amounts. That made the per-stage
table confusing at high B: numbers moved but formulas didn't. Updated
every attention + FFN stage's `formula` / `flops_formula` /
`mem_formula` / `comm_formula` to include B explicitly. Weight-bytes
lines now say "(shared across batch)" / "(weight, B-invariant)" so it's
obvious what does and doesn't scale.
Stages touched: S1..S10 (attention), C1/C2/C3 (attn comm), F1..F5
(FFN), CF1 (FFN AR).
2) Renamed the button/spinner/help copy from "latency-optimal" to
"memory-optimal" to match the smallest-fit pick semantics I put in
place earlier. The buttons already picked smallest-fit; the labels
just still said "latency-optimal" from the previous iteration.
Left the Auto Hardware sensitivity chart's "latency-optimal baseline"
label alone — that panel is a HW co-design view where "how fast can
each HW go?" is the intended question.
Verified: 24 pytest tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Finishes the batch scaffold. All stages that produce per-request work
now scale flops / activation-memory / comm-bytes by B; weight bytes stay
fixed (shared across the batch).
Attention block, previously partial:
- S6 stage_softmax: elems + bytes × B
- S8 stage_merge: flops × B; O/m/l AR bytes M × B
- S9 stage_normalize: flops × B
- S10 stage_wo: flops × B
- C1 comm_cp_ring: decode O/m/l AR M × B; prefill K/V ring + Q+O/m/l
variants both × B
- C2 comm_tp_allreduce (W_O output): bytes × B
- C3 comm_kv_split_allreduce (head-split scores): bytes_per_hop × B
FFN block, previously untouched:
- F1 stage_ffn_rmsnorm: activation × B (weight fixed)
- F2 stage_ffn_gate (via _ffn_gemm): flops × B
- F3 stage_ffn_up: same
- F4 stage_ffn_swiglu: elems + flops × B
- F5 stage_ffn_down: flops × B
- CF1 comm_ffn_allreduce (batched FFN output): bytes × B
Verified with a smoke check on Qwen 3 8B / 128K decode:
B=1 per-layer visible latency: 363 us
B=8 716 us (sub-linear — many stages
stay weight-bound)
B=64 4023 us (approaching linear scaling
as batch dominates)
All 24 pytest tests still pass at default B=1 (backward compat).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three additions to the tensor sharding view when batch B > 1:
1. Shape label appends "B={B}": e.g. (128, S_kv=131072, B=8).
2. Note under the KV cache: "batch B={B} => {B}x KV bytes per PE".
3. Visual "stack" — up to 5 dashed offset rectangles drawn behind
the KV cache to hint at the batch stacking dimension. Capped at 5
so a big B doesn't overwhelm the diagram.
4. Title also gets B={B} between CP and FFN scope.
Attention/FFN weight tensors are NOT stacked — weights are shared
across the batch (correct: only activations + KV scale with B).
At B=1, all four additions are no-ops so the diagram looks unchanged
from before.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New sidebar widget in the Workload expander:
Batch B (concurrent requests) — selectbox 1/2/4/…/256, default 1.
Wired end-to-end:
- app.py sidebar → forwards b_batch when constructing the main topo
- auto_suggest(..., b=1) — passes b to _score_candidate, which builds
TopologyConfig with b=b
- run_auto_explore(..., b=1) — sets topo.b = b for every enumerated
candidate before scoring
- joint_explore(..., b=1) — forwards b to _best_parallelism_for_hw
and _best_parallelism_two_stage; both set topo.b = b before scoring
All button handlers (sidebar Apply-scope, Physical Layout scope,
Auto Suggest tab sweep, Auto Hardware tab sweep) now pass b_batch.
Combined with the earlier partial-scaffold commits (memory_layout
scales KV by B; stage_rmsnorm / stage_wq / stage_wkv / stage_kv_append /
_per_hop_qkT_pv scale flops + activation memory by B), changing B in
the sidebar now affects reported latency and per-PE memory footprint
in the visible parts of the pipeline. The remaining FFN + comm-AR
stages still ignore B (they'll be next); their contribution is small
for the memory-bound decode case that matters most, but latency for
FFN-heavy configs at high B will be slightly under-reported until
those are scaled too.
Verified: 24 pytest tests pass at default B=1 (backward compat).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The reload guard didn't include model_config.py. When the batch (B)
field was added to TopologyConfig, Streamlit kept the stale
pre-B dataclass in sys.modules and every downstream module hit
"AttributeError: 'TopologyConfig' object has no attribute 'b'".
Fix: add model_config and model_presets to the reload list, and order
model_config FIRST so downstream modules that reload after it pick up
the new dataclass definition. Also reordered the rest so upstream
dependencies (autosuggest, memory_layout, stage_latencies) reload
before their consumers (auto_explore, auto_hardware).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a second table beneath the Pareto configurations table, sorted by
(hbm_utilization ↑, pes_used ↑, latency ↑) — top 10 smallest-memory
Pareto configs first.
Columns highlight the memory story:
HBM % | weights (GB) | KV (GB) | PEs | SIPs | lat (ms) | CP TP PP DP | kv | ffn | tp_place | cp_place
Useful when memory pressure is the real binding constraint of a
deployment — e.g., picking a config for a per-PE HBM-limited SIP,
or spotting configs that trade small latency headroom for large
memory savings.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sidebar Attention/FFN/Attn+FFN buttons and Physical Layout tab buttons
now select the SMALLEST-FIT config from the scope's Pareto set — key
changed from (latency ↑, pes_used ↑, hbm_utilization ↑) to (pes_used ↑,
hbm_utilization ↑, latency ↑). Answers "smallest deployment that's
still Pareto-optimal for this scope" instead of "fastest deployment
achievable".
Left unchanged:
- "Best latency" metric displays in Auto Suggest / Auto Hardware tabs
still show the fastest number (informational; the metric labels say
"Best latency").
- auto_hardware._best_parallelism_for_hw stays latency-min: it's
inside HW co-design, where "how fast can each HW candidate go" is
the primary question.
Verified: 24 pytest tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
WIP toward batch-size support. This first commit is behavior-preserving
at the default B=1 (every new multiplication is by max(1, cfg.topo.b) =
1 today) so all existing tests pass. Follow-up commits will:
- scale the remaining stages (S6/S7/S8/S9/S10 + C1/C2/C3 + FFN + FFN AR)
- add a batch selectbox to the sidebar
- forward b through auto_suggest / auto_explore / auto_hardware
Changes so far:
- TopologyConfig: new b: int = 1 field (batch size).
- memory_layout.per_pe_kv_cache_bytes: * B (each concurrent request
keeps its own KV cache slice).
- stage_rmsnorm / stage_wq / stage_wkv / stage_kv_append /
_per_hop_qkT_pv: FLOPs and activation memory scaled by B; weight
bytes stay fixed (weights shared across the batch).
Verified: 24 pytest tests still pass at default B=1.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Every place that picks a single "best" config from the Pareto set now
uses a compound sort key: (latency_ns ↑, pes_used ↑, hbm_utilization ↑).
Primary = latency; when configs tie on latency (common — cp_ring
variants, some placement variants often produce identical numbers),
prefer smaller-footprint picks.
Places updated:
- app.py: sidebar Apply Attn/FFN/Attn+FFN button
- app.py: Physical Layout tab Attn/FFN/Attn+FFN button
- app.py: Auto Suggest tab "Best latency" metric
- app.py: Auto Hardware tab "Best latency" metric (uses parallelism.pes_used
+ parallelism.hbm_utilization since JointScore wraps ConfigScore)
- auto_hardware.py: _best_parallelism_for_hw iteration key
No behavior change when there's a strict latency winner. When there are
ties, the picked config uses fewer PEs and lower HBM utilization.
Verified: all 24 pytest tests pass (default include_attention=True and
include_ffn=True paths unchanged).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The topology map previously colored a whole cube (border + all PE
fills) by the single cp_rank assigned to that cube in the cube→(pp,cp)
mapping. Under cp_placement=cube this is right (each cube = one CP
rank). Under cp_placement=pe, however, multiple CP ranks are PACKED
into the same cube's PEs, so the whole-cube coloring makes every PE
look identical (defaulting to cp_rank=0's palette entry — bright red).
Fix: in the per-PE loop, if cp_packed = (cp_placement=="pe" and cp>1),
compute each PE's own cp_rank = pe_id // tp and look up its color via
_cp_color(pp_stage, pe_cp_rank, cp_size). Border still uses the
cube-level color (cp_rank=0), so the outer bounding box is unchanged,
but the interior PEs now show all four (or however many) CP-rank
colors visibly.
For cp_placement=cube: unchanged (single pe_fill per cube).
Verified with a headless render at Qwen 3 8B, CP=4, TP=2,
cp_placement=pe: 145 patches → 8 distinct facecolors (4 for the CP
ranks in the packed cube + inactive/border tints), where before it
was fewer distinct colors.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously only reloaded auto_explore / auto_hardware / autosuggest /
stage_latencies / memory_layout. When we edit pe_weight_layout,
tensor_sharding, topology_map, pipeline_diagram, or optimization_report,
Streamlit was still holding the pre-edit versions until a full restart.
Add all five to the reload list so any visualization-module edit lands
on the next Streamlit rerun without needing Ctrl+C.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When cp_placement=pe packs multiple CP ranks intra-cube, the PE-level
layout now:
1. Colors each PE's background by its CP rank (8-color palette wraps
for cp > 8). Cp_placement=cube keeps the historical light-blue
styling (only one CP rank per cube tile).
2. Adds "TP=N | CP=M" to the per-PE header so users can read the
(cp_rank, tp_rank) pair without inferring from position.
3. Fixes a head-assignment bug: _q_heads_for_pe / _kv_heads_for_pe /
_per_pe_bytes used to be called with the raw PE-in-group index,
which treated every PE as a distinct TP rank. Under cp_placement=pe
this gave wrong Q/KV head lists for PEs beyond the first TP group
(PE 2..7 with tp=2, cp=4 would ask for head slots 32..127 in a
32-head model). Now called with tp_rank = pe_id % tp, so all CP
ranks in the cube share the correct head split.
Verified via a headless matplotlib smoke test with CP=4/TP=2 on Qwen 3
8B: 8 PE patches + 1 cube patch → 5 distinct facecolors (cube + 4 CP
ranks), no errors.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
An "Apply memory-min" button was removed when the 3 latency-optimal
buttons were added. Consequence: after clicking any latency-optimal
button, the sidebar sliders drift from the memory-min caption and
there was no way to sync them back without restarting the app or
manually adjusting each slider.
Restore the button, positioned below the caption and above the 3
latency-optimal buttons. Clicking it snaps cp/tp/pp/dp/cp_placement
to what the caption shows and clears _pl_active_scope to "full" so
the Physical Layout tab stops filtering.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Was 512 GB/s in the analytical viz MachineParams default. Bringing the
default down to 128 GB/s to match what the modelled physical link now
represents (parity with inter-cube D2D and topology.yaml).
Files touched (all three needed to keep defaults coherent):
- model_config.py: MachineParams.bw_intra_gbs = 128.0 (was 512.0)
- app.py: sidebar selectbox default index = 0 (128 GB/s) instead of 2 (512)
- auto_hardware.py: _HW_KNOB_DEFAULTS + BALANCED + COARSE bw_intra_gbs
baselines shifted so cost_score at defaults remains 6.0 and the
sensitivity sweep starts from the new baseline.
Verified: 24 pytest tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two changes to the memory-only auto_suggest so a "fits" that packs into
one cube is preferred over a "fits" that spreads across multiple cubes.
Suggestion dataclass gains cubes_used and cp_placement fields.
_score_candidate: for each (CP,TP,PP) triple, try cp_placement="cube"
(historical default: CP spans cubes) and, when CP·TP fits in one cube's
PE count, also cp_placement="pe" (pack CP into intra-cube PEs). Keep
the placement with fewer cubes; break ties toward "cube".
auto_suggest: switch sort key from (pes_used ↑, pp ↑, tp ↑, cp ↑) to
(cubes_used ↑, pes_used ↑, pp ↑, tp ↑, cp ↑). Fewer cubes wins first
because a cube is the physical die-level unit; PE count is the
tiebreaker.
Sidebar caption now also displays cubes_used + cp_placement so the
user can see the packed layout at a glance. Preset-change auto-reset
also applies the picked cp_placement.
Observed effect (verified):
Qwen 3 8B / 128K decode: CP=4 TP=2, "pe" → 1 cube, 8 PEs
(was 4 cubes with default "cube")
Llama 3.1 70B / 128K: CP=4 TP=16, "cube" → 8 cubes (unchanged;
TP=16 > 8 PEs/cube, can't pack)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New tab order:
Physical layout / Auto Suggest Parallelism / Memory breakdown /
Per-stage latency / Save & compare / Auto Hardware
Physical layout is what a user typically wants to see first when
loading the app, so it takes the leftmost slot again. Auto Suggest
Parallelism moves to second — still prominent, still usable as a
first step, but doesn't push the layout view behind it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the single memory-only "Apply auto-suggest" button in the
sidebar Parallelism expander with three latency-optimal buttons:
"Attention", "FFN/MoE", "Attn+FFN".
Each clicked button runs run_auto_explore for its scope, picks the
latency-minimum Pareto config, snaps the parallelism knobs to the
sidebar's selectbox option sets (CP/TP/PP/DP), and loads the other
knobs directly (tp_placement, cp_placement, cp_ring_variant, kv_mode,
ffn_scope_label). Also sets _pl_active_scope so the Physical Layout
tab's stage-table filter follows the scope automatically.
The caption above the buttons still shows the memory-only autosuggest
values as a reference — separately labeled "(memory-min)" to avoid
confusion with the latency-optimal buttons below.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Clicking one of the three Physical Layout tab buttons now persists the
chosen scope in session_state["_pl_active_scope"] and filters the
per-stage latency tables accordingly:
- attn scope → show only "Attention" stage table
- ffn scope → show only "FFN" stage table
- full scope → show both (default; also matches a fresh sidebar-driven
config with no button click yet)
Added a "Layout scope: {label}" header so the user can tell at a glance
which scope's config is loaded.
The pipeline diagram + topology map + weight/tensor sharding + PE
layout continue to show the full model layout — those diagrams give
context that stays useful even when the user is focusing on attn or
FFN. Deeper filtering (e.g., attention-only pipeline stripe) can come
later if needed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Streamlit's hot-reload re-executes app.py on every interaction, but
sub-modules imported from app.py stay cached in sys.modules across
reruns. When we edit auto_explore.py or auto_hardware.py while
Streamlit is alive, the app keeps using the pre-edit version until a
full Ctrl+C + restart — leading to confusing "unexpected keyword
argument" errors after a signature change.
Force importlib.reload() on our own modules at the top of app.py so
future signature changes land without a full restart. Only reloads if
the module is already in sys.modules (first run just imports normally).
Applied to:
- tests.analytical_visualization.auto_explore
- tests.analytical_visualization.auto_hardware
- tests.analytical_visualization.autosuggest
- tests.analytical_visualization.stage_latencies
- tests.analytical_visualization.memory_layout
Third-party modules (streamlit, matplotlib, pandas, numpy) NOT reloaded
— unnecessary and slow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Shortcut: click any of {Attention, FFN/MoE, Attn+FFN/MoE} → runs
run_auto_explore for that scope, picks the latency-minimum Pareto config,
loads it into the sidebar's session_state (cp, tp, pp, dp, tp_placement,
cp_placement, cp_ring_variant, kv_mode, ffn_scope_label), then st.rerun().
Because the Physical Layout tab reads model + machine + cfg from the
sidebar, the sidebar update automatically redraws the pipeline diagram,
per-stage table, and everything below. No preview / parallel-display
state — WYSIWYG.
Ffn_scope_label reconstruction handles the dynamic "(div=…)" suffix the
sidebar shows (same pattern as auto_explore + auto_hardware tabs' "Load
into sidebar" widgets).
Verified: app.py parses; 24 pytest tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both auto tabs now offer three sweep scopes via three buttons instead
of two:
- Run sweep — Attention → include_attention=T, include_ffn=F
- Run sweep — FFN/MoE → include_attention=F, include_ffn=T
- Run sweep — Attn + FFN/MoE → include_attention=T, include_ffn=T
Each button caches its result under its own session_state key; the most
recently clicked button drives the display. All three caches persist so
users can flip between scopes without re-running.
Core changes:
auto_explore.py + auto_hardware.py:
- New include_attention: bool = True param alongside include_ffn
- _sum_visible_latency, _efficiency, score_config, run_auto_explore,
compute_parallelism_sensitivity, joint_explore, compute_sensitivity,
_best_parallelism_for_hw, _best_parallelism_two_stage all wired.
- Attention and FFN are additive with no overlap: measured 7.35 ms
(attn) + 5.50 ms (ffn) = 12.85 ms (full) for Llama 70B decode 128K.
Bug fix (drive-by): the display block in both tab render functions was
incorrectly nested inside the "cached ctx is stale" warning branch, so
metrics/scatter/table/sensitivity/load only rendered when the cache
was stale. Un-indented and removed the dead trailing else-info block.
Verified:
- 24 pytest tests pass (added 1 new for FFN-only scope invariants)
- Smoke: Llama 70B decode 128K all three scopes produce sensible Pareto:
* Attention only: 7.35 ms (14 pareto configs)
* FFN / MoE only: 5.50 ms (34 pareto configs)
* Attn + FFN/MoE: 12.85 ms (6 pareto configs)
Sums add up exactly, confirming no overlap in stage summation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both auto tabs now accept a scope choice at run time:
- "Run sweep — Attention" → include_ffn=False
- "Run sweep — Attn + FFN/MoE" → include_ffn=True
Each button runs an independent sweep and caches its result under its
own session_state key. The most recently clicked button determines the
displayed view; both caches persist so users can flip between the two
scopes without re-running.
Tabs:
- Renamed "Auto Explore" → "Auto Suggest Parallelism"
(accurately reflects that it only varies parallelism knobs; HW is
held at the sidebar values).
- "Auto Hardware" tab unchanged.
- Still 6 top-level tabs; no additional tabs added.
Core changes:
auto_explore.py:
- New include_ffn: bool = True parameter on _sum_visible_latency,
_efficiency, score_config, run_auto_explore, compute_parallelism_
sensitivity. False drops all FFN stages from the summed latency.
auto_hardware.py:
- New include_ffn: bool = True parameter on joint_explore,
compute_sensitivity, _best_parallelism_for_hw, _best_parallelism_
two_stage. Forwards to score_config.
Both defaults keep existing tests byte-identical.
Verified:
- 23 pytest tests pass (added 3 new: attn-only latency lower, attn-only
Pareto non-empty, joint HW attn-only faster than full).
- Smoke: Llama 70B decode 128K:
* Attn+FFN best latency: 12.85 ms (unchanged)
* Attention-only best: 7.35 ms (~57% of full)
* Both sensitivities top-rank bw_hbm_gbs (physics preserved).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds compute_parallelism_sensitivity() + ParallelismSensitivityRow to
auto_explore.py. For a baseline ConfigScore (picked from the Pareto set),
sweeps each parallelism knob (CP, TP, PP, DP, EP) individually while
holding others fixed. Reports latency + memory-fit per value.
Sweep values start at 1 and step by 2 (multiples of 2 rather than only
powers of 2), giving finer granularity for the visual than the enumerator's
sparser set. CP goes up to 256 (per real-deployment scale), TP to 64,
PP to 32.
New UI panel in Auto Explore tab: 5 subplots (log/log), one per knob:
- Solid line = latency where the config fits memory
- Red X markers = infeasible (out of budget)
- Dotted vertical line = baseline value
User picks which Pareto row is the "baseline" via a number input; the
sensitivity chart re-computes around it.
Behaviour caveat noted during verification: for large PP the FFN AR can
cross a SIP boundary (uses stage_latencies.py:730 sips_used tier
selection), producing a step-up in latency. This is the existing model's
choice, honestly reflected in the chart. Whether the FFN AR should span
only one PP stage's ranks (thus not cross SIPs) is a separate discussion
about stage_latencies.py.
Verified:
- 11 pytest tests pass (added 2 new for parallelism sensitivity)
- Smoke: at Llama 70B decode 128K with baseline CP=8/TP=16/PP=1/DP=1:
* CP sweep: 4 fits, 8 = baseline optimum, 16+ overshoots memory
* TP sweep: 8/16 fit, 16 = baseline optimum, 32 slower (spans SIPs)
* PP sweep: 1 = baseline, 2+ slower due to sips_used tier drop for FFN AR
* DP sweep: same shape as PP
* EP sweep: monotone decreasing (bigger EP = smaller per-PE FFN)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Consumes auto_hardware.joint_explore(). UI:
- Sweep-depth radio: two_stage / balanced (default) / coarse
- Run joint sweep button + spinner
- 4 metric cards: HW candidates, feasible joint, Pareto count,
best latency
- Panel 1 (Pareto scatter): latency vs hardware cost proxy, feasible
in grey, Pareto colored by PE count, dashed line connects Pareto in
cost order — this is the "optimal HW config for the model" plot
- Panel 2 (sensitivity bar chart): per-knob relative speedup when
doubled from the baseline. Green bars, sorted biggest first;
annotated with the baseline → doubled values. Answers "where to
invest next?"
- Panel 3 (table): sortable Pareto joint configs with parallelism +
HW spec columns (PE HBM, HBM BW, TFLOPs, PE↔PE, D2D, C2C)
- Load into sidebar: picks a Pareto row and syncs both the HW
selectboxes (Per-PE + Interconnect sub-tabs) AND the parallelism
sliders. Values only get loaded when the target is one of the
selectbox options; otherwise the sidebar keeps its current value.
Session-state cache under _hw_result keyed by (model, s_kv, mode, depth);
if any of these drift, a warning suggests refreshing.
Verified: app.py parses; auto_hardware smoke run on Llama 70B decode
128K in ~20s produces sensible HW co-design signal (HBM BW 33% speedup,
everything else <1.5%).
Next: verification across Qwen 3 8B, Mixtral 8x7B (Commit 6).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New module extends auto_explore into the hardware co-design space. For a
fixed model + workload, sweeps hardware knobs (pe_hbm_gb, bw_hbm_gbs,
peak_tflops_f16, bw_intra_gbs, bw_inter_gbs, bw_intersip_gbs) and, for
each hardware candidate, searches parallelism for the latency-minimum
that fits memory. Returns:
- all_scores: every (hw, parallelism) pair that fits, sorted by latency
- pareto_scores: 2D Pareto frontier on (latency ↓, cost_score ↓)
- sensitivity: per-knob rel_speedup when doubled from the best-fast HW
baseline. Ranks which HW knob gives the biggest speedup — a co-design
signal.
Three sweep depths trade coverage for time:
- two_stage: 1 HW candidate (defaults) × autosuggest's memory-min
parallelism. Fast (~1s), useful for the sensitivity ranking alone.
- balanced: 64 HW × ~2k reduced-parallelism configs = ~130k joint evals,
~10-20s. Default UI setting.
- coarse: 729 HW × ~2k configs = ~1.4M joint evals, ~2-5 min.
Reduced parallelism sweep for the inner loop: CP × TP × PP × DP ×
kv_shard_mode (1,920 configs), other 4 knobs held at latency-friendly
defaults (ffn_shard_scope='TP+CP', tp_placement='cube', cp_placement='pe',
cp_ring_variant='qoml' for decode, 'kv' for prefill). Full 28,800-config
auto_explore per HW would take 6+ minutes — too slow.
Cost proxy: sum of (knob / knob_default). 6.0 at defaults. Not dollars —
a rough capability score where higher = "more spec'd hardware".
Verified:
- 9 pytest tests pass:
* enumeration counts match expected (1, 64, 729)
* default cost_score = 6.0
* Pareto non-dominated + subset of all_scores
* every knob is monotone-non-worsening when doubled
* for Llama 70B decode, bw_hbm_gbs tops the sensitivity ranking
(physically correct: memory-bound workload)
- Smoke: Llama 70B decode 128K balanced sweep in ~20s produces
5 Pareto configs; best 7.57 ms with 1024 GB/s HBM BW. Doubling
HBM BW gives 33% additional speedup; every other knob < 1.5%.
Next: Streamlit UI tab consuming this in Commit 5.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New tab consumes auto_explore.run_auto_explore(). UI:
- Header showing current model + workload + per-PE HBM budget
- Run sweep button (spinner while ~28k configs run in ~5-10s)
- 4 metric cards: enumerated, feasible, pareto count, best latency
- 2-panel scatter:
* latency vs PEs (feasible in grey, Pareto coloured by efficiency,
dashed line connects Pareto in PE order to show the trade-off curve)
* HBM utilization vs latency for Pareto, coloured by PE count
- Sortable Pareto table
- "Load into sidebar" widget: pick a row, sets the sidebar
session_state keys (cp, tp, pp, dp, tp_placement, cp_placement,
cp_ring_variant, kv_mode, ffn_scope_label) and st.rerun()s so the
user can flip to another tab and see the full breakdown.
Session-state caches the last sweep result under _auto_explore_result;
if the model/workload/HBM change without re-running, a warning suggests
refreshing.
Ffn_scope_label mapping is dynamic (contains substituted divisors), so
the Load button reconstructs the exact label using the target
cp/tp/dp values before assigning to session_state["ffn_scope_label"].
Verified:
- app.py parses cleanly
- All 9 pytest tests in test_auto_explore.py still pass
- Smoke: matplotlib + pandas + auto_explore imports round-trip
Next: verification pass across Qwen 3 8B and Mixtral 8x7B; polish.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extends the memory-only autosuggest to search the full 9-dimensional
parallelism space (CP, TP, PP, DP, kv_shard_mode, ffn_shard_scope,
tp_placement, cp_placement, cp_ring_variant) and rank feasible configs
on a 3D Pareto frontier: (latency ↓, pes_used ↓, efficiency ↑).
Throughput is stored on ConfigScore for display but is deliberately NOT
a Pareto axis because for a single-request analysis it collapses to
1 / latency, which would collapse the frontier.
Reuses existing physics (stage_latencies.all_stages + all_ffn_stages,
memory_layout.compute_memory) — no new formulas.
Single-request latency formula fix: PP does NOT reduce single-request
decode/prefill latency because the request has to traverse every layer
sequentially regardless of pipeline depth. The initial version had
latency ~ per_layer × layers_per_stage, which incorrectly rewarded high
PP. Corrected to latency ~ per_layer × model.layers.
Enumerator prunes:
- PP > model.layers
- TP > 4 × h_q
- ffn_shard_scope contains 'DP' when dp=1 (redundant)
- cp_ring_variant='qoml' when cp=1 (no-op)
Full sweep on Llama 3.1 70B: ~28,800 configs enumerated in ~7s, ~7k-10k
feasible (varies with S_kv), 2-7 unique Pareto configs. Faster context
lengths produce richer frontiers; at 1M, memory forces a single
dominant config (128 PEs, HBM 85%).
Verified:
- 9 pytest tests pass (enumerate, score, Pareto, subset invariants)
- Manual: Llama 70B decode at 8K/64K/128K/1M produces physically
sensible Pareto (CP=8/TP=16 wins latency; smaller-PE options
appear at longer context up to memory limits)
Next: Streamlit tab UI in app.py + verification against more presets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wire per-flit loop in engine._wire (ADR-0033 Phase 2c) iterates
nbytes/flit_bytes times per hop, firing ~2 SimPy events per flit.
At long-context decode, K/V tile loads are hundreds of KB per shard,
producing thousands of flit-iterations per hop. The default 256-byte
flit size made the wire loop the dominant wall-clock cost.
Raising flit_bytes to 4096 (still a multiple of hbm_ctrl.burst_bytes=256
per ADR-0033 D1) cuts flit iterations 16x with no meaningful change to
modeled latency. Wormhole pipelining and the available_at BW-share
chain in _wire behave identically at any flit size.
Measured impact on Case 4 (Cube-SP x PE-SP) decode, C=8, P=8, T_q=1,
LLaMA-3.1-70B single-KV-group, enable_data=False:
S_kv flit=256 wall flit=4096 wall speedup latency drift
128K 233.71 s 18.16 s 12.9x +0.05% (0.25us)
256K 485.75 s 25.31 s 19.2x +0.01% (0.09us)
512K 1045.64 s 48.45 s 21.6x +0.01% (0.17us)
1M ~2250 s (proj) 104.75 s ~21x within extrap
Total sweep at flit=4096: ~3.3 min. Baseline would have been ~65 min.
1M decode iteration is now practical.
op_log records are byte-identical (same counts, same components); the
tiny latency drift (order of a few ns) comes from the last flit's
env.timeout landing on a slightly different granularity when flits are
larger. Well within noise for any test that asserts on structural
invariants (dma_write cubes, ipcq_copy counts).
Verified:
- tests/attention/test_milestone_gqa_decode_long_ctx_4cases.py: 18/18 pass
(also 7x faster: 174s -> 24.83s per full run)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per-instance cache keyed by (id(adj), start, goal). All 4 adj dicts
(_adj, _adj_all, _adj_local, _adj_mcpu_dma) are built in __init__ and
never mutated (topology is static per ADR-0006 / SPEC §0.1), so id(adj)
is stable for the router's lifetime. Cache is populated on the
successful-return paths; RoutingError paths intentionally re-run each
call (rare, keeps error semantics unchanged).
Motivation: cProfile of Case 4 decode at S_kv=8K showed 1,142
_run_dijkstra_with_dist calls consuming ~1.95s tottime. Paths depend
only on (adj, src, dst) so ~99% of those calls are recomputing the same
result.
Verified:
- tests/attention/test_milestone_gqa_decode_long_ctx_4cases.py: 18/18 pass
- 128K decode wall: 237.28s -> 233.71s (-1.5%)
- Modeled kernel latency: 461.13us (byte-identical before/after)
- op_log_len: 3057 (unchanged)
Small win but no risk: memoization returns byte-identical results and
paths cannot change during a sim run. Larger event-count reductions
require touching the per-hop hot path (zero-latency chain collapse
etc.).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three coupled fixes so enable_data=False now reports byte-identical
kernel-body sim latency (max(t_end) - min(t_start) over op_log) as
enable_data=True, and skips setup-write sim events that were pure
wall-clock overhead.
Before this change, at Case 4 (Cube-SP x PE-SP) decode, S_kv=8K:
enable_data=False -> latency_ns = 0 (op_log empty)
enable_data=True -> latency_ns = 30.646 us (correct)
After:
enable_data=False -> latency_ns = 30.646 us (byte-equal)
enable_data=True -> latency_ns = 30.646 us (unchanged)
engine.py: always create OpLogger. Previously OpLogger was gated on
enable_data=True together with MemoryStore, so op_log stayed empty when
data mode was off. Decouple: MemoryStore is gated (Phase 2 DataExecutor
still needs it) but OpLogger runs unconditionally, receiving
memory_store=None when data mode is off. OpLogger already guards its
arr.copy() snapshot paths on `if self._memory_store is not None`.
pe_cpu.py: always use the ADR-0020 greenlet execution path. The
if store is not None: _execute_greenlet(); else: _execute_legacy branch
gated the *execution model* on data-mode presence. The legacy
command-list path predates ADR-0020 and doesn't route IpcqSendCmd
through PE_IPCQ (it goes to PE_SCHEDULER instead), so all 189 Case-4
ipcq_copy fabric transfers were silently no-op'd in that mode. Forcing
greenlet gives identical sim behavior in both modes. KernelRunner
already guards its store reads on `self._store is not None`.
context.py: gate setup MemoryWriteMsg on memory_store presence. Under
enable_data=False there is no MemoryStore to populate and no Phase 2
replay, so the per-shard sim events for Q/K/V deploy are pure wall-clock
overhead with no effect on reported kernel latency (Yangwook's max-min
formula excludes pre-kernel ops). Handle count for Case 4 at S_kv=8K
drops 229 -> 37 under enable_data=False.
Verified:
- tests/attention/test_milestone_gqa_decode_long_ctx_4cases.py: 18/18 pass
- Parity probe: enable_data=False/True both report 30.646 us kernel sim
time, 1649 op_log records, 189 ipcq_copy events
Known follow-up: pe_cpu._execute_legacy is now dead code but left in
place; separate cleanup once we confirm no downstream caller.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New tests/analytical_visualization/ module - an interactive dashboard
for exploring memory / latency tradeoffs of transformer inference on
the SIP architecture.
Highlights:
- 30+ model presets (Qwen, Llama 2/3/3.1, Mistral, Gemma 2, Phi 3,
DeepSeek MLA, Mixtral/Qwen 3 MoE, Grok-1, ...)
- Placement toggles: TP and CP each on PE-level vs cube-level
- CP ring variant: K/V ring vs Q+O/m/l ring (prefill); in decode the
O/m/l all-reduce is folded into S8 (no separate C1 row)
- SIP interconnect: ring / mesh2d / torus2d with matching link drawing
- Per-stage latency table with compute + memory + comm formulas,
auto-scaled ns/us/ms, colored by dominant bound
- Ring attention loop indicator on the pipeline diagram (purple arc
over S5-S8 with 'xN hops' badge)
- Tensor sharding view with optional physical PE/cube annotations
- Replication-waste + optimization-hints panel
- Save & compare configurations (config1, config2, ...): summary table
plus side-by-side per-stage attention and FFN latency, best-in-row
highlighting
- Symbol glossary with current values for every symbol used in formulas
Not tied to production sim_engine or runtime API; purely analytical
tooling for design-space exploration.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both 1-kv (2 users x 64 PE) and 8-kv (16 users x 8 PE) fill all 128 PEs of
the SIP; the throughput gap is per-PE HBM efficiency. Splitting a head over
8 PEs leaves each a single tile, so pipeline-fill and the 8-PE softmax
reduce dominate -> 46% HBM util; one head per PE streams contiguously with
no reduce -> 76%. Throughput ratio (1.6x) tracks the utilization ratio
(1.7x): decode is bandwidth-bound, so throughput follows total HBM
efficiency, not PE count. 1-kv spends hardware on latency (8 PE/head for a
4.8x speedup = 60% strong-scaling efficiency).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beyond a mapping's SIP capacity (16/C) the throughput plateaus: extra
users run in waves at the saturated rate, they are not un-runnable. Draw
a dashed horizontal extension at the ceiling so 1-kv-per-cube (cap 2)
reads as saturating, not stopping, at B=2. Caption updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Concurrency fix (completes the cube_base change): the decode kernel's
output store o_base still used the global cube_id while every load used
the user-local cube_local. For a single user (cube_base=0) they coincide,
so it was masked; a second batched user (cube_base>0) overshot its o shard
into an unmapped VA, mis-decoded to a phantom sip0.cube0.pe8 and raised a
RoutingError. Switch o_base to cube_local (one line); single-user results
are byte-identical (decode smoke/correctness pass).
Batch harness (un-skipped): create all users' tensors first, then submit
all launches deferred and drain together, so deploys don't drive an
already-submitted launch and serialize the batch. Concurrent users on
disjoint CUBE groups now overlap to within 3-5% of the single-user
latency. Swept B in {1,2,capacity} at 8K (high-B dense runs are the
expensive ones; throughput is near-linear in B).
Measured result: aggregate throughput at a full SIP rises from 0.86
(1-kv-per-cube, 2 users) to 1.41 requests/us (8-kv-per-cube, 16 users) —
the dense mapping wins throughput, the spread mapping wins latency.
S5.2 batch paragraph updated projected -> measured; add batch_scaling
figure and plot_batch_scaling.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Metric fixes (test harnesses, deploy artifact + wrong constant):
- wall = max(t_end) - min(t_start): exclude the one-time KV-cache deploy
from the measured decode/prefill step (it was 92-99% of wall, mapping-
invariant, and masked the real per-mapping separation).
- PEAK_PE_HBM_BPS 128 -> 256 B/ns (8 channels/PE x 32 GB/s); the old value
was half the modeled per-PE HBM BW, so hbm_bw_util read >1.0 once wall
was corrected. All six short-context sweep CSVs regenerated/repatched.
Result: the four KV mappings now separate along a {64,32,16,8}-active-PE
ladder (decode 8-kv/1-kv = 4.8x at 8K, 7.4x at 64K), not "modest/tied" as
before; decode is bandwidth-bound at 46-76% of the 256 GB/s per-PE ceiling.
Report (S5.2 rewrite):
- Replace the tied-wall / 8x-per-PE-util claims (both deploy artifacts)
with the corrected separation and a density trade-off (per-CUBE KV
footprint, Fig 16 top panel switched to a wall-invariant metric).
- Add a projected latency-vs-batched-throughput analysis (marked
projected, not measured): dense mappings win throughput, 1-kv wins
latency; converges at long context.
- Regenerate Fig 15/16/17/18; fix plot script hardcoded ROOT path.
Batch experiment (Part 2, cube_base):
- Add backward-compatible cube_base=0 scalar to the decode kernel so a
batched user placed at DPPolicy.cube_start addresses 0-based shards.
Default preserves single-user behavior (32 decode tests pass unchanged).
- New batch harness (skipped): concurrent B>=2 launches hit a sim routing
issue (sip0.cube0.pe8); single-user path verified. Concurrency fix next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Soften short-context claim: 1-kv-/8-kv-per-cube are tied at 8K and
separate as context grows (1-kv faster at the large end).
- Add a bridge noting the six-case placement taxonomy (§5.1) is the
long-context lens; short context turns on head-to-CUBE distribution.
- Remove caption-duplicating numbers from the long-context conclusion.
- Reduce the closing subsection to a GQA-scoped Summary; defer the
cross-cutting hardware-investment thesis to the Discussion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make reduce_mlo derive its submesh dimensions (sub_w, sub_h, root_col,
root_row, root_cube) from C at call time, with peer-existence guards on
every inter-cube send/recv so any C ≥ 1 completes cleanly (previously
hardcoded to C=8's 4×2 mesh; non-rectangular C like 7 or 12 hit
IpcqDeadlock at the root cube's east receive). Byte-equal at C=8 —
existing composite digest tests still pass 8/8.
Callsites in the four Case-6 kernels (primitive / primitive-tiled /
composite / composite_extended) updated to pass C into reduce_mlo and
use root_cube_for(C) for the final tl.store gate.
Add the multi-model attention bench: sweeps the composite kernel at
S_kv=128K across six GQA models (Gemma-2 27B, LLaMA-3 8B, Qwen 2.5 7B,
LLaMA-3 70B, Qwen 2.5 72B, Command R+), with per-model topology
C = h_q (cubes per KV group = query heads per KV group). Captures
per-op-kind occupancy (matmul GEMM+MATH, comm DMA) alongside latency
and PE_CPU dispatch. Three-panel plot writes to bench-output and
paper-figures dir.
Headline result: same-h_q-different-family models land within 0.2 µs
(model-agnostic given fixed h_kv=1 per KV group + d_head=128); latency
climbs 263 → 492 µs as C climbs 2 → 12, driven by reduce depth
(comm/matmul ratio 0.4 → 0.6 across the sweep).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch primitive hand-tiled Q·Kᵀ / P·V from a deferred-K-sum tl.dot
chain to per-block GemmCmd writes into a shared (M, N) out handle
(implicit MAC-side accumulation). Full-shape coarse Q / K_T / V loads
carry data-mode correctness; per-block DMAs remain for the streaming
architecture dispatch story. The kernel now runs in engine mode
end-to-end, so its 128K latency is measured instead of derived as
primitive + Δdispatch. Drop the coarse-primitive baseline from the
sweep and plots; keep the kernel file for reference.
At S_kv=128K: primitive hand-tiled = 959.5 µs vs composite = 460.7 µs
(2.08× from 5 235 vs 94 PE_CPU commands).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires a fourth variant into the Cube-SP × PE-SP long-context decode
composite command-form study to surface the worst-case PE_CPU dispatch
inflation that the coarse composite forms delegate to PE_SCHEDULER
(ADR-0065): per-block DMA of Q/K/V slices, tl.dot per 16³ block,
deferred K-inner sum outside the K loop.
Renamed _tiled.py → _hand_tiled_16x16x16.py; coarse-primitive kernel
retained for the 4-cases bench, paper scripts, and the golden byte-equal
regression guard.
New breakdown bar chart at S_kv=128K shows engine (~460 μs) dominates
all three variants; hand-tiled adds ~68 μs PE_CPU dispatch on top —
composite forms sit essentially on the memory-bound floor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mukesh
ywkang
eusonice