New section at the bottom of the Chip Roofline tab implementing the
three-axis GPU count decision:
N_GPUs = max(capacity_floor, KV_headroom, throughput_SLO) × N_replicas
Interactive inputs: concurrent users, avg context/user, TPOT SLO.
Four KPI cards report each axis + the recommended total, with a
'binds:' delta chip showing which axis is the bottleneck.
- Axis A (capacity): PEs to hold weights alone
- Axis B (KV): PEs to hold weights + all KV of one replica's users
- Axis C (throughput): replicas needed at B_at_SLO users each
- Total: max(A,B) × N_replicas
Contextual caption explains what to do about the binding axis. If SLO
is infeasible (even B=1 exceeds SLO), an error block explains the
options (loosen SLO, shorten context, or scale up FLOPs/BW knobs).
Plus three collapsed expanders explaining the hybrid deployment
pattern hyperscalers use:
- Layer 1: one elastic pool + PagedAttention + continuous batching
- Layer 2: length-tier routing (standard vs long-context)
- Layer 3: disaggregated prefill/decode (DistServe, Splitwise)
Pure additions in chip_roofline.py:
- max_batch_within_slo(machine, model, s_kv, slo_s) — analytical
inverse of step_latency to find the largest per-replica B under SLO
- size_deployment(machine, model, n_users, avg_ctx, slo_s) — returns
GpuSizingResult with all three axes + binding info
6 new tests cover: capacity scales with model size, KV grows with
users/context, binding axis flips with workload shape, tight SLO
shrinks max batch, weight-time-exceeds-SLO returns 0, total ==
per-replica × replicas.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three additions to the Chip Roofline tab:
1. **PE memory budget** section (two side-by-side stacked-area plots)
- Left: per-PE memory vs S_kv (128..1M, log). Stacks: weights,
KV, transient. HBM budget line + current-S_kv marker.
- Right: per-PE memory vs B (1..256). Same stacks, current-B
marker. KV grows linearly with either axis; weights are
invariant of B and S_kv (fixed by CP·TP·PP sharding).
Uses actual sharded per-PE quantities from memory_layout so the
numbers match what the deployment would really allocate.
2. **AI sensitivity** section (two side-by-side line plots)
- Left: AI vs chip-parameter multiplier. Two curves: scale FLOPs
(AI grows linearly), scale HBM BW (AI shrinks inversely).
- Right: B* vs multiplier. Same shape as AI (for BF16 where B*=AI).
Vertical markers show current FLOPs/BW knob positions.
3. **Two new knobs** in the top row: FLOPs × multiplier and HBM BW ×
multiplier (0.25..8, step 0.25). The AI-sensitivity plot markers
move with them; the header caption shows scaled AI + B* for the
selected point.
Also: formula annotation on the headline 'Latency per decode step'
plot header showing t_step = N·b/W + 2·N·B/C + B·S_kv·kv_bpt/W.
Pure additions in chip_roofline.py (all testable):
- MemoryBudgetPoint dataclass
- memory_budget_curve_vs_skv, memory_budget_curve_vs_batch
- AISensitivityPoint dataclass
- ai_sensitivity_curve(machine, model, mults, axis='flops'|'bw')
9 new tests cover linear scaling of KV with S_kv/B, weight
invariance across sweeps, over-budget flag flip, free_gb clamp,
FLOPs multiplier → AI linear, BW multiplier → AI inverse, B*
tracks AI for BF16, bad axis raises.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two new plots at the top of the Chip Roofline tab (right after the
knob row), side by side:
1. **Latency per decode step** — one forward pass's total time as B
grows. weight_fetch (flat in B), compute (linear), KV (linear),
total. The SLO / TTFT view — bigger B = longer step.
2. **Cost per token** = step latency ÷ B. weight (shrinks 1/B),
compute (flat), KV (flat), total. The efficiency / cost view —
bigger B (up to ~2·B*) = cheaper per token.
Same underlying decomposition, two divisors. Puts the classic
throughput ↔ latency tradeoff in one glance.
Also added a **B (batch size) slider** next to the existing S_kv
slider. Both plots draw a vertical marker at the current B; the
regime KPI ("memory-bound / compute-bound / kv-bound") now uses the
slider B, so you can sweep B live and watch the label flip when you
cross B*.
step_latency_curve + StepLatencyPoint added to chip_roofline; 5 new
tests cover: weight_s batch-invariant, compute_s/kv_s linear in B,
step_total == per_token_total × B (definitional), and step ==
per_token at B=1.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bundle of roofline-tab enhancements requested in-thread:
1. **S_kv slider at top of tab** — override the sidebar's S_kv locally
so all four plots update as you drag it. Handy for exploring how
the KV wall arrives without disturbing the rest of the app config.
2. **Good B / Good L KPI cards** — 3 metrics under the main KPI row:
- Good B = 2·B* (Pope's rule of thumb)
- Good L ceiling = L* (compute-friendly context ceiling)
- Utilization @ current S_kv = 1 / (1 + S_kv/L*)
3. **Plot 4 — latency-decomposition per decode step** — five curves
on one axis:
- Compute (dashed, flat)
- Weight fetch (triangles, shrinks 1/B)
- KV fetch (squares, flat — batching doesn't help)
- Memory total (weights + KV, purple)
- Total (compute + memory, black bold)
Makes "which term dominates at this B?" visible at a glance.
4. **Regime formulas table** — one row per cost term (t_com, weight
fetch, KV fetch, bottleneck) × two columns (short-context vs
long-context regime), plus a 'value now' column using the
current S_kv slider.
5. **How to pick B and S_kv — one-paragraph guidance** with the
numeric recommendations plugged in.
6. **Split formula table** — was cramming symbolic + substituted
form into one cell with newlines; now has four proper columns:
Symbol / Formula / With numbers / Meaning / Value.
New pure functions in chip_roofline.py: t_mem_short, t_mem_long,
t_com, good_batch, good_context, utilization_at. All roofline math
stays in the pure module; app.py just calls them and formats.
10 new tests bring the roofline test count to 27 (all 67 tests still
green): t_mem_long(L*) == t_com, doubling B halves t_mem_short,
good_batch scales with sparsity, utilization at 0/L*/2L* returns
1.0/0.5/0.333, monotonic decrease with context.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New tab surfaces the arithmetic-intensity story from LLM-serving
practice for the current sidebar chip + model. All derived from
MachineParams + ModelConfig; no new configuration.
chip_roofline module (pure functions):
- arithmetic_intensity = C / W (FLOPs per byte HBM BW)
- critical_batch (B*) = C * b / (2 * W) * sparsity
- balance_context (L*) = 2 * N_active * W / (C * kv_bpt)
- knee_batch (B_knee) = B* / (1 - S_kv/L*) (None past L*)
- per_token_latency_curve = weight/B + compute + KV, per B
- bound_regime = which term dominates now
Peak roofline convention (no compute_util factor) so weight_s ==
compute_s exactly at B*. Comm + TP/CP sharding intentionally
excluded — stage_latencies is the full latency model; this is the
back-of-envelope chip-vs-model view.
Tab shows:
- 4 KPI cards: AI, B*, L*, current regime at (B, S_kv)
- MoE hint when preset flags MoE ('300 x sparsity' rule)
- Plot 1: cost vs B at current S_kv (weight, compute floor, KV
floor, total) with B* marker line
- Plot 2: cost curves at S_kv/L* = 0.25, 0.5, 1, 2, 5 — shows the
no-knee regime past L*
- Plot 3: B_knee vs S_kv — knee slides right, diverges at L*
- Formulas + interpretation table
test_chip_roofline covers B* against H100 reference (295), sparsity
scaling, L* scaling with HBM BW, knee divergence at L*, per-token
curve monotonicity + asymptote to compute+KV floor, regime
classification. Smoke test guards the tab is wired in app.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mukesh