analytical-viz: roofline tab — 'How hyperscalers pick GPU count' section

New section at the bottom of the Chip Roofline tab implementing the three-axis GPU count decision: N_GPUs = max(capacity_floor, KV_headroom, throughput_SLO) × N_replicas Interactive inputs: concurrent users, avg context/user, TPOT SLO. Four KPI cards report each axis + the recommended total, with a 'binds:' delta chip showing which axis is the bottleneck. - Axis A (capacity): PEs to hold weights alone - Axis B (KV): PEs to hold weights + all KV of one replica's users - Axis C (throughput): replicas needed at B_at_SLO users each - Total: max(A,B) × N_replicas Contextual caption explains what to do about the binding axis. If SLO is infeasible (even B=1 exceeds SLO), an error block explains the options (loosen SLO, shorten context, or scale up FLOPs/BW knobs). Plus three collapsed expanders explaining the hybrid deployment pattern hyperscalers use: - Layer 1: one elastic pool + PagedAttention + continuous batching - Layer 2: length-tier routing (standard vs long-context) - Layer 3: disaggregated prefill/decode (DistServe, Splitwise) Pure additions in chip_roofline.py: - max_batch_within_slo(machine, model, s_kv, slo_s) — analytical inverse of step_latency to find the largest per-replica B under SLO - size_deployment(machine, model, n_users, avg_ctx, slo_s) — returns GpuSizingResult with all three axes + binding info 6 new tests cover: capacity scales with model size, KV grows with users/context, binding axis flips with workload shape, tight SLO shrinks max batch, weight-time-exceeds-SLO returns 0, total == per-replica × replicas. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-29 15:07:20 -07:00
parent 6410c0843c
commit b7c4c680aa
3 changed files with 321 additions and 0 deletions
@@ -389,6 +389,106 @@ class AISensitivityPoint:
    b_star: float              # C·b/(2·W)


+# ── GPU count sizing (three-axis) ─────────────────────────────────
+
+
+@dataclass
+class GpuSizingResult:
+    # Inputs echoed back for display / debugging
+    n_users: int
+    avg_ctx_tokens: int
+    tpot_slo_s: float
+    # Per-replica values
+    pes_axis_a_capacity: int    # PEs to hold weights alone (bare floor)
+    pes_axis_b_kv: int          # PEs to hold weights + KV of this replica's users
+    pes_per_replica: int        # max(A, B)
+    # Throughput axis
+    b_at_slo: int               # largest per-replica B satisfying SLO
+    users_per_replica: int      # min(n_users, b_at_slo), at least 1
+    n_replicas: int             # ceil(n_users / users_per_replica)
+    # Grand total
+    total_pes: int
+    binding_axis: str           # 'capacity' | 'kv' | 'throughput'
+
+
+def max_batch_within_slo(machine: MachineParams, model: ModelConfig,
+                          s_kv: int, tpot_slo_s: float) -> int:
+    """Largest per-replica B such that decode step latency ≤ SLO.
+
+    step_latency(B) = N·b/W + 2·N·B/C + B·S_kv·kv_bpt/W
+                    = weight_fetch + B · (compute + kv_read)
+
+    Returns 0 if even B=1 exceeds SLO (weight fetch alone too large,
+    or per-sequence term already blows the budget).
+    """
+    n = total_active_params(model)
+    b_elem = model.bytes_per_elem
+    weight_time = n * b_elem / machine.bw_hbm
+    per_seq_time = (_FLOPS_PER_PARAM_PER_TOKEN * n / machine.peak_flops
+                    + s_kv * kv_bytes_per_token(model) / machine.bw_hbm)
+    if weight_time + per_seq_time > tpot_slo_s:
+        return 0
+    remaining = tpot_slo_s - weight_time
+    b_max = int(remaining / per_seq_time)
+    return max(1, b_max)
+
+
+def size_deployment(machine: MachineParams, model: ModelConfig,
+                     n_users: int, avg_ctx: int,
+                     tpot_slo_s: float) -> GpuSizingResult:
+    """Three-axis GPU count for a target workload.
+
+    Axis A — capacity floor: PEs to hold one replica's weights.
+    Axis B — KV headroom:    PEs to hold weights + all KV of the
+                             users assigned to this replica.
+    Axis C — throughput SLO: replicas needed so per-replica B ≤ b_at_slo.
+
+    Result: pes_per_replica = max(A, B); total = pes_per_replica × replicas.
+    Binding axis is whichever grew the count the most.
+    """
+    n = total_active_params(model)
+    b_elem = model.bytes_per_elem
+    weight_bytes = n * b_elem
+    hbm_pe = int(machine.pe_hbm_gb * 1e9)
+    kv_bpt = kv_bytes_per_token(model)
+
+    # Axis C: throughput
+    b_at_slo = max_batch_within_slo(machine, model, avg_ctx, tpot_slo_s)
+    users_per_replica = max(1, min(n_users, b_at_slo)) if b_at_slo > 0 else 1
+    n_replicas = (n_users + users_per_replica - 1) // users_per_replica
+
+    # Axis A: bare weights
+    pes_a = (weight_bytes + hbm_pe - 1) // hbm_pe
+
+    # Axis B: weights + KV load for this replica's users
+    kv_per_replica = users_per_replica * avg_ctx * kv_bpt
+    pes_b = (weight_bytes + kv_per_replica + hbm_pe - 1) // hbm_pe
+
+    pes_per_replica = max(pes_a, pes_b)
+    total = pes_per_replica * n_replicas
+
+    if n_replicas > 1:
+        binding = "throughput"
+    elif pes_b > pes_a:
+        binding = "kv"
+    else:
+        binding = "capacity"
+
+    return GpuSizingResult(
+        n_users=n_users,
+        avg_ctx_tokens=avg_ctx,
+        tpot_slo_s=tpot_slo_s,
+        pes_axis_a_capacity=int(pes_a),
+        pes_axis_b_kv=int(pes_b),
+        pes_per_replica=int(pes_per_replica),
+        b_at_slo=b_at_slo,
+        users_per_replica=int(users_per_replica),
+        n_replicas=int(n_replicas),
+        total_pes=int(total),
+        binding_axis=binding,
+    )
+
+
 def ai_sensitivity_curve(machine: MachineParams, model: ModelConfig,
                          multipliers: list[float],
                          axis: str = "flops") -> list[AISensitivityPoint]: