Finishes the batch scaffold. All stages that produce per-request work
now scale flops / activation-memory / comm-bytes by B; weight bytes stay
fixed (shared across the batch).
Attention block, previously partial:
- S6 stage_softmax: elems + bytes × B
- S8 stage_merge: flops × B; O/m/l AR bytes M × B
- S9 stage_normalize: flops × B
- S10 stage_wo: flops × B
- C1 comm_cp_ring: decode O/m/l AR M × B; prefill K/V ring + Q+O/m/l
variants both × B
- C2 comm_tp_allreduce (W_O output): bytes × B
- C3 comm_kv_split_allreduce (head-split scores): bytes_per_hop × B
FFN block, previously untouched:
- F1 stage_ffn_rmsnorm: activation × B (weight fixed)
- F2 stage_ffn_gate (via _ffn_gemm): flops × B
- F3 stage_ffn_up: same
- F4 stage_ffn_swiglu: elems + flops × B
- F5 stage_ffn_down: flops × B
- CF1 comm_ffn_allreduce (batched FFN output): bytes × B
Verified with a smoke check on Qwen 3 8B / 128K decode:
B=1 per-layer visible latency: 363 us
B=8 716 us (sub-linear — many stages
stay weight-bound)
B=64 4023 us (approaching linear scaling
as batch dominates)
All 24 pytest tests still pass at default B=1 (backward compat).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mukesh
50aab8d591