analytical-viz: partial batch (B) scaffold — TopologyConfig field + KV / early stages
WIP toward batch-size support. This first commit is behavior-preserving
at the default B=1 (every new multiplication is by max(1, cfg.topo.b) =
1 today) so all existing tests pass. Follow-up commits will:
- scale the remaining stages (S6/S7/S8/S9/S10 + C1/C2/C3 + FFN + FFN AR)
- add a batch selectbox to the sidebar
- forward b through auto_suggest / auto_explore / auto_hardware
Changes so far:
- TopologyConfig: new b: int = 1 field (batch size).
- memory_layout.per_pe_kv_cache_bytes: * B (each concurrent request
keeps its own KV cache slice).
- stage_rmsnorm / stage_wq / stage_wkv / stage_kv_append /
_per_hop_qkT_pv: FLOPs and activation memory scaled by B; weight
bytes stay fixed (weights shared across the batch).
Verified: 24 pytest tests still pass at default B=1.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -43,6 +43,7 @@ class TopologyConfig:
|
||||
pp: int = 1 # pipeline stages (layer shard)
|
||||
dp: int = 1 # data-parallel replicas
|
||||
ep: int = 1 # expert-parallel degree (MoE only)
|
||||
b: int = 1 # batch size (concurrent requests per PP stage)
|
||||
s_kv: int = 4096
|
||||
mode: str = "decode" # "decode" or "prefill"
|
||||
kv_shard_mode: str = "split" # "split" or "replicate" (used when TP > H_kv)
|
||||
|
||||
Reference in New Issue
Block a user