mukesh
fc4747668e
Three new subsections at the top of §5, motivated by the deployment-sizing questions the analytical visualization tool exposes: - 5.1 Roofline Analysis — B*, L*, short vs long context batch effect (measured: 19× per-token cost reduction at short context, 28% at long) - 5.2 Capacity Planning — 3-axis sizing (weights / KV / SLO), SLO targets, regime rules, deployment templates, per-axis playbook - 5.3 Parallelism Selection — TP × CP × PP × DP × EP comparison, symptom-driven axis selection, add/stop criteria, misconceptions Restructure: - 05-gqa.tex trimmed to section header + intro - Existing 5.4-5.7 content (placement, short/long ctx, composite) moved to 05x-fused-kernel.tex - Summary extracted to 05z-summary.tex, cross-refs updated - main.tex \input order sets the requested subsection sequence - lmodern loaded to satisfy microtype font-expansion Two new figures generated from tests/analytical_visualization/chip_roofline: - roofline_short_context.png (S_kv=8K, batch wins) - roofline_long_context.png (S_kv=1M, batch stalls at KV floor) - Generator: tests/analytical_visualization/_gen_roofline_paper_figs.py 4 tables (regime rules, deployment templates, playbook, parallelism criteria) use table* so they span both columns without overflow. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
92 lines
3.8 KiB
TeX
92 lines
3.8 KiB
TeX
\subsection{Parallelism Selection: TP $\times$ CP $\times$ PP $\times$ DP $\times$ EP}
|
|
\label{sec:parallelism-selection}
|
|
|
|
Given a capacity-feasible layout (\S\ref{sec:capacity-planning}), the
|
|
remaining decision is \emph{how to shard}: which axes to enable and at
|
|
what degrees. Memory is the feasibility filter; once you fit, the
|
|
design problem is minimising \emph{exposed} communication (communication
|
|
that could not be overlapped with compute). The core principle is to
|
|
\textbf{rank techniques by how much they communicate per unit of
|
|
compute, and assign the chattiest ones to the fastest links.}
|
|
|
|
\begin{table*}[h]
|
|
\centering
|
|
\small
|
|
\caption{Parallelism axes at a glance. TP AllReduce volume does not
|
|
shrink with degree — only compute does — which sets its practical
|
|
ceiling near a fast-link (NVLink / intra-SIP) domain.}
|
|
\label{tab:parallelism-compare}
|
|
\begin{tabular}{@{}l l l l l@{}}
|
|
\toprule
|
|
Axis & Shards & Comm pattern & Frequency & Hard ceiling \\
|
|
\midrule
|
|
DP & optimizer state (w/ ZeRO) & AllReduce (grads) & 1$\times$/step & critical batch \\
|
|
TP & weights + acts + KV heads & AllReduce & 2$\times$/layer & fast-link domain; $\lesssim h_{kv}$ \\
|
|
PP & weights + KV by layer & P2P activation hand-off & per stage boundary & $n_{\text{layers}}$ \\
|
|
CP & activations + KV by position & Ring P2P per hop & per layer (overlappable) & $\mathrm{seq\_len} / \mathrm{block}$ \\
|
|
EP & expert weights (MoE only) & All-to-all & 2$\times$/MoE layer & $n_{\text{experts}}$ \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
|
|
\paragraph{Symptom-driven axis selection.} The dominant problem dictates
|
|
the first knob to try (Table~\ref{tab:parallelism-problem}).
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\small
|
|
\caption{Which axis to reach for first, by dominant problem.}
|
|
\label{tab:parallelism-problem}
|
|
\begin{tabular}{@{}l l@{}}
|
|
\toprule
|
|
Dominant problem & First axis \\
|
|
\midrule
|
|
Weight memory doesn't fit & TP within a fast domain \\
|
|
KV of ONE long sequence & CP \\
|
|
KV for MANY sequences & DP replicas \\
|
|
Single-request TPOT & TP ($\lesssim 8$) \\
|
|
Aggregate throughput & Outer DP \\
|
|
Model spans multiple nodes & TP intra-node, PP inter-node \\
|
|
MoE expert weight memory & EP \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\paragraph{When to add, when to stop.} Every axis has a natural
|
|
saturation point past which further degree hurts more than it helps
|
|
(Table~\ref{tab:parallelism-criteria}). Sizing decisions are almost
|
|
always constrained by two axes simultaneously (e.g.\ TP by NVLink
|
|
domain and $h_{kv}$; CP by ring-hop hiding and per-rank block size).
|
|
|
|
\begin{table*}[t]
|
|
\centering
|
|
\small
|
|
\caption{Add / stop criteria per axis.}
|
|
\label{tab:parallelism-criteria}
|
|
\begin{tabular}{@{}l l l@{}}
|
|
\toprule
|
|
Axis & Add when & Stop when \\
|
|
\midrule
|
|
DP & more independent requests & global batch $>$ critical \\
|
|
TP & weights need sharding or TPOT tight & collectives dominate; local GEMMs shrink \\
|
|
PP & depth too large; TP would cross slow links & bubble $>$ $\sim$10\% \\
|
|
CP & one sequence needs position sharding & ring hops can't overlap compute \\
|
|
EP & MoE expert memory & expert GEMMs too small; routing imbalance \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
|
|
\paragraph{Common misconceptions.} Three widely repeated rules survive
|
|
contact with real workloads only partially:
|
|
\emph{(i)} memory is not just a feasibility filter but a continuous
|
|
performance variable --- more free HBM enables larger $B$, more prefix
|
|
cache, higher throughput even after weights fit;
|
|
\emph{(ii)} the ``TP $\leq h_{kv}$'' ceiling is an efficiency
|
|
preference, not a correctness constraint --- vLLM and others support
|
|
KV-head replication and head-dim splitting for TP $>$ $h_{kv}$;
|
|
\emph{(iii)} ``always start at TP=8'' is disproven by public
|
|
disclosures (DeepSeek-V3 inference uses TP=4, some MLPs at TP=1;
|
|
DeepSeek-V3 training uses PP=16 + EP=64 with no TP), which shows the
|
|
right starting point is the smallest TP that fits memory and meets
|
|
latency, followed by measurement.
|