\subsection{Parallelism Selection: TP $\times$ CP $\times$ PP $\times$ DP $\times$ EP}
\label{sec:parallelism-selection}

Given a capacity-feasible layout (\S\ref{sec:capacity-planning}), the
remaining decision is \emph{how to shard}: which axes to enable and at
what degrees. Memory is the feasibility filter; once you fit, the
design problem is minimising \emph{exposed} communication (communication
that could not be overlapped with compute). The core principle is to
\textbf{rank techniques by how much they communicate per unit of
compute, and assign the chattiest ones to the fastest links.}

\begin{table*}[h]
\centering
\small
\caption{Parallelism axes at a glance. TP AllReduce volume does not
shrink with degree — only compute does — which sets its practical
ceiling near a fast-link (NVLink / intra-SIP) domain.}
\label{tab:parallelism-compare}
\begin{tabular}{@{}l l l l l@{}}
\toprule
Axis & Shards & Comm pattern & Frequency & Hard ceiling \\
\midrule
DP & optimizer state (w/ ZeRO) & AllReduce (grads) & 1$\times$/step & critical batch \\
TP & weights + acts + KV heads & AllReduce & 2$\times$/layer & fast-link domain; $\lesssim h_{kv}$ \\
PP & weights + KV by layer & P2P activation hand-off & per stage boundary & $n_{\text{layers}}$ \\
CP & activations + KV by position & Ring P2P per hop & per layer (overlappable) & $\mathrm{seq\_len} / \mathrm{block}$ \\
EP & expert weights (MoE only) & All-to-all & 2$\times$/MoE layer & $n_{\text{experts}}$ \\
\bottomrule
\end{tabular}
\end{table*}

\paragraph{Symptom-driven axis selection.} The dominant problem dictates
the first knob to try (Table~\ref{tab:parallelism-problem}).

\begin{table}[h]
\centering
\small
\caption{Which axis to reach for first, by dominant problem.}
\label{tab:parallelism-problem}
\begin{tabular}{@{}l l@{}}
\toprule
Dominant problem & First axis \\
\midrule
Weight memory doesn't fit & TP within a fast domain \\
KV of ONE long sequence & CP \\
KV for MANY sequences & DP replicas \\
Single-request TPOT & TP ($\lesssim 8$) \\
Aggregate throughput & Outer DP \\
Model spans multiple nodes & TP intra-node, PP inter-node \\
MoE expert weight memory & EP \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{When to add, when to stop.} Every axis has a natural
saturation point past which further degree hurts more than it helps
(Table~\ref{tab:parallelism-criteria}). Sizing decisions are almost
always constrained by two axes simultaneously (e.g.\ TP by NVLink
domain and $h_{kv}$; CP by ring-hop hiding and per-rank block size).

\begin{table*}[t]
\centering
\small
\caption{Add / stop criteria per axis.}
\label{tab:parallelism-criteria}
\begin{tabular}{@{}l l l@{}}
\toprule
Axis & Add when & Stop when \\
\midrule
DP & more independent requests & global batch $>$ critical \\
TP & weights need sharding or TPOT tight & collectives dominate; local GEMMs shrink \\
PP & depth too large; TP would cross slow links & bubble $>$ $\sim$10\% \\
CP & one sequence needs position sharding & ring hops can't overlap compute \\
EP & MoE expert memory & expert GEMMs too small; routing imbalance \\
\bottomrule
\end{tabular}
\end{table*}

\paragraph{Common misconceptions.} Three widely repeated rules survive
contact with real workloads only partially:
\emph{(i)} memory is not just a feasibility filter but a continuous
performance variable --- more free HBM enables larger $B$, more prefix
cache, higher throughput even after weights fit;
\emph{(ii)} the ``TP $\leq h_{kv}$'' ceiling is an efficiency
preference, not a correctness constraint --- vLLM and others support
KV-head replication and head-dim splitting for TP $>$ $h_{kv}$;
\emph{(iii)} ``always start at TP=8'' is disproven by public
disclosures (DeepSeek-V3 inference uses TP=4, some MLPs at TP=1;
DeepSeek-V3 training uses PP=16 + EP=64 with no TP), which shows the
right starting point is the smallest TP that fits memory and meets
latency, followed by measurement.