\subsection{Roofline Analysis: Batch and Context Regimes}
\label{sec:roofline}

Before deploying any sharding strategy, the workload's position on the
decode roofline sets what \emph{can} be improved and what \emph{cannot}.
Two quantities matter: the machine's \textbf{critical batch}
$B^\ast = C \cdot b / (2 W)$ (where $C$ is per-PE FLOPs, $W$ is per-PE
HBM bandwidth, $b$ is bytes per parameter), and its \textbf{balance
context} $L^\ast = 2 N / (\mathrm{AI} \cdot \mathrm{kv\_bpt})$ (where
$N$ is active parameters and $\mathrm{AI} = C/W$ is arithmetic
intensity). Above $B^\ast$ the deployment leaves the memory-bound
regime; above $L^\ast$ per-user KV streaming dominates and no amount
of batching hides it. For Llama-3.1-70B on the default AHBM machine
($C \!=\!  \SI{8}{TFLOPS}$/PE, $W\!=\!  \SI{256}{GB/s}$/PE), we get
$B^\ast \!\approx\! 31$ and $L^\ast \!\approx\! \SI{13.4}{K}$ tokens.

\paragraph{Short-context regime ($S_{kv} \!<\! L^\ast$).}
Figure~\ref{fig:roofline-short} shows one decode step at
$S_{kv}\!=\!  \SI{8}{K}$. The step-latency panel (left) makes it
obvious that weight fetch dominates at low $B$: at $B\!=\!1$, one
step spends $\sim\!\SI{535}{ms}$ streaming weights and only
$\SI{28}{ms}$ on per-sequence compute and KV combined. The
cost-per-token panel (right) is where the batch story lives: dividing
step latency by $B$ shrinks the weight term as $1/B$, so per-token
cost drops from $\SI{562}{ms}$ at $B\!=\!1$ to $\SI{29.7}{ms}$ at
$B\!=\!256$ --- a $19\times$ reduction. This is the memory-bound-to-
compute-bound crossover: past $B^\ast$, weight cost is fully
amortized and per-token time asymptotes to the compute + KV floor
($\sim\!\SI{27}{ms}$). \textbf{At short context, batching directly
lowers cost per token.}

\begin{figure*}[h]
\centering
\includegraphics[width=\textwidth]{roofline_short_context.png}
\caption{Roofline decomposition for Llama-3.1-70B decode at
$S_{kv}\!=\!  \SI{8}{K}$ (short, well below $L^\ast\!\approx\!  \SI{13.4}{K}$).
\textbf{Left}: step latency vs.\ batch $B$; weight fetch is flat
(one HBM sweep per step), compute and KV grow linearly with $B$.
\textbf{Right}: per-token cost ($=$ step $\div B$); the $1/B$
weight-fetch term amortizes rapidly, dropping total per-token time
from \SI{562}{ms} at $B\!=\!1$ to \SI{29.7}{ms} at $B\!=\!256$
--- a $19\times$ throughput gain from batching alone.}
\label{fig:roofline-short}
\end{figure*}

\paragraph{Long-context regime ($S_{kv} \!\gg\! L^\ast$).}
Figure~\ref{fig:roofline-long} shows the same decomposition at
$S_{kv}\!=\!  \SI{1}{M}$ ($\sim\!78 \times L^\ast$). The step-latency
panel shows KV fetch has swelled by two orders of magnitude:
per-token KV cost is now $\sim\!\SI{1342}{ms}$, dwarfing both
weight ($\SI{535}{ms}$ at $B\!=\!1$) and compute ($\SI{17}{ms}$).
The cost-per-token panel is the punchline: increasing $B$ still
shrinks the weight term but leaves the giant KV floor untouched,
because \textbf{KV fetch is per-sequence} --- adding another user
adds a full extra copy of their KV read. Total per-token cost falls
only from \SI{1894}{ms} at $B\!=\!1$ to \SI{1361}{ms} at $B\!=\!256$
--- a mere $28\%$ reduction despite $256\times$ the batch.
\textbf{At long context, batching stops paying because per-user KV
streaming dominates.}

\begin{figure*}[h]
\centering
\includegraphics[width=\textwidth]{roofline_long_context.png}
\caption{Same decomposition at $S_{kv}\!=\!  \SI{1}{M}$ (long,
$\sim\!78\times L^\ast$). KV fetch has grown to \SI{1342}{ms}/token
and is now the dominant term at every $B$. Batching only shrinks
the weight component; the KV floor is unmovable because each new
user brings a full per-sequence KV read. Total per-token cost falls
just $28\%$ from $B\!=\!1$ to $B\!=\!256$, versus $19\times$ at
short context.}
\label{fig:roofline-long}
\end{figure*}

\paragraph{Implication for deployment.} The two regimes call for
opposite strategies. In the short-context regime, the operator packs
$B$ as high as HBM allows to sit on the compute floor --- this is
where cost-per-token is minimized and hardware utilization is
highest. In the long-context regime, per-user KV is the binding
resource; batching offers little benefit, so the operator instead
shrinks $\mathrm{kv\_bpt}$ (GQA / MQA / MLA, INT4 KV, sparse
attention) and shards the sequence dimension itself (CP), routing
long-context requests to a dedicated pool with more chips per user.
The next two subsections (\S\ref{sec:capacity-planning},
\S\ref{sec:parallelism-selection}) turn these regime observations
into concrete sizing and sharding rules.