mukesh
fc4747668e
Three new subsections at the top of §5, motivated by the deployment-sizing questions the analytical visualization tool exposes: - 5.1 Roofline Analysis — B*, L*, short vs long context batch effect (measured: 19× per-token cost reduction at short context, 28% at long) - 5.2 Capacity Planning — 3-axis sizing (weights / KV / SLO), SLO targets, regime rules, deployment templates, per-axis playbook - 5.3 Parallelism Selection — TP × CP × PP × DP × EP comparison, symptom-driven axis selection, add/stop criteria, misconceptions Restructure: - 05-gqa.tex trimmed to section header + intro - Existing 5.4-5.7 content (placement, short/long ctx, composite) moved to 05x-fused-kernel.tex - Summary extracted to 05z-summary.tex, cross-refs updated - main.tex \input order sets the requested subsection sequence - lmodern loaded to satisfy microtype font-expansion Two new figures generated from tests/analytical_visualization/chip_roofline: - roofline_short_context.png (S_kv=8K, batch wins) - roofline_long_context.png (S_kv=1M, batch stalls at KV floor) - Generator: tests/analytical_visualization/_gen_roofline_paper_figs.py 4 tables (regime rules, deployment templates, playbook, parallelism criteria) use table* so they span both columns without overflow. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
119 lines
4.1 KiB
TeX
119 lines
4.1 KiB
TeX
\subsection{Capacity Planning: LLM Serving on AHBM}
|
|
\label{sec:capacity-planning}
|
|
|
|
Once a fused kernel is chosen, the deployment question is orthogonal:
|
|
\emph{how many cubes (or SIPs) does a given model, context length, and
|
|
latency SLO require?} Hyperscalers decide this along three axes that
|
|
must all be satisfied simultaneously. The total PE count is
|
|
$\max(A, B, C) \times N_{\text{replicas}}$, where $A$ is the capacity
|
|
floor, $B$ the KV-cache headroom, and $C$ the throughput SLO
|
|
(Table~\ref{tab:capacity-axes}).
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\small
|
|
\caption{Three-axis sizing. $N$ = active params, $b$ = bytes per param,
|
|
$\mathrm{HBM}_{\mathrm{PE}}$ = per-PE HBM budget, $S_{kv}$ = context
|
|
length, $\mathrm{kv\_bpt}$ = KV cache bytes per token per user
|
|
($=2 \cdot h_{kv} \cdot d_{\text{head}} \cdot b \cdot L$).}
|
|
\label{tab:capacity-axes}
|
|
\begin{tabular}{@{}l l@{}}
|
|
\toprule
|
|
Axis & Formula \\
|
|
\midrule
|
|
A. Capacity floor & $\lceil N b / \mathrm{HBM}_{\mathrm{PE}} \rceil$ \\
|
|
B. KV headroom & $\lceil (N b + u \cdot S_{kv} \cdot \mathrm{kv\_bpt}) / \mathrm{HBM}_{\mathrm{PE}} \rceil$ \\
|
|
C. Throughput SLO & $N_{\text{replicas}} = \lceil n_{\text{users}} / B_{\mathrm{SLO}} \rceil$ \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\paragraph{SLO targets.} The per-token latency budget $B_{\mathrm{SLO}}$
|
|
follows the use case. TTFT (Time To First Token) is dominated by
|
|
prefill; TPOT (Time Per Output Token) by one decode step
|
|
(Table~\ref{tab:slo-targets}).
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\small
|
|
\caption{Typical SLO targets. Voice needs a tight per-token cadence;
|
|
chat balances both; batch is priced on throughput not latency.}
|
|
\label{tab:slo-targets}
|
|
\begin{tabular}{@{}l l l@{}}
|
|
\toprule
|
|
Workload & TTFT & TPOT \\
|
|
\midrule
|
|
Voice / real-time & 200--300 ms & 10--25 ms \\
|
|
Interactive chat & 300 ms -- 1 s & 20--50 ms \\
|
|
Batch / offline & seconds -- minutes & N/A \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\paragraph{Regime-aware rules of thumb.} Context length relative to the
|
|
machine's balance point $L^*=2N/(\mathrm{AI}\cdot\mathrm{kv\_bpt})$
|
|
determines which strategy applies. Below $L^*$ the deployment is
|
|
compute-bound and batch scales to $\sim\!2B^*$; above, KV streaming
|
|
dominates and $B$ must shrink (Table~\ref{tab:regime-rules}).
|
|
|
|
\begin{table*}[t]
|
|
\centering
|
|
\small
|
|
\caption{Deployment strategy by context regime.}
|
|
\label{tab:regime-rules}
|
|
\begin{tabular}{@{}l l l@{}}
|
|
\toprule
|
|
Regime & Batch & Cost/token \\
|
|
\midrule
|
|
Short ($S_{kv} < L^*$) & $B \approx 2 B^*$ & low \\
|
|
Long ($S_{kv} > L^*$) & smaller $B$, more CP & higher \\
|
|
Extreme ($S_{kv} \gg L^*$) & dedicated pool, disagg.\ prefill & much higher \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
|
|
\paragraph{Sample deployment templates.} A single base model is
|
|
typically routed to one of several sharding tiers depending on the
|
|
request's expected context length; the API gateway dispatches to the
|
|
appropriate replica pool (Table~\ref{tab:deploy-templates}).
|
|
|
|
\begin{table*}[t]
|
|
\centering
|
|
\small
|
|
\caption{Sample deployment tiers for one base model. TP is fixed to
|
|
match $h_{kv}$; CP grows with context; $B$ shrinks to hold TPOT.}
|
|
\label{tab:deploy-templates}
|
|
\begin{tabular}{@{}l c c c c@{}}
|
|
\toprule
|
|
Tier & CP$\times$TP & Cubes/repl. & Max $S_{kv}$ & Typical $B$ \\
|
|
\midrule
|
|
Small & $1 \times 8$ & 1 & 32k & 64 \\
|
|
Medium & $4 \times 8$ & 4 & 128k & 32 \\
|
|
Large & $32 \times 8$ & 32 & 1M & 4 \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
|
|
\paragraph{What to do when each axis binds.} The dominant axis dictates
|
|
the first lever to pull (Table~\ref{tab:axis-playbook}). Axes $A$ and
|
|
$C$ scale nearly linearly with hardware; axis $B$ is the one where
|
|
algorithmic techniques (GQA, MLA, INT4 KV, sparse attention) buy the
|
|
most because they attack $\mathrm{kv\_bpt}$ directly rather than adding
|
|
chips.
|
|
|
|
\begin{table*}[t]
|
|
\centering
|
|
\small
|
|
\caption{Playbook per binding axis.}
|
|
\label{tab:axis-playbook}
|
|
\begin{tabular}{@{}l l l@{}}
|
|
\toprule
|
|
Binding & First lever & Second lever \\
|
|
\midrule
|
|
A. Capacity & $\uparrow$ TP / PP & bigger-HBM chip; FP8 / INT4 weights \\
|
|
B. KV & $\uparrow$ CP; $\downarrow B$ & GQA / MQA / MLA; INT4 KV; sparse attn \\
|
|
C. Throughput & $\uparrow$ replicas (DP) & loosen SLO; smaller model; spec.\ decoding \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|