\subsection{Capacity Planning: LLM Serving on AHBM}
\label{sec:capacity-planning}

Once a fused kernel is chosen, the deployment question is orthogonal:
\emph{how many cubes (or SIPs) does a given model, context length, and
latency SLO require?} Hyperscalers decide this along three axes that
must all be satisfied simultaneously. The total PE count is
$\max(A, B, C) \times N_{\text{replicas}}$, where $A$ is the capacity
floor, $B$ the KV-cache headroom, and $C$ the throughput SLO
(Table~\ref{tab:capacity-axes}).

\begin{table}[h]
\centering
\small
\caption{Three-axis sizing. $N$ = active params, $b$ = bytes per param,
$\mathrm{HBM}_{\mathrm{PE}}$ = per-PE HBM budget, $S_{kv}$ = context
length, $\mathrm{kv\_bpt}$ = KV cache bytes per token per user
($=2 \cdot h_{kv} \cdot d_{\text{head}} \cdot b \cdot L$).}
\label{tab:capacity-axes}
\begin{tabular}{@{}l l@{}}
\toprule
Axis & Formula \\
\midrule
A. Capacity floor & $\lceil N b / \mathrm{HBM}_{\mathrm{PE}} \rceil$ \\
B. KV headroom & $\lceil (N b + u \cdot S_{kv} \cdot \mathrm{kv\_bpt}) / \mathrm{HBM}_{\mathrm{PE}} \rceil$ \\
C. Throughput SLO & $N_{\text{replicas}} = \lceil n_{\text{users}} / B_{\mathrm{SLO}} \rceil$ \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{SLO targets.} The per-token latency budget $B_{\mathrm{SLO}}$
follows the use case. TTFT (Time To First Token) is dominated by
prefill; TPOT (Time Per Output Token) by one decode step
(Table~\ref{tab:slo-targets}).

\begin{table}[h]
\centering
\small
\caption{Typical SLO targets. Voice needs a tight per-token cadence;
chat balances both; batch is priced on throughput not latency.}
\label{tab:slo-targets}
\begin{tabular}{@{}l l l@{}}
\toprule
Workload & TTFT & TPOT \\
\midrule
Voice / real-time & 200--300 ms & 10--25 ms \\
Interactive chat & 300 ms -- 1 s & 20--50 ms \\
Batch / offline & seconds -- minutes & N/A \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Regime-aware rules of thumb.} Context length relative to the
machine's balance point $L^*=2N/(\mathrm{AI}\cdot\mathrm{kv\_bpt})$
determines which strategy applies. Below $L^*$ the deployment is
compute-bound and batch scales to $\sim\!2B^*$; above, KV streaming
dominates and $B$ must shrink (Table~\ref{tab:regime-rules}).

\begin{table*}[t]
\centering
\small
\caption{Deployment strategy by context regime.}
\label{tab:regime-rules}
\begin{tabular}{@{}l l l@{}}
\toprule
Regime & Batch & Cost/token \\
\midrule
Short ($S_{kv} < L^*$) & $B \approx 2 B^*$ & low \\
Long ($S_{kv} > L^*$) & smaller $B$, more CP & higher \\
Extreme ($S_{kv} \gg L^*$) & dedicated pool, disagg.\ prefill & much higher \\
\bottomrule
\end{tabular}
\end{table*}

\paragraph{Sample deployment templates.} A single base model is
typically routed to one of several sharding tiers depending on the
request's expected context length; the API gateway dispatches to the
appropriate replica pool (Table~\ref{tab:deploy-templates}).

\begin{table*}[t]
\centering
\small
\caption{Sample deployment tiers for one base model. TP is fixed to
match $h_{kv}$; CP grows with context; $B$ shrinks to hold TPOT.}
\label{tab:deploy-templates}
\begin{tabular}{@{}l c c c c@{}}
\toprule
Tier & CP$\times$TP & Cubes/repl. & Max $S_{kv}$ & Typical $B$ \\
\midrule
Small & $1 \times 8$ & 1 & 32k & 64 \\
Medium & $4 \times 8$ & 4 & 128k & 32 \\
Large & $32 \times 8$ & 32 & 1M & 4 \\
\bottomrule
\end{tabular}
\end{table*}

\paragraph{What to do when each axis binds.} The dominant axis dictates
the first lever to pull (Table~\ref{tab:axis-playbook}). Axes $A$ and
$C$ scale nearly linearly with hardware; axis $B$ is the one where
algorithmic techniques (GQA, MLA, INT4 KV, sparse attention) buy the
most because they attack $\mathrm{kv\_bpt}$ directly rather than adding
chips.

\begin{table*}[t]
\centering
\small
\caption{Playbook per binding axis.}
\label{tab:axis-playbook}
\begin{tabular}{@{}l l l@{}}
\toprule
Binding & First lever & Second lever \\
\midrule
A. Capacity & $\uparrow$ TP / PP & bigger-HBM chip; FP8 / INT4 weights \\
B. KV & $\uparrow$ CP; $\downarrow B$ & GQA / MQA / MLA; INT4 KV; sparse attn \\
C. Throughput & $\uparrow$ replicas (DP) & loosen SLO; smaller model; spec.\ decoding \\
\bottomrule
\end{tabular}
\end{table*}