ywkang/kernbench2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gqa(prefill-4cases): add long-context prefill 4-cases comparative study

Browse Source 
Mirrors the existing decode 4-cases study for prefill on the LLaMA-3.1-70B
single-KV-head group (h_q=8, h_kv=1, d_head=128) across 8 cubes × 8 PEs:

  Case 1  Cube-SP × PE-TP  → KV S_kv-Ring across cubes; Q/O T_q-split 64-way
  Case 2  Cube-Repl × PE-TP → full KV per cube; only CUBE 0's P PEs split T_q
  Case 3  Cube-Repl × PE-SP → full KV per cube; PEs SP on S_kv (intra-cube AR)
  Case 4  Cube-SP × PE-SP  → KV split 64-way + Q T_q-split across cubes
                              (Ring KV + per-cube intra-CUBE AR)  ★ optimal

Cases 2 and 3 use Q-axis tiling (TILE_Q) so the per-PE scratch is bounded
by TILE_Q × TILE_S_KV regardless of T_q (the full Q tensor would be
G·T_q·d_head·2 bytes which scales linearly with T_q and blew the 1 MB
budget at T_q≥128 without tiling).

Per-panel try/except in run_sweep tolerates per-panel failures so
sweep.json always lands with whichever cases succeed plus a failures
list. Case 4 uses configure_sfr_intercube_ring with the snake submesh
(2,4) which installs both the Ring E/W lanes and the intra_* lanes
needed for the intra-CUBE reduce — single SFR config for all 4 cases.

The plot script generates 4 PNGs (latency, traffic, memory, parallelism)
into 1H_milestone_output/gqa/long_ctx/. The parallelism chart shows
total compute work (active_PE × T_q_per_PE × S_kv_processed) — exposes
Case 3's 8× redundancy vs. Cases 1/2/4 which all do unique work.

gqa_prefill_long_ctx_4cases.py exposes run_sweep() rather than a @bench
decorator — the umbrella bench in the next commit invokes it.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Mukesh Garg
2026-06-16 13:06:08 -07:00
parent e45626c036
commit 443ede99c7
7 changed files with 1735 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/kernbench/benches/gqa_helpers/long_ctx/_gqa_attention_prefill_long_ctx_cube_repl_pe_sp.py
+177
View File
@@ -0,0 +1,177 @@
"""GQA prefill kernel — Case 3 (Cube-Repl × PE-SP) at single-KV-head group.
Per GQA_full_deck.pptx slide 11 (prefill counterpart, T_q ≫ 1):
- K, V replicated across all 8 cubes (the 8× memory waste).
- PEs SP on S_kv inside each cube: each PE attends to its
``S_local = S_kv / P`` slice.
- Q replicated on every rank (every cube does full T_q × S_local).
- Intra-CUBE 8-way reduce on the partial ``(m, ℓ, O)`` triple
per Q-tile (row chain along intra_W + col bridge along intra_N
over the 2×4 PE grid).
- NO inter-CUBE communication — every cube ends with the full
answer (8× redundant compute is the inherent cost of Case 3).
- Designated writer: only PE 0 of CUBE 0 stores ``O`` to avoid
8 redundant DMA writes.
Q-axis tiling (vs decode Case 3): an outer Q-tile loop wraps the
entire per-tile pipeline (Q load → bootstrap → S_kv-tile fold →
intra-CUBE AR → store) in ``tl.scratch_scope``. Peak scratch is
bounded by ``TILE_Q × TILE_S_KV`` instead of ``T_q × S_kv``, so the
kernel runs at large T_q without exceeding the ~1 MB per-PE budget.
Each Q-tile's (m, ℓ, O) is independent of other Q-tiles, so no
state leaks across iterations.
Tensor layout:
Q : (T_q, h_q · d_head) replicated on every rank; loaded per
Q-tile as ``(TILE_Q, d_head)`` slices (G is folded into the
T_q row axis, total ``G·T_q`` rows).
K : (S_kv, h_kv · d_head) with cube=replicate, pe=row_wise — each
PE owns its ``(S_local, h_kv·d_head)`` shard within its cube.
V : same as K.
O : (T_q, h_q · d_head) — only PE 0 of CUBE 0 stores, one
``(TILE_Q, d_head)`` slice per outer Q-tile iteration.
Topology / SFR:
- Requires intra_* lanes; ``configure_sfr_intercube_ring`` (with the
snake submesh) or ``configure_sfr_intercube_multisip`` both install
them.
"""
from __future__ import annotations
TILE_S_KV = 64 # per-tile S_kv width (small to keep ``scores`` bounded).
TILE_Q = 64 # per-tile Q row count (outer-loop tile size).
def _merge_running(m_local, l_local, O_local, m_other, l_other, O_other, *, tl):
"""Online-softmax merge of two partial ``(m, ℓ, O)`` triples."""
m_new = tl.maximum(m_local, m_other)
scale_old = tl.exp(m_local - m_new)
scale_new = tl.exp(m_other - m_new)
l_new = l_local * scale_old + l_other * scale_new
O_new = O_local * scale_old + O_other * scale_new
return m_new, l_new, O_new
def gqa_attention_prefill_long_ctx_cube_repl_pe_sp_kernel(
q_ptr: int,
k_ptr: int,
v_ptr: int,
o_ptr: int,
T_q: int,
S_kv: int,
h_q: int,
h_kv: int,
d_head: int,
C: int,
P: int,
*,
tl,
) -> None:
"""Case-3 prefill: PE-SP S_kv split + Q-tiled intra-CUBE AR."""
G = h_q // h_kv
S_local = S_kv // P # each PE owns S_kv/P (cube=replicate, pe=row_wise)
pe_id = tl.program_id(axis=0)
cube_id = tl.program_id(axis=1)
PE_GRID_COLS = 4
pe_col = pe_id % PE_GRID_COLS
pe_row = pe_id // PE_GRID_COLS
pe_cols_used = min(PE_GRID_COLS, P)
pe_rows_used = (P + PE_GRID_COLS - 1) // PE_GRID_COLS
KV_ROW_BYTES = d_head * 2 # f16
Q_ROW_BYTES = d_head * 2 # G is folded into the row axis, not d_head
total_q_rows = G * T_q
n_q_tiles = (total_q_rows + TILE_Q - 1) // TILE_Q
n_kv_tiles = (S_local + TILE_S_KV - 1) // TILE_S_KV
# ── Outer Q-tile loop ──
# Per Q-tile: load Q-slice, run full S_kv attention, intra-CUBE AR,
# store. All scratch is recycled at outer-scope exit.
for q_idx in range(n_q_tiles):
q_start = q_idx * TILE_Q
q_rows = min(TILE_Q, total_q_rows - q_start)
with tl.scratch_scope():
Q = tl.load(q_ptr + q_start * Q_ROW_BYTES,
shape=(q_rows, d_head), dtype="f16")
# ── Bootstrap (S_kv tile 0) ──
# K_T, V, scores, exp_scores live in the OUTER (Q-tile)
# scope. They survive the inner S_kv-tile loop (each
# iteration's allocations are recycled) and are freed
# together when the outer scope exits.
tile_s0 = min(TILE_S_KV, S_local)
K_T = tl.load(k_ptr, shape=(d_head, tile_s0), dtype="f16")
V = tl.load(v_ptr, shape=(tile_s0, d_head), dtype="f16")
scores = tl.dot(Q, K_T)
m_local = tl.max(scores, axis=-1)
exp_scores = tl.exp(scores - m_local)
l_local = tl.sum(exp_scores, axis=-1)
O_local = tl.dot(exp_scores, V)
# ── S_kv tiles 1..N: fold via online-softmax merge ──
for tile_idx in range(1, n_kv_tiles):
tile_start = tile_idx * TILE_S_KV
tile_s = min(TILE_S_KV, S_local - tile_start)
with tl.scratch_scope():
K_T_t = tl.load(k_ptr + tile_start * KV_ROW_BYTES,
shape=(d_head, tile_s), dtype="f16")
V_t = tl.load(v_ptr + tile_start * KV_ROW_BYTES,
shape=(tile_s, d_head), dtype="f16")
scores_t = tl.dot(Q, K_T_t)
m_tile = tl.max(scores_t, axis=-1)
exp_scores_t = tl.exp(scores_t - m_tile)
l_tile = tl.sum(exp_scores_t, axis=-1)
O_tile = tl.dot(exp_scores_t, V_t)
m_new, l_new, O_new = _merge_running(
m_local, l_local, O_local,
m_tile, l_tile, O_tile, tl=tl,
)
tl.copy_to(m_local, m_new)
tl.copy_to(l_local, l_new)
tl.copy_to(O_local, O_new)
# ── Intra-CUBE 8-way reduce-to-PE0 (row chain + col bridge) ──
# (m_local, l_local, O_local) are still alive in the outer
# scope; the send/recv operate on them directly.
if pe_cols_used > 1:
if pe_col < pe_cols_used - 1:
with tl.scratch_scope():
m_other = tl.recv(dir="intra_E", shape=m_local.shape, dtype="f16")
l_other = tl.recv(dir="intra_E", shape=l_local.shape, dtype="f16")
O_other = tl.recv(dir="intra_E", shape=O_local.shape, dtype="f16")
m_new, l_new, O_new = _merge_running(
m_local, l_local, O_local,
m_other, l_other, O_other, tl=tl,
)
tl.copy_to(m_local, m_new)
tl.copy_to(l_local, l_new)
tl.copy_to(O_local, O_new)
if pe_col > 0:
tl.send(dir="intra_W", src=m_local)
tl.send(dir="intra_W", src=l_local)
tl.send(dir="intra_W", src=O_local)
if pe_col == 0 and pe_rows_used > 1:
if pe_row < pe_rows_used - 1:
with tl.scratch_scope():
m_other = tl.recv(dir="intra_S", shape=m_local.shape, dtype="f16")
l_other = tl.recv(dir="intra_S", shape=l_local.shape, dtype="f16")
O_other = tl.recv(dir="intra_S", shape=O_local.shape, dtype="f16")
m_new, l_new, O_new = _merge_running(
m_local, l_local, O_local,
m_other, l_other, O_other, tl=tl,
)
tl.copy_to(m_local, m_new)
tl.copy_to(l_local, l_new)
tl.copy_to(O_local, O_new)
if pe_row > 0:
tl.send(dir="intra_N", src=m_local)
tl.send(dir="intra_N", src=l_local)
tl.send(dir="intra_N", src=O_local)
# ── Store this Q-tile (designated writer: cube 0, PE 0) ──
if pe_id == 0 and cube_id == 0:
O_final = O_local / l_local
tl.store(o_ptr + q_start * Q_ROW_BYTES, O_final)
src/kernbench/benches/gqa_helpers/long_ctx/_gqa_attention_prefill_long_ctx_cube_repl_pe_tp.py
+128
View File
@@ -0,0 +1,128 @@
"""GQA prefill kernel — Case 2 (Cube-Repl × PE-TP) at single-KV-head group.
Per GQA_full_deck.pptx slide 11 (prefill counterpart, T_q ≫ 1):
- K, V replicated across all 8 cubes (the 8× memory waste — this is
the inherent cost Case 2 demonstrates). PE-replicate within each
cube too: every PE on every cube holds an HBM copy.
- PE-TP on T_q: only CUBE 0 has work; within CUBE 0 the P PEs split
the T_q axis disjointly. CUBEs 1..7 idle (slide-11 "PE-TP only
uses one cube; replicate is pure waste here").
- NO inter-rank communication (each active rank has full KV and
disjoint T_q rows — slide 11 lists comm cost as "none").
Distinction from decode Case 2: decode T_q=1 makes PE-TP wasteful
(only PE 0 of CUBE 0 works → 1 of 64 ranks active). Prefill T_q≫1
makes PE-TP useful — all P PEs of CUBE 0 work on disjoint T_q rows
(P of 64 ranks active; CUBEs 1..7 are pure memory waste).
Q-axis tiling: an outer Q-tile loop wraps the entire per-tile
pipeline (Q load → bootstrap → S_kv-tile fold → store) in
``tl.scratch_scope``. Peak scratch is bounded by
``TILE_Q × TILE_S_KV`` instead of ``T_q_local × S_kv``, so the
kernel runs at large T_q without exceeding the ~1 MB per-PE budget.
Each Q-tile is independent (disjoint output rows; no inter-tile state).
Tensor layout:
Q : (T_q, h_q · d_head) cube=replicate, pe=row_wise on T_q. Only
CUBE 0's PEs load — each owns ``G · T_q_local`` total rows with
``T_q_local = T_q / P``; loaded per Q-tile.
K : (S_kv, h_kv · d_head) replicated on every rank.
V : (S_kv, h_kv · d_head) replicated on every rank.
O : (T_q, h_q · d_head) cube=replicate, pe=row_wise on T_q —
only CUBE 0's PEs write their disjoint ``T_q_local`` rows,
one ``(TILE_Q, d_head)`` slice per outer iteration.
Topology / SFR:
- SFR setup is benign (no inter-rank sends/recvs); any of the
``configure_sfr_intercube_*`` helpers works.
Requires ``T_q % P == 0``.
"""
from __future__ import annotations
TILE_S_KV = 64 # per-tile S_kv width (small to keep ``scores`` bounded).
TILE_Q = 64 # per-tile Q row count (outer-loop tile size).
def gqa_attention_prefill_long_ctx_cube_repl_pe_tp_kernel(
q_ptr: int,
k_ptr: int,
v_ptr: int,
o_ptr: int,
T_q: int,
S_kv: int,
h_q: int,
h_kv: int,
d_head: int,
C: int,
P: int,
*,
tl,
) -> None:
"""Case-2 prefill: CUBE 0 only; Q-tiled PE-TP; full KV per rank; no comm."""
if T_q % P != 0:
raise ValueError(
f"Case 2 prefill requires T_q={T_q} divisible by P={P}"
)
cube_id = tl.program_id(axis=1)
# CUBE 0 only — CUBEs 1..7 are pure memory-replication waste, no work.
if cube_id != 0:
return
G = h_q // h_kv
T_q_local = T_q // P
n_kv_tiles = (S_kv + TILE_S_KV - 1) // TILE_S_KV
KV_ROW_BYTES = d_head * 2 # f16
Q_ROW_BYTES = d_head * 2
total_q_rows = G * T_q_local
n_q_tiles = (total_q_rows + TILE_Q - 1) // TILE_Q
# ── Outer Q-tile loop ──
# Per Q-tile: load Q-slice, run full S_kv attention, store. All
# scratch is recycled at outer-scope exit, so peak usage is bounded
# by TILE_Q × TILE_S_KV.
for q_idx in range(n_q_tiles):
q_start = q_idx * TILE_Q
q_rows = min(TILE_Q, total_q_rows - q_start)
with tl.scratch_scope():
Q = tl.load(q_ptr + q_start * Q_ROW_BYTES,
shape=(q_rows, d_head), dtype="f16")
# ── Bootstrap (S_kv tile 0) ──
tile_s0 = min(TILE_S_KV, S_kv)
K_T = tl.load(k_ptr, shape=(d_head, tile_s0), dtype="f16")
V = tl.load(v_ptr, shape=(tile_s0, d_head), dtype="f16")
scores = tl.dot(Q, K_T)
m_local = tl.max(scores, axis=-1)
exp_scores = tl.exp(scores - m_local)
l_local = tl.sum(exp_scores, axis=-1)
O_local = tl.dot(exp_scores, V)
# ── S_kv tiles 1..N: fold via online-softmax merge ──
for tile_idx in range(1, n_kv_tiles):
tile_start = tile_idx * TILE_S_KV
tile_s = min(TILE_S_KV, S_kv - tile_start)
with tl.scratch_scope():
K_T_t = tl.load(k_ptr + tile_start * KV_ROW_BYTES,
shape=(d_head, tile_s), dtype="f16")
V_t = tl.load(v_ptr + tile_start * KV_ROW_BYTES,
shape=(tile_s, d_head), dtype="f16")
scores_t = tl.dot(Q, K_T_t)
m_tile = tl.max(scores_t, axis=-1)
exp_scores_t = tl.exp(scores_t - m_tile)
l_tile = tl.sum(exp_scores_t, axis=-1)
O_tile = tl.dot(exp_scores_t, V_t)
m_new = tl.maximum(m_local, m_tile)
scale_old = tl.exp(m_local - m_new)
scale_new = tl.exp(m_tile - m_new)
l_new = l_local * scale_old + l_tile * scale_new
O_new = O_local * scale_old + O_tile * scale_new
tl.copy_to(m_local, m_new)
tl.copy_to(l_local, l_new)
tl.copy_to(O_local, O_new)
# ── Store this Q-tile (each active PE writes its slice) ──
O_final = O_local / l_local
tl.store(o_ptr + q_start * Q_ROW_BYTES, O_final)
src/kernbench/benches/gqa_helpers/long_ctx/_gqa_attention_prefill_long_ctx_cube_sp_pe_sp.py
+186
View File
@@ -0,0 +1,186 @@
"""GQA prefill kernel — Case 4 (Cube-SP × PE-SP) at single-KV-head group. ★
Per GQA_full_deck.pptx slide 11 (prefill counterpart, T_q ≫ 1):
- K, V split 64-way (cube=row_wise, pe=row_wise on S_kv); each rank
owns ``S_local_pe = S_kv / (C·P)``.
- Q, O split T_q-wise across cubes (cube=row_wise on T_q),
pe=replicate within each cube — each cube owns
``T_q_cube = T_q / C`` rows, all P PEs of the cube hold a copy.
- Inter-CUBE Ring KV (snake over 4×2 sub-mesh, ADR-0060 §5.5):
P parallel rings — each PE drives its own same-lane ring via
independent IPCQ channels. Over C ring steps each PE has
processed its strided lane of S_kv (= ``S_kv / P`` tokens) for
the cube's T_q_cube query rows.
- Intra-CUBE 8-way reduce on the partial ``(m, ℓ, O)`` triple
(row chain along intra_W + col bridge along intra_N over the
2×4 PE grid) — combines the P PE-lanes per cube so cube's PE 0
holds the full S_kv attention for its T_q_cube rows.
- NO inter-CUBE reduce — each cube writes its own disjoint
T_q_cube rows of O (PE 0 of each cube is the writer).
Distinction from decode Case 4: decode T_q=1 forces an inter-CUBE
lrab reduce-to-root over (m,ℓ,O). Prefill T_q≫1 allows Q to be
T_q-sharded across cubes, so disjoint output rows avoid the
inter-cube reduce — the prefill-native pattern (per user's
"Ring KV" choice).
Tensor layout:
Q : (T_q, h_q · d_head) sharded cube=row_wise on T_q, pe=replicate
within cube; each rank loads ``(G · T_q_cube, d_head)``.
K : (S_kv, h_kv · d_head) cube=row_wise + pe=row_wise on S_kv;
each rank owns ``(S_local_pe, h_kv·d_head)``.
V : same as K.
O : (T_q, h_q · d_head) sharded same as Q — PE 0 of each cube
writes its ``(G · T_q_cube, d_head)`` slice; PEs 1..P-1 idle
the write.
Topology / SFR:
- Requires ``configure_sfr_intercube_ring(submesh_shape=(2, 4))`` —
that helper installs both the snake E/W ring across 8 cubes and
the intra_* lanes needed for the intra-CUBE reduce.
Requires ``T_q % C == 0`` and ``S_kv % (C · P) == 0``.
"""
from __future__ import annotations
TILE_S_KV = 1024 # ADR-0063 §A.2 S_kv-axis tile sweep (per-tile width).
def _merge_running(m_local, l_local, O_local, m_other, l_other, O_other, *, tl):
"""Online-softmax merge of two partial ``(m, ℓ, O)`` triples."""
m_new = tl.maximum(m_local, m_other)
scale_old = tl.exp(m_local - m_new)
scale_new = tl.exp(m_other - m_new)
l_new = l_local * scale_old + l_other * scale_new
O_new = O_local * scale_old + O_other * scale_new
return m_new, l_new, O_new
def gqa_attention_prefill_long_ctx_cube_sp_pe_sp_kernel(
q_ptr: int,
k_ptr: int,
v_ptr: int,
o_ptr: int,
T_q: int,
S_kv: int,
h_q: int,
h_kv: int,
d_head: int,
C: int,
P: int,
*,
tl,
) -> None:
"""Case-4 prefill: Ring KV across cubes + intra-CUBE AR; disjoint T_q/C rows per cube."""
if T_q % C != 0:
raise ValueError(
f"Case 4 prefill requires T_q={T_q} divisible by C={C}"
)
if S_kv % (C * P) != 0:
raise ValueError(
f"Case 4 prefill requires S_kv={S_kv} divisible by C·P={C * P}"
)
G = h_q // h_kv
T_q_cube = T_q // C
S_local_pe = S_kv // (C * P)
KV_ROW_BYTES = d_head * 2 # f16
n_tiles = (S_local_pe + TILE_S_KV - 1) // TILE_S_KV
pe_id = tl.program_id(axis=0)
# Q is this cube's T_q slice (pe=replicate within cube ⇒ every PE
# loads the same G·T_q_cube rows).
Q = tl.load(q_ptr, shape=(G * T_q_cube, d_head), dtype="f16")
# ── Bootstrap (t=0, k=0): load own rank's tile 0; send W for ring ──
# Persistent (m, ℓ, O) must live OUTSIDE tl.scratch_scope (kernbench
# scope teardown discards in-scope tensors).
tile_s = min(TILE_S_KV, S_local_pe)
K_T = tl.load(k_ptr, shape=(d_head, tile_s), dtype="f16")
V = tl.load(v_ptr, shape=(tile_s, d_head), dtype="f16")
if C > 1:
tl.send(dir="W", src=K_T)
tl.send(dir="W", src=V)
scores = tl.dot(Q, K_T)
m_local = tl.max(scores, axis=-1)
exp_scores = tl.exp(scores - m_local)
l_local = tl.sum(exp_scores, axis=-1)
O_local = tl.dot(exp_scores, V)
# ── Nested loop (outer tile, inner ring step) — P parallel rings ──
for t in range(n_tiles):
tile_start = t * TILE_S_KV
tile_s = min(TILE_S_KV, S_local_pe - tile_start)
k_start = 1 if t == 0 else 0
for k in range(k_start, C):
with tl.scratch_scope():
if k == 0:
K_T_t = tl.load(k_ptr + tile_start * KV_ROW_BYTES,
shape=(d_head, tile_s), dtype="f16")
V_t = tl.load(v_ptr + tile_start * KV_ROW_BYTES,
shape=(tile_s, d_head), dtype="f16")
else:
K_T_t = tl.recv(dir="E", shape=(d_head, tile_s), dtype="f16")
V_t = tl.recv(dir="E", shape=(tile_s, d_head), dtype="f16")
if k < C - 1:
tl.send(dir="W", src=K_T_t)
tl.send(dir="W", src=V_t)
scores_t = tl.dot(Q, K_T_t)
m_step = tl.max(scores_t, axis=-1)
exp_scores_t = tl.exp(scores_t - m_step)
l_step = tl.sum(exp_scores_t, axis=-1)
O_step = tl.dot(exp_scores_t, V_t)
m_new, l_new, O_new = _merge_running(
m_local, l_local, O_local, m_step, l_step, O_step, tl=tl,
)
tl.copy_to(m_local, m_new)
tl.copy_to(l_local, l_new)
tl.copy_to(O_local, O_new)
# ── Intra-CUBE 8-way reduce-to-PE0 (row chain + col bridge) ──
PE_GRID_COLS = 4
pe_col = pe_id % PE_GRID_COLS
pe_row = pe_id // PE_GRID_COLS
pe_cols_used = min(PE_GRID_COLS, P)
pe_rows_used = (P + PE_GRID_COLS - 1) // PE_GRID_COLS
if pe_cols_used > 1:
if pe_col < pe_cols_used - 1:
with tl.scratch_scope():
m_other = tl.recv(dir="intra_E", shape=m_local.shape, dtype="f16")
l_other = tl.recv(dir="intra_E", shape=l_local.shape, dtype="f16")
O_other = tl.recv(dir="intra_E", shape=O_local.shape, dtype="f16")
m_new, l_new, O_new = _merge_running(
m_local, l_local, O_local, m_other, l_other, O_other, tl=tl,
)
tl.copy_to(m_local, m_new)
tl.copy_to(l_local, l_new)
tl.copy_to(O_local, O_new)
if pe_col > 0:
tl.send(dir="intra_W", src=m_local)
tl.send(dir="intra_W", src=l_local)
tl.send(dir="intra_W", src=O_local)
if pe_col == 0 and pe_rows_used > 1:
if pe_row < pe_rows_used - 1:
with tl.scratch_scope():
m_other = tl.recv(dir="intra_S", shape=m_local.shape, dtype="f16")
l_other = tl.recv(dir="intra_S", shape=l_local.shape, dtype="f16")
O_other = tl.recv(dir="intra_S", shape=O_local.shape, dtype="f16")
m_new, l_new, O_new = _merge_running(
m_local, l_local, O_local, m_other, l_other, O_other, tl=tl,
)
tl.copy_to(m_local, m_new)
tl.copy_to(l_local, l_new)
tl.copy_to(O_local, O_new)
if pe_row > 0:
tl.send(dir="intra_N", src=m_local)
tl.send(dir="intra_N", src=l_local)
tl.send(dir="intra_N", src=O_local)
# ── Final normalise + store (PE 0 of each cube writes its T_q_cube rows) ──
if pe_id == 0:
O_final = O_local / l_local
tl.store(o_ptr, O_final)
src/kernbench/benches/gqa_helpers/long_ctx/_gqa_attention_prefill_long_ctx_cube_sp_pe_tp.py
+136
View File
@@ -0,0 +1,136 @@
"""GQA prefill kernel — Case 1 (Cube-SP × PE-TP) at single-KV-head group.
Per GQA_full_deck.pptx slide 11 (prefill counterpart, T_q ≫ 1):
- K, V split S_kv-wise across the 8 cubes (cube=row_wise);
replicated within each cube (pe=replicate). Each cube owns
``S_local = S_kv / C``.
- Q, O split T_q-wise across cubes AND across PEs intra-cube
(cube=row_wise, pe=row_wise on T_q). Each rank owns disjoint
``T_q_local = T_q / (C · P)`` query rows.
- Inter-CUBE Ring KV (snake over 4×2 sub-mesh, ADR-0060 §5.5):
P parallel rings — each PE drives its own same-lane ring via
independent IPCQ channels. Over C ring steps the cube-owned
K/V blocks rotate through all C cubes, so every rank sees
every block.
- NO inter-CUBE reduce — each rank writes its own disjoint
T_q_local rows of O.
- NO intra-CUBE reduce — PEs are disjoint on T_q.
Distinction from decode Case 1: decode T_q=1 makes PE-TP wasteful
(only PE 0 of each cube works). Prefill T_q≫1 makes PE-TP useful —
all 64 ranks work on disjoint Q rows.
Tensor layout:
Q : (T_q, h_q · d_head) sharded (cube_row_wise, pe_row_wise) on T_q;
each rank loads ``(G · T_q_local, d_head)``.
K : (S_kv, h_kv · d_head) with cube=row_wise, pe=replicate — each
cube's PE 0..P-1 each hold an HBM copy of the cube's
``(S_local, h_kv·d_head)`` shard.
V : same as K.
O : (T_q, h_q · d_head) sharded same as Q — each rank writes its
own ``(G · T_q_local, d_head)`` slice.
Topology / SFR:
- Requires ``configure_sfr_intercube_ring(submesh_shape=(2, 4))``
for the snake E/W ring across 8 cubes (wrap from cube 3 ↔ cube 7).
Requires ``T_q % (C · P) == 0``.
"""
from __future__ import annotations
TILE_S_KV = 1024 # ADR-0063 §A.2 S_kv-axis tile sweep (per-tile width).
def _merge_running(m_local, l_local, O_local, m_other, l_other, O_other, *, tl):
"""Online-softmax merge of two partial ``(m, ℓ, O)`` triples."""
m_new = tl.maximum(m_local, m_other)
scale_old = tl.exp(m_local - m_new)
scale_new = tl.exp(m_other - m_new)
l_new = l_local * scale_old + l_other * scale_new
O_new = O_local * scale_old + O_other * scale_new
return m_new, l_new, O_new
def gqa_attention_prefill_long_ctx_cube_sp_pe_tp_kernel(
q_ptr: int,
k_ptr: int,
v_ptr: int,
o_ptr: int,
T_q: int,
S_kv: int,
h_q: int,
h_kv: int,
d_head: int,
C: int,
P: int,
*,
tl,
) -> None:
"""Case-1 prefill: Ring KV across cubes; disjoint T_q rows per rank."""
if T_q % (C * P) != 0:
raise ValueError(
f"Case 1 prefill requires T_q={T_q} divisible by C·P={C * P}"
)
G = h_q // h_kv
T_q_local = T_q // (C * P)
S_local = S_kv // C
KV_ROW_BYTES = d_head * 2 # f16
n_tiles = (S_local + TILE_S_KV - 1) // TILE_S_KV
# Q is this rank's own disjoint T_q slice (G·T_q_local rows for the
# G-grouped attention).
Q = tl.load(q_ptr, shape=(G * T_q_local, d_head), dtype="f16")
# ── Bootstrap (t=0, k=0): load own cube's tile 0; send W for ring ──
# Persistent (m, ℓ, O) must live OUTSIDE tl.scratch_scope (kernbench
# scope teardown discards in-scope tensors). Mirrors decode Cases
# 1, 3, 4 and the existing prefill_long bootstrap pattern.
tile_s = min(TILE_S_KV, S_local)
K_T = tl.load(k_ptr, shape=(d_head, tile_s), dtype="f16")
V = tl.load(v_ptr, shape=(tile_s, d_head), dtype="f16")
if C > 1:
tl.send(dir="W", src=K_T)
tl.send(dir="W", src=V)
scores = tl.dot(Q, K_T)
m_local = tl.max(scores, axis=-1)
exp_scores = tl.exp(scores - m_local)
l_local = tl.sum(exp_scores, axis=-1)
O_local = tl.dot(exp_scores, V)
# ── Nested loop (outer tile, inner ring step) ──
# Each tile propagates around the ring before the next tile starts;
# IPCQ in-flight depth stays at 1 per direction per PE-lane.
for t in range(n_tiles):
tile_start = t * TILE_S_KV
tile_s = min(TILE_S_KV, S_local - tile_start)
k_start = 1 if t == 0 else 0
for k in range(k_start, C):
with tl.scratch_scope():
if k == 0:
K_T_t = tl.load(k_ptr + tile_start * KV_ROW_BYTES,
shape=(d_head, tile_s), dtype="f16")
V_t = tl.load(v_ptr + tile_start * KV_ROW_BYTES,
shape=(tile_s, d_head), dtype="f16")
else:
K_T_t = tl.recv(dir="E", shape=(d_head, tile_s), dtype="f16")
V_t = tl.recv(dir="E", shape=(tile_s, d_head), dtype="f16")
if k < C - 1:
tl.send(dir="W", src=K_T_t)
tl.send(dir="W", src=V_t)
scores_t = tl.dot(Q, K_T_t)
m_step = tl.max(scores_t, axis=-1)
exp_scores_t = tl.exp(scores_t - m_step)
l_step = tl.sum(exp_scores_t, axis=-1)
O_step = tl.dot(exp_scores_t, V_t)
m_new, l_new, O_new = _merge_running(
m_local, l_local, O_local, m_step, l_step, O_step, tl=tl,
)
tl.copy_to(m_local, m_new)
tl.copy_to(l_local, l_new)
tl.copy_to(O_local, O_new)
# ── Final normalise + store (every rank writes its own T_q_local rows) ──
O_final = O_local / l_local
tl.store(o_ptr, O_final)
src/kernbench/benches/gqa_helpers/long_ctx/gqa_prefill_long_ctx_4cases.py
+384
View File
@@ -0,0 +1,384 @@
"""milestone-gqa-prefill-long-ctx-4cases: long-context prefill 4-cases study.
Prefill counterpart to milestone-gqa-decode-long-ctx-4cases. Per
GQA_full_deck.pptx slides 11-17: 4 KV-cache sharding strategies on
the LLaMA-3.1-70B single-KV-head group (1 KV head, 8 Q heads,
d_head=128) on 8 cubes × 8 PEs.
Bench size T_q=S_kv=512 (equal — prompt length matches K/V length
for prefill). Cases 2 and 3 use Q-axis tiling (``TILE_Q``) so the
per-PE scratch is bounded by ``TILE_Q × TILE_S_KV`` regardless of
T_q — much larger sizes are possible at the cost of bench
wall-clock (Case 2's single-cube serial path dominates).
Case 1 Cube-SP / PE-TP → KV split S_kv across cubes (Ring),
Q/O split T_q across all 64 ranks
Case 2 Cube-Repl / PE-TP → full KV per cube; only CUBE 0's P PEs
split T_q; CUBEs 1..7 are pure memory waste
Case 3 Cube-Repl / PE-SP → full KV per cube; PEs SP on S_kv
(intra-cube AR); 8× redundant compute
Case 4 Cube-SP / PE-SP → KV split 64-way + Q split T_q across cubes;
Ring KV + intra-cube AR per cube ★ optimal
Each case is a separate panel. The bench drives all panels in one
invocation and writes per-panel op_log_summary + latency to sweep.json
so the comparative analysis (latency, comm volume, parallelism,
memory) can be generated from a single sweep.
Distinction from decode: decode T_q=1 makes PE-TP cases wasteful (1
active PE). Prefill T_q≫1 makes PE-TP useful (full parallelism). The
narrative differs — the figures emphasise Case 2/3 memory waste and
Case 3 compute redundancy, not PE-TP-at-B=1 wasted ranks.
Per-panel try/except in ``run()`` writes whichever cases succeed —
the bench tolerates per-panel scratch / SFR failures so sweep.json
always lands with at least the successful rows + a ``failures`` list.
Gated by ``GQA_PREFILL_LONG_CTX_4CASES_RUN=1``.
"""
from __future__ import annotations
import json
import os
from pathlib import Path
from kernbench.benches.gqa_helpers.long_ctx._gqa_attention_prefill_long_ctx_cube_repl_pe_sp import (
gqa_attention_prefill_long_ctx_cube_repl_pe_sp_kernel,
)
from kernbench.benches.gqa_helpers.long_ctx._gqa_attention_prefill_long_ctx_cube_repl_pe_tp import (
gqa_attention_prefill_long_ctx_cube_repl_pe_tp_kernel,
)
from kernbench.benches.gqa_helpers.long_ctx._gqa_attention_prefill_long_ctx_cube_sp_pe_sp import (
gqa_attention_prefill_long_ctx_cube_sp_pe_sp_kernel,
)
from kernbench.benches.gqa_helpers.long_ctx._gqa_attention_prefill_long_ctx_cube_sp_pe_tp import (
gqa_attention_prefill_long_ctx_cube_sp_pe_tp_kernel,
)
from kernbench.benches.gqa_helpers.shared._gqa_panel_helpers import (
_ccl_cfg,
_summarize_op_log,
)
from kernbench.ccl.sfr_config import configure_sfr_intercube_ring
from kernbench.policy.placement.dp import DPPolicy
# File is at benches/gqa_helpers/long_ctx/ — go up 2 parents to reach
# benches/, then into 1H_milestone_output/gqa/long_ctx/.
_OUTPUT_DIR = (
Path(__file__).resolve().parents[2]
/ "1H_milestone_output" / "gqa" / "long_ctx"
)
_SWEEP_JSON = _OUTPUT_DIR / "sweep_prefill.json"
# ── Panel registry ───────────────────────────────────────────────────
_PANELS = (
"single_kv_group_prefill_long_ctx_gqa_cube_sp_pe_sp", # Case 4 ★ optimal
"single_kv_group_prefill_long_ctx_gqa_cube_repl_pe_tp", # Case 2
"single_kv_group_prefill_long_ctx_gqa_cube_repl_pe_sp", # Case 3
"single_kv_group_prefill_long_ctx_gqa_cube_sp_pe_tp", # Case 1
)
# Each entry: (kind, panel-specific params).
# LLaMA-3.1-70B single-KV-head group target:
# 1 KV head, h_q = 8 (G = 8 group), d_head = 128
# 8 cubes (head-parallel group), 8 PEs/cube
# T_q = S_kv = 4096 (long-context prefill)
_PANEL_DISPATCH: dict[str, tuple[str, dict]] = {
"single_kv_group_prefill_long_ctx_gqa_cube_sp_pe_sp": ("prefill_long_ctx_cube_sp_pe_sp", {
# Case 4: KV split 64-way + Q split T_q across cubes; Ring KV
# + intra-CUBE AR per cube. PE 0 of each cube writes its
# T_q/C disjoint output rows.
"C": 8, "P": 8,
"T_q": 512, "S_kv": 512,
"d_head": 128, "h_q": 8, "h_kv": 1,
}),
"single_kv_group_prefill_long_ctx_gqa_cube_repl_pe_tp": ("prefill_long_ctx_cube_repl_pe_tp", {
# Case 2: K, V replicated everywhere (8× memory waste); within
# CUBE 0 the P PEs split T_q disjointly. CUBEs 1..7 idle.
# No inter-rank communication.
"C": 8, "P": 8,
"T_q": 512, "S_kv": 512,
"d_head": 128, "h_q": 8, "h_kv": 1,
}),
"single_kv_group_prefill_long_ctx_gqa_cube_repl_pe_sp": ("prefill_long_ctx_cube_repl_pe_sp", {
# Case 3: K, V replicated per cube (8× memory); PEs SP on S_kv
# within each cube. Intra-CUBE 8-way reduce; no inter-CUBE comm
# (every cube does redundant T_q × S_kv compute; designated
# writer = cube 0).
"C": 8, "P": 8,
"T_q": 512, "S_kv": 512,
"d_head": 128, "h_q": 8, "h_kv": 1,
}),
"single_kv_group_prefill_long_ctx_gqa_cube_sp_pe_tp": ("prefill_long_ctx_cube_sp_pe_tp", {
# Case 1: K, V split S_kv across cubes (S_local = S_kv/C per
# cube); Q/O split T_q across all 64 ranks. Inter-CUBE snake
# Ring KV (P parallel rings); no intra-CUBE comm.
"C": 8, "P": 8,
"T_q": 512, "S_kv": 512,
"d_head": 128, "h_q": 8, "h_kv": 1,
}),
}
# ── Per-panel runner ─────────────────────────────────────────────────
def _ring_sfr(ctx):
"""Install snake Ring SFR (also provides intra_* lanes for AR cases).
Snake submesh of shape (2, 4) over the default 4×4 cube mesh
yields a Hamiltonian cycle through all 8 cubes
(0→1→2→3→7→6→5→4→0), with intra_* lanes available for the
Case 3/4 intra-CUBE reduce.
"""
configure_sfr_intercube_ring(
ctx.engine, ctx.spec, _ccl_cfg(),
submesh_shape=(2, 4),
)
def _run_prefill_panel_long_ctx_cube_repl_pe_tp(
ctx, *, panel: str, C: int, P: int,
T_q: int, S_kv: int,
d_head: int, h_q: int, h_kv: int,
) -> None:
"""Case 2 runner: K, V replicated everywhere; CUBE 0 PE-TP on T_q.
DPPolicy models the cluster-wide 8× memory waste — every rank
holds full K, V in its HBM region. Within CUBE 0, P PEs split
T_q (so P of 64 ranks actually compute). CUBEs 1..7 idle.
"""
_ring_sfr(ctx)
dp_repl = DPPolicy(cube="replicate", pe="replicate",
num_cubes=C, num_pes=P)
dp_qo = DPPolicy(cube="replicate", pe="row_wise",
num_cubes=C, num_pes=P)
q = ctx.zeros((T_q, h_q * d_head),
dtype="f16", dp=dp_qo, name=f"{panel}_q")
k = ctx.zeros((S_kv, h_kv * d_head),
dtype="f16", dp=dp_repl, name=f"{panel}_k")
v = ctx.zeros((S_kv, h_kv * d_head),
dtype="f16", dp=dp_repl, name=f"{panel}_v")
o = ctx.empty((T_q, h_q * d_head),
dtype="f16", dp=dp_qo, name=f"{panel}_o")
ctx.launch(
panel, gqa_attention_prefill_long_ctx_cube_repl_pe_tp_kernel,
q, k, v, o,
T_q, S_kv, h_q, h_kv, d_head, C, P,
_auto_dim_remap=False,
)
def _run_prefill_panel_long_ctx_cube_repl_pe_sp(
ctx, *, panel: str, C: int, P: int,
T_q: int, S_kv: int,
d_head: int, h_q: int, h_kv: int,
) -> None:
"""Case 3 runner: K, V replicated per cube; PEs SP on S_kv intra-cube.
DPPolicy models the cluster-wide 8× memory waste — every cube
holds full K, V; intra-cube splits the S_kv axis row_wise across
8 PEs. Every rank holds full Q (cube=replicate, pe=replicate).
The kernel does an intra-CUBE 8-way reduce on (m, ℓ, O); only
cube 0's PE 0 writes the output.
"""
_ring_sfr(ctx)
dp_full = DPPolicy(cube="replicate", pe="replicate",
num_cubes=C, num_pes=P)
dp_kv = DPPolicy(cube="replicate", pe="row_wise",
num_cubes=C, num_pes=P)
q = ctx.zeros((T_q, h_q * d_head),
dtype="f16", dp=dp_full, name=f"{panel}_q")
k = ctx.zeros((S_kv, h_kv * d_head),
dtype="f16", dp=dp_kv, name=f"{panel}_k")
v = ctx.zeros((S_kv, h_kv * d_head),
dtype="f16", dp=dp_kv, name=f"{panel}_v")
o = ctx.empty((T_q, h_q * d_head),
dtype="f16", dp=dp_full, name=f"{panel}_o")
ctx.launch(
panel, gqa_attention_prefill_long_ctx_cube_repl_pe_sp_kernel,
q, k, v, o,
T_q, S_kv, h_q, h_kv, d_head, C, P,
_auto_dim_remap=False,
)
def _run_prefill_panel_long_ctx_cube_sp_pe_tp(
ctx, *, panel: str, C: int, P: int,
T_q: int, S_kv: int,
d_head: int, h_q: int, h_kv: int,
) -> None:
"""Case 1 runner: K, V split S_kv across cubes (Ring); Q/O split T_q 64-way.
DPPolicy: K, V are cube=row_wise on S_kv (S_local = S_kv/C per
cube), pe=replicate within cube. Q, O are cube=row_wise +
pe=row_wise on T_q — every rank owns T_q/(C·P) disjoint Q rows.
The kernel runs P parallel Rings through the snake submesh
(each PE has its own IPCQ lane); no intra-CUBE reduce.
"""
_ring_sfr(ctx)
dp_qo = DPPolicy(cube="row_wise", pe="row_wise",
num_cubes=C, num_pes=P)
dp_kv = DPPolicy(cube="row_wise", pe="replicate",
num_cubes=C, num_pes=P)
q = ctx.zeros((T_q, h_q * d_head),
dtype="f16", dp=dp_qo, name=f"{panel}_q")
k = ctx.zeros((S_kv, h_kv * d_head),
dtype="f16", dp=dp_kv, name=f"{panel}_k")
v = ctx.zeros((S_kv, h_kv * d_head),
dtype="f16", dp=dp_kv, name=f"{panel}_v")
o = ctx.empty((T_q, h_q * d_head),
dtype="f16", dp=dp_qo, name=f"{panel}_o")
ctx.launch(
panel, gqa_attention_prefill_long_ctx_cube_sp_pe_tp_kernel,
q, k, v, o,
T_q, S_kv, h_q, h_kv, d_head, C, P,
_auto_dim_remap=False,
)
def _run_prefill_panel_long_ctx_cube_sp_pe_sp(
ctx, *, panel: str, C: int, P: int,
T_q: int, S_kv: int,
d_head: int, h_q: int, h_kv: int,
) -> None:
"""Case 4 runner: K, V split 64-way; Q split T_q across cubes.
DPPolicy: K, V are cube=row_wise + pe=row_wise on S_kv — each
rank owns S_local_pe = S_kv/(C·P). Q, O are cube=row_wise on
T_q (T_q_cube = T_q/C per cube), pe=replicate within cube. Ring
KV through the snake submesh (P parallel lanes per cube); after
the ring, an intra-CUBE 8-way AR combines PE lanes so cube's
PE 0 has the full attention for its T_q_cube rows. PE 0 of each
cube writes its disjoint output rows (no inter-cube reduce).
"""
_ring_sfr(ctx)
dp_qo = DPPolicy(cube="row_wise", pe="replicate",
num_cubes=C, num_pes=P)
dp_kv = DPPolicy(cube="row_wise", pe="row_wise",
num_cubes=C, num_pes=P)
q = ctx.zeros((T_q, h_q * d_head),
dtype="f16", dp=dp_qo, name=f"{panel}_q")
k = ctx.zeros((S_kv, h_kv * d_head),
dtype="f16", dp=dp_kv, name=f"{panel}_k")
v = ctx.zeros((S_kv, h_kv * d_head),
dtype="f16", dp=dp_kv, name=f"{panel}_v")
o = ctx.empty((T_q, h_q * d_head),
dtype="f16", dp=dp_qo, name=f"{panel}_o")
ctx.launch(
panel, gqa_attention_prefill_long_ctx_cube_sp_pe_sp_kernel,
q, k, v, o,
T_q, S_kv, h_q, h_kv, d_head, C, P,
_auto_dim_remap=False,
)
def _make_bench_fn(panel: str):
kind, params = _PANEL_DISPATCH[panel]
def _bench_fn(ctx):
if kind == "prefill_long_ctx_cube_sp_pe_sp":
_run_prefill_panel_long_ctx_cube_sp_pe_sp(ctx, panel=panel, **params)
elif kind == "prefill_long_ctx_cube_repl_pe_tp":
_run_prefill_panel_long_ctx_cube_repl_pe_tp(ctx, panel=panel, **params)
elif kind == "prefill_long_ctx_cube_repl_pe_sp":
_run_prefill_panel_long_ctx_cube_repl_pe_sp(ctx, panel=panel, **params)
elif kind == "prefill_long_ctx_cube_sp_pe_tp":
_run_prefill_panel_long_ctx_cube_sp_pe_tp(ctx, panel=panel, **params)
else:
raise RuntimeError(
f"milestone-gqa-prefill-long-ctx-4cases panel {panel!r} has "
f"unsupported kind={kind!r}."
)
return _bench_fn
# ── Panel metrics (sweep.json carries these for the comparative plot) ─
_ENGINE_SUFFIXES = (
"pe_gemm", "pe_math", "pe_dma", "pe_fetch_store", "pe_ipcq", "pe_cpu",
)
def _end_to_end_ns(op_log) -> float:
"""End-to-end window: ``max(t_end) - min(t_start)`` over all records."""
if not op_log:
return 0.0
return max(r.t_end for r in op_log) - min(r.t_start for r in op_log)
def _engine_occupancy_ns(op_log) -> dict[str, float]:
"""Per-engine summed occupancy (component_id suffix match)."""
return {
eng: sum(
r.t_end - r.t_start
for r in op_log
if r.component_id.endswith("." + eng)
)
for eng in _ENGINE_SUFFIXES
}
def _run_panel(panel: str, topology: str) -> dict:
from kernbench.runtime_api.bench_runner import run_bench
from kernbench.runtime_api.types import resolve_device
from kernbench.sim_engine.engine import GraphEngine
from kernbench.topology.builder import resolve_topology
topo = resolve_topology(topology)
result = run_bench(
topology=topo, bench_fn=_make_bench_fn(panel),
device=resolve_device(None),
engine_factory=lambda t, d: GraphEngine(
getattr(t, "topology_obj", t), enable_data=True,
),
)
if not result.completion.ok:
raise RuntimeError(
f"milestone-gqa-prefill-long-ctx-4cases panel {panel!r} failed: "
f"{result.completion}"
)
kind, params = _PANEL_DISPATCH[panel]
op_log = result.engine.op_log
return {
"panel": panel,
"kind": kind,
**params,
"op_log_summary": _summarize_op_log(op_log),
"latency_ns": _end_to_end_ns(op_log),
"engine_occupancy_ns": _engine_occupancy_ns(op_log),
}
# ── Sweep entry (called by the umbrella ``milestone_1h_gqa``) ────────
def run_sweep(topology: str = "topology.yaml") -> int:
"""Drive all prefill case panels; write sweep.json. Returns row count.
Per-panel try/except: even with Q-axis tiling, very large T_q or
unusual config combinations can hit the per-PE scratch budget. We
keep going so sweep.json carries whichever panels succeed; failures
land in a ``failures`` list for follow-up.
"""
_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
rows: list[dict] = []
failures: list[dict] = []
for panel in _PANELS:
try:
rows.append(_run_panel(panel, topology))
except Exception as e:
print(f" panel {panel!r} FAILED: {e}")
failures.append({"panel": panel, "error": str(e)})
sweep = {
"version": 1,
"panels": list(_PANELS),
"rows": rows,
"failures": failures,
}
_SWEEP_JSON.write_text(json.dumps(sweep, indent=2))
print(f" gqa-prefill-long-ctx-4cases: {len(rows)} rows -> {_SWEEP_JSON}")
return len(rows)