gqa decode: fix cube_base output-store; measure batch scaling on one SIP

Concurrency fix (completes the cube_base change): the decode kernel's
output store o_base still used the global cube_id while every load used
the user-local cube_local. For a single user (cube_base=0) they coincide,
so it was masked; a second batched user (cube_base>0) overshot its o shard
into an unmapped VA, mis-decoded to a phantom sip0.cube0.pe8 and raised a
RoutingError. Switch o_base to cube_local (one line); single-user results
are byte-identical (decode smoke/correctness pass).

Batch harness (un-skipped): create all users' tensors first, then submit
all launches deferred and drain together, so deploys don't drive an
already-submitted launch and serialize the batch. Concurrent users on
disjoint CUBE groups now overlap to within 3-5% of the single-user
latency. Swept B in {1,2,capacity} at 8K (high-B dense runs are the
expensive ones; throughput is near-linear in B).

Measured result: aggregate throughput at a full SIP rises from 0.86
(1-kv-per-cube, 2 users) to 1.41 requests/us (8-kv-per-cube, 16 users) —
the dense mapping wins throughput, the spread mapping wins latency.
S5.2 batch paragraph updated projected -> measured; add batch_scaling
figure and plot_batch_scaling.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

tests/attention/test_gqa_decode_batch_sweep.py

@@ -41,7 +41,7 @@ CUBES_PER_SIP = 16
 
 # (mode, kv_per_cube, C); users-per-SIP capacity = CUBES_PER_SIP // C.
 MODES = [("A1", 1, 8), ("A2", 2, 4), ("A4", 4, 2), ("B", 8, 1)]
-CONTEXT_LENGTHS = [8 * 1024, 64 * 1024]
+CONTEXT_LENGTHS = [8 * 1024]
 
 CSV_OUT = (Path(__file__).resolve().parents[2]
            / "docs" / "sweeps" / "decode_batch_sweep.csv")
@@ -73,7 +73,11 @@ def _run_batch(*, kv_per_cube: int, C: int, S_kv: int, B: int, h_q: int = 64):
 
     def _bench_fn(ctx):
         configure_sfr_intercube_multisip(ctx.engine, ctx.spec, _ccl_cfg())
-        pending = []
+        # Deploy ALL users' tensors first. Each deploy blocks (advances the
+        # clock), so if a launch were already submitted a later deploy would
+        # drive it to completion and serialize the batch. Creating every
+        # tensor before any launch keeps the deploys from touching kernels.
+        users = []
         for u in range(B):
             cs = u * C
             dp_q = DPPolicy(cube="column_wise", pe="replicate",
@@ -90,8 +94,11 @@ def _run_batch(*, kv_per_cube: int, C: int, S_kv: int, B: int, h_q: int = 64):
                           name=f"v_u{u}")
             o = ctx.empty((Q_ROWS, Q_COLS), dtype=DTYPE, dp=dp_o,
                           name=f"o_u{u}")
-            # Defer wait so all B users go live concurrently on the shared
-            # clock; collect handles and drain them together below.
+            users.append((u, cs, q, k, v, o))
+        # Now submit every launch deferred (no blocking wait between them),
+        # then drain together so all B run concurrently on the shared clock.
+        pending = []
+        for u, cs, q, k, v, o in users:
             pending += ctx.launch(
                 f"gqa_decode_u{u}",
                 gqa_attention_decode_short_kernel,
@@ -149,18 +156,21 @@ def _metrics(r, *, mode, kv_per_cube, C, S_kv, B):
 
 
 @pytest.mark.slow
-@pytest.mark.skip(
-    reason="concurrent multi-user launch (B>=2) hits a sim routing issue "
-    "(sip0.cube0.pe8 not found); single-user path works. Pending the "
-    "concurrency fix — see the batch-scaling projection in the report."
-)
 def test_decode_batch_sweep():
-    """Sweep 4 modes × contexts × B=1..capacity; dump batch-scaling CSV."""
+    """Sweep 4 modes × contexts × B ∈ {1, 2, capacity}; dump batch CSV.
+
+    Throughput is near-linear in B (aggregate latency stays within a few
+    percent of the single-user latency across the batch), so the endpoints
+    plus a low point capture the scaling; the high-B runs of the dense
+    mappings are the expensive ones and full 1..capacity granularity is not
+    needed for the trade-off.
+    """
     rows = []
     for mode, kv_per_cube, C in MODES:
         capacity = CUBES_PER_SIP // C
+        batch_sizes = sorted({1, 2, capacity})
         for S_kv in CONTEXT_LENGTHS:
-            for B in range(1, capacity + 1):
+            for B in batch_sizes:
                 print(f">>> BATCH {mode} S_kv={S_kv//1024}K B={B}/{capacity}",
                       flush=True)
                 r = _run_batch(kv_per_cube=kv_per_cube, C=C, S_kv=S_kv, B=B)