ywkang
13f5cbc317
Concurrency fix (completes the cube_base change): the decode kernel's
output store o_base still used the global cube_id while every load used
the user-local cube_local. For a single user (cube_base=0) they coincide,
so it was masked; a second batched user (cube_base>0) overshot its o shard
into an unmapped VA, mis-decoded to a phantom sip0.cube0.pe8 and raised a
RoutingError. Switch o_base to cube_local (one line); single-user results
are byte-identical (decode smoke/correctness pass).
Batch harness (un-skipped): create all users' tensors first, then submit
all launches deferred and drain together, so deploys don't drive an
already-submitted launch and serialize the batch. Concurrent users on
disjoint CUBE groups now overlap to within 3-5% of the single-user
latency. Swept B in {1,2,capacity} at 8K (high-B dense runs are the
expensive ones; throughput is near-linear in B).
Measured result: aggregate throughput at a full SIP rises from 0.86
(1-kv-per-cube, 2 users) to 1.41 requests/us (8-kv-per-cube, 16 users) —
the dense mapping wins throughput, the spread mapping wins latency.
S5.2 batch paragraph updated projected -> measured; add batch_scaling
figure and plot_batch_scaling.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
907 B
907 B
| 1 | mode | kv_per_cube | C | S_kv | B | capacity | agg_latency_us | mean_user_latency_us | throughput_users_per_us |
|---|---|---|---|---|---|---|---|---|---|
| 2 | A1 | 1 | 8 | 8192 | 1 | 2 | 2.2471645000005376 | 2.2471645000005376 | 0.44500524994932983 |
| 3 | A1 | 1 | 8 | 8192 | 2 | 2 | 2.3131845000012543 | 2.2471645000012357 | 0.864608940617973 |
| 4 | A2 | 2 | 4 | 8192 | 1 | 4 | 3.3555127500005475 | 3.3555127500005475 | 0.2980170467240325 |
| 5 | A2 | 2 | 4 | 8192 | 2 | 4 | 3.3935227500010514 | 3.355512750001042 | 0.5893580645656141 |
| 6 | A2 | 2 | 4 | 8192 | 4 | 4 | 3.4695427500010703 | 3.3555127500011586 | 1.1528896711241752 |
| 7 | A4 | 4 | 2 | 8192 | 1 | 8 | 5.586135500000557 | 5.586135500000557 | 0.17901463364071643 |
| 8 | A4 | 4 | 2 | 8192 | 2 | 8 | 5.602295500000823 | 5.586135500000673 | 0.35699652044411195 |
| 9 | A4 | 4 | 2 | 8192 | 8 | 8 | 5.783285499986028 | 5.586135499984957 | 1.3832967436968704 |
| 10 | B | 8 | 1 | 8192 | 1 | 16 | 10.816091000000947 | 10.816091000000947 | 0.09245484343649775 |
| 11 | B | 8 | 1 | 8192 | 2 | 16 | 10.848411000001535 | 10.816091000001236 | 0.18435879687815265 |
| 12 | B | 8 | 1 | 8192 | 16 | 16 | 11.38492099986691 | 10.816090999974403 | 1.4053676788962384 |