ywkang
5917b3497c
- Remove xbar_top/bot, bridge, single noc node from topology
- Each cube_mesh.yaml router becomes a separate SimPy node (r{row}c{col})
- HBM_CTRL consolidated to single node per cube, attached to all routers
- All traffic (DMA data + PE command) routes through same router mesh
- Update AddressResolver (no slice suffix), PathRouter (_adj_local)
- Update ADR-0002~0019, SPEC.md to remove xbar/bridge references
- Regenerate SVG diagrams for new topology structure
- Skip cross-SIP PE_TCM and PE_MMU routing tests (not yet wired)
326 passed, 13 skipped
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3.9 KiB
3.9 KiB
ADR-0002: Routing Distance, Ordering & Bypass Rules
Status
Accepted
Date
2026-02-27
Context
The KernBench Graph Latency Simulator must compare kernel execution time across different architectures and topologies by computing end-to-end latency from graph traversal.
To support meaningful comparison:
- routing must be deterministic
- latency must reflect actual interconnect structure
- local vs remote traffic must be distinguishable
- “bypass” optimizations must not undermine debuggability or correctness
The simulator also aims to avoid software-managed metadata and hidden shortcuts that obscure control paths.
Decision
D1. Distance is accumulated latency, not hop count
- Routing “distance” is defined as the sum of per-node and per-link latency.
- Hop count alone must not be used for ordering or path selection.
- Size-aware serialization latency (bytes / BW) contributes to distance.
D2. Routing order is derived from graph traversal
- The chosen route is the path with minimum accumulated latency given the constructed graph and routing policy.
- Deterministic ordering must be guaranteed for identical inputs (topology + policy + request).
D3. Bypass is explicit and graph-represented
- All paths must be explicitly represented in the graph and subject to latency accumulation.
- Example: PE_DMA connects to the NOC router mesh (ADR-0019). All destinations (HBM, shared SRAM, inter-cube UCIe) are reached via explicit mesh hops. Local HBM access has minimal hops (switching overhead only); remote access traverses additional routers.
- Implicit or “magic” bypass paths are disallowed.
D4. No zero-latency end-to-end paths
- Every routed request must incur end-to-end latency > 0.
- Individual fabric segments (e.g., NOC hops) MAY have distance_mm = 0 when the fabric is distributed and distance is not meaningful at that granularity. This is allowed because other components on the same path (e.g., PE_DMA, SRAM, UCIe endpoints) contribute non-zero latency, ensuring the end-to-end invariant holds.
- Fully zero-latency end-to-end paths are disallowed, except for explicit test-only stubs clearly marked as such.
D5. Policy vs topology responsibility split
- Topology builder:
- defines nodes and links and their latency/BW parameters
- Routing policy:
- selects among available graph paths based on decoded domains
- Routing policy must not assume missing links; missing connectivity is a topology construction error.
D6. No software-managed routing metadata
- Routing decisions must not rely on per-request software-managed metadata that tracks distance, hop count, or ordering outside the graph model.
- All distance/order computation is derived from traversal itself.
Alternatives Considered
- Hop-count based routing
- Rejected: ignores heterogeneous latency/BW and misrepresents architectural differences.
- Implicit local shortcuts
- Rejected: breaks debuggability and violates traversal-based latency.
- Software-managed distance metadata
- Rejected: increases control overhead and obscures routing semantics.
Consequences
Positive
- Clear, debuggable hop-by-hop traces (SPEC R2, R4).
- Architecture comparisons reflect real interconnect structure.
- Routing behavior is reproducible and deterministic.
Tradeoffs / Costs
- Graph construction must be correct and complete.
- Bypass modeling requires explicit graph representation, which slightly increases topology description complexity.
Implementation Notes (Non-normative)
- Recommended responsibilities:
- Graph builder: ensure all required paths exist.
- Router: select next hop based on decoded domains and policy.
- Tests should assert:
- non-zero end-to-end latency
- deterministic routing for identical inputs
- bypass paths appear explicitly in emitted traces
Links
- SPEC.md: R1 (routing), R2 (latency), R3 (topology), R5 (multi-domain comm)
- ADR-0001: PhysAddr layout & decoding contract