ywkang
5917b3497c
- Remove xbar_top/bot, bridge, single noc node from topology
- Each cube_mesh.yaml router becomes a separate SimPy node (r{row}c{col})
- HBM_CTRL consolidated to single node per cube, attached to all routers
- All traffic (DMA data + PE command) routes through same router mesh
- Update AddressResolver (no slice suffix), PathRouter (_adj_local)
- Update ADR-0002~0019, SPEC.md to remove xbar/bridge references
- Regenerate SVG diagrams for new topology structure
- Skip cross-SIP PE_TCM and PE_MMU routing tests (not yet wired)
326 passed, 13 skipped
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
99 lines
3.3 KiB
Markdown
99 lines
3.3 KiB
Markdown
# ADR-0016: IOChiplet NOC and Memory Data Path
|
|
|
|
## Status
|
|
|
|
Accepted
|
|
|
|
## Context
|
|
|
|
ADR-0003 D2 defines IO chiplets as SIP-level components providing PCIe-EP and
|
|
IO_CPU interfaces, but does not specify internal routing within the IO chiplet.
|
|
ADR-0015 D4 was updated to document the M_CPU bypass for Memory R/W, but the
|
|
IO chiplet's internal NOC architecture that enables this routing was not
|
|
formally documented.
|
|
|
|
The IO chiplet needs an internal routing fabric (io_noc) to:
|
|
|
|
- connect pcie_ep, io_cpu, and per-cube UCIe PHY ports
|
|
- route memory operations (MemoryWrite/Read) directly to cube fabric without
|
|
passing through io_cpu
|
|
- route kernel launch commands through io_cpu for command interpretation
|
|
|
|
## Decision
|
|
|
|
### D1. IOChiplet internal NOC (io_noc)
|
|
|
|
Each IO chiplet instance contains an internal NOC node (`io_noc`) that connects:
|
|
|
|
- `pcie_ep` — host-facing PCIe endpoint
|
|
- `io_cpu` — command processor for kernel launch interpretation
|
|
- `io_ucie-{PHY}.conn{N}` — per-PHY connection nodes to cube UCIe ports
|
|
|
|
The io_noc is a forwarding-only fabric (`forwarding_v1` implementation) with
|
|
zero overhead. All routing decisions are made by the simulation engine based
|
|
on message type, not by io_noc itself.
|
|
|
|
### D2. IOChiplet UCIe decomposition
|
|
|
|
Each IO chiplet PHY port is decomposed into:
|
|
|
|
- `io_ucie-{PHY}` — the UCIe protocol endpoint (overhead = 8ns)
|
|
- `io_ucie-{PHY}.conn{N}` — N connection nodes between io_noc and io_ucie
|
|
|
|
This mirrors the cube-side UCIe decomposition (ADR-0015 D1) and allows
|
|
multiple independent NOC-to-UCIe connections per PHY.
|
|
|
|
### D3. Memory R/W path (M_CPU bypass)
|
|
|
|
Memory operations (MemoryWrite, MemoryRead) are routed directly from pcie_ep
|
|
through io_noc to the target cube, bypassing io_cpu entirely:
|
|
|
|
```text
|
|
pcie_ep → io_noc → conn → io_ucie → [cube UCIe] → router mesh → hbm_ctrl
|
|
```
|
|
|
|
This avoids the 10ns io_cpu overhead for pure data transfers. The simulation
|
|
engine's `_process_memory_direct()` method uses `find_memory_path()` which
|
|
resolves the shortest path from pcie_ep to the target HBM node.
|
|
|
|
### D4. Kernel Launch path (via io_cpu)
|
|
|
|
Kernel launch commands require io_cpu for command interpretation and PE
|
|
fan-out setup:
|
|
|
|
```text
|
|
pcie_ep → io_noc → io_cpu → io_noc → conn → io_ucie → [cube UCIe]
|
|
→ noc → m_cpu → PE
|
|
```
|
|
|
|
The engine's `_entry_points()` method routes KernelLaunchMsg through both
|
|
pcie_ep (entry) and io_cpu (command processing).
|
|
|
|
### D5. IOChiplet-to-cube port mapping
|
|
|
|
Each IO chiplet instance declares which cube ports it connects to:
|
|
|
|
```yaml
|
|
cube_ports:
|
|
- { cube: {xy: [0,0]}, cube_side: N, phy: P0, distance_mm: 2.0 }
|
|
- { cube: {xy: [1,0]}, cube_side: N, phy: P1, distance_mm: 2.0 }
|
|
```
|
|
|
|
The topology builder creates edges from io_ucie PHY nodes to the
|
|
corresponding cube UCIe port nodes, with the specified distance and
|
|
the IO chiplet's `per_connection_bw_gbs` as link bandwidth.
|
|
|
|
## Consequences
|
|
|
|
- IO chiplet has a well-defined internal routing fabric
|
|
- Memory operations avoid unnecessary io_cpu overhead
|
|
- Kernel launch commands still get proper command interpretation
|
|
- The io_noc pattern is consistent with cube-level NOC design
|
|
- ADR-0003 D2 is extended (not contradicted) by this ADR
|
|
|
|
## Links
|
|
|
|
- ADR-0003 D2 (IO chiplet definition)
|
|
- ADR-0015 D4 (fabric paths for Memory R/W and Kernel Launch)
|
|
- ADR-0012 D1 (host-to-IO_CPU message schema)
|