kernbench2/SPEC.md

# KernBench System-Level Simulator — SPEC

This document defines the architectural contract for the KernBench
system-level discrete-event simulator for our AI Accelerator SIP-based systems.
All implementations, tests, and changes MUST conform to this SPEC.

---

## 0. Goal

Build a **system-level, discrete-event simulator** to evaluate the performance of
**LLM kernels running on our AI Accelerator SIP-based systems**, under varying
**SIP architectures, topologies, and interconnect configurations**.

The simulator models **data-movement and control paths across the full hardware
hierarchy** and computes **end-to-end execution latency** for kernel executions
dispatched to Processing Elements (PEs).

Primary objectives:

- compare LLM kernel execution latency under different system configurations
- model PE↔HBM, PE↔PE, CUBE↔CUBE, and SIP↔SIP communication and control paths
- guarantee deterministic, verifiable behavior with strong debuggability
- support visual inspection of the modeled system at multiple abstraction levels

---

## 0.1 Golden Invariants (Must NOT be violated)

- End-to-end latency is computed **strictly by explicit traversal** over modeled
  components and links.
- Every routed request MUST incur **latency > 0**.
- Routing decisions MUST be **deterministic** given
  (topology + routing policy + request).
- All valid request flows MUST have explicit connectivity in the model.
- No hidden shortcuts, implicit bypasses, or magic paths are allowed.
- Architectural decisions documented in ADRs override local optimizations.

---

## 0.2 Architectural References (ADRs)

Major architectural decisions are documented in ADRs and referenced by number.

- ADR-0001: PhysAddr layout & address decoding contract
- ADR-0002: Routing distance, ordering, and bypass rules
- ADR-0003: Target system hierarchy & modeling scope (Tray / SIP / CUBE / PE / IO chiplet)
- ADR-0004: Memory semantics & local-HBM bandwidth guarantee contract
- ADR-0005: Diagram views (SIP / CUBE / PE) and distance-aware layout rules
- ADR-0006: Topology compilation, distance extraction, and automatic diagram generation
- ADR-0007: runtime_api vs sim_engine responsibility boundaries
- ADR-0008: Tensor deployment and allocation (Host allocator, PA-first)
- ADR-0009: Kernel execution fan-out and completion semantics
- ADR-0010: CLI device selection and multi-device execution semantics
- ADR-0011: Memory addressing simplification (PA-first)
- ADR-0012: Host ↔ IO_CPU message schema (PA-first, PE-tagged shards)
- ADR-0013: Verification strategy and Phase 1 test plan
- ADR-0014: PE internal execution model (PE_CPU, PE_SCHEDULER, composite commands)
- ADR-0015: Component port/wire model, BW occupancy, and fabric routing
- ADR-0016: IOChiplet NOC and memory data path (M_CPU bypass)
- ADR-0017: Cube NOC 2D mesh architecture (XY routing, contention, attachments)

SPEC MUST remain consistent with accepted ADRs.

---

## 1. Core Requirements

### R1. Correct Routing and Control Path

- A request MUST traverse the correct sequence of components based on:
  - source location,
  - destination address or placement tags,
  - routing policy and available topology connectivity.
- Local vs remote traffic MUST be distinguishable:
  - same SIP vs different SIP,
  - same CUBE vs different CUBE,
  - (optional) same PE-group vs cross PE-group.
- Routing behavior MUST be reproducible and deterministic.

---

### R2. Latency is Computed by Traversal

End-to-end latency is the sum of:

- per-node fixed latency (processing / router delay),
- per-link latency (fixed and/or size-aware serialization: bytes / BW),
- per-service latency (e.g., memory controller service time).

The simulator MUST:

- support both fixed and size-aware latency,
- emit hop-by-hop traces with timestamps and component identifiers.

---

### R3. Topology is Configurable and Variable

Topology MUST NOT be hardcoded.

The simulator MUST accept multiple topologies (YAML / JSON / dict), varying:

- SIP count,
- CUBE count per SIP,
- PE count per CUBE,
- on-chip fabric structure (e.g., mesh / NoC router grid),
- IO chiplets and interconnects,
- link bandwidth, latency, and capacity parameters.

Given a topology:

- all required request flows MUST have valid connectivity,
- missing links are a topology construction error, not a routing error.

---

### R4. DI-First Component Design (Swappable Implementations)

All components MUST be replaceable behind stable interfaces, including:

- routers and fabrics (NoC router mesh, switches),
- DMA engines and queues,
- memory controllers and services (HBM, TCM, queues),
- management and control processors (modeled components).

The simulator MUST:

- use dependency injection (DI) to bind node specifications to implementation classes,
- allow component swapping without changing test logic,
- avoid leaking routing or policy logic into unrelated components.

---

### R5. Multi-Domain Communication Modeling

The simulator MUST model communication across hierarchical domains, including:

- PE ↔ local HBM
- PE ↔ remote HBM in the same CUBE
- PE ↔ remote HBM in other CUBEs within the same SIP
- PE ↔ remote HBM in other SIPs
- PE ↔ PE messaging (e.g., IPCQ)
- PE ↔ IO chiplets
- CUBE ↔ CUBE (e.g., via UCIe)
- SIP ↔ SIP (e.g., via PCIe or UAL)

Policy-based bypass is allowed ONLY if:

- the bypass path is explicitly represented in the model,
- the bypass incurs non-zero latency,
- the bypass is visible in traces and diagrams.

---

### R6. Verification-Driven Development

Development MUST follow a verification-driven workflow:

- behavior is validated by tests with meaningful input cases,
- tests encode SPEC-defined invariants, not incidental implementation details,
- changes without clear verification coverage are not allowed.

---

## R7. Runtime API

The simulator MUST provide a host-facing runtime API that:

- exposes tensor deployment and kernel execution operations,
- submits requests to endpoint components: PCIE_EP for memory operations
  (MemoryWrite/Read), IO_CPU for kernel launch,
- owns host-side tensor handles and allocation metadata as PA shard maps,
- remains topology-agnostic and does not perform routing or fan-out.

Tensor deployment in Phase 0 produces **device physical-address (PA) shard mappings**.
Each shard explicitly identifies its target `(sip, cube, pe)` and PA range.
No separate host-visible allocation RPC (e.g., AllocateTensorMeta) exists.

---

## R8. Simulation Engine

The simulator MUST include a discrete-event simulation engine that:

- injects requests into the system graph,
- schedules events deterministically,
- tracks completion via correlation identifiers,
- decomposes runtime API operations into explicit graph requests
  (e.g., MemoryWrite, MemoryRead, KernelLaunch).

---

## R9. CLI Execution Semantics

The CLI MUST support executing benchmarks:

- on a specified device.

Benchmarks are executed once per invocation within a single simulation instance.
If multiple devices are present in the topology, a benchmark MAY interact with
multiple devices internally, but the CLI does not launch multiple independent
benchmark instances by default.

---

## R10. Memory Addressing (Phase 0)

The simulator uses a **VA/PA memory model** (ADR-0011):

- Tensors are assigned a contiguous virtual address (VA) range at deployment.
- PE_MMU translates VA→PA per access; TLB overhead is configurable.
- Mapping installation (MmuMapMsg) traverses the fabric with measured latency.
- Replicate tensors use per-cube local PA mapping; sharded tensors broadcast.
- PA-only fallback is retained for backward compatibility.
- Tensor placement is represented as a list of PA shards, each explicitly tagged
  with `(sip, cube, pe)`, plus a tensor-wide `va_base`.

All memory access latency MUST be modeled explicitly via graph traversal.
No implicit translation or hidden latency is allowed.

---

## 2. Model Concepts

### 2.1 Graph Execution Model

- Nodes represent modeled components (PE blocks, NoC routers,
  HBM controllers, IO components, etc.).
- Directed edges represent interconnect links with latency and bandwidth attributes.
- Execution model:
  - a node receives a request,
  - incurs node or service latency,
  - emits the request to the next hop via a link,
  - repeats until the destination service completes.

---

### 2.2 Routing

Routing MAY be implemented as:

- policy-based routing (code-driven),
- routing tables (config-driven),
- topology-driven routing (e.g., mesh XY),
- or a hybrid approach.

Routing MUST:

- consume decoded address domains or explicit placement tags,
- operate only on explicit topology connectivity,
- remain deterministic.

Kernel execution requests reference tensors via PA shard mappings.
Each shard explicitly identifies its target PE, allowing IO_CPU to
deterministically fan-out execution without relying on PA decoding.

---

## 3. Inputs and Identity

### 3.1 Node Identity Scheme

Nodes MUST have stable, parsable identifiers sufficient for domain inference
and trace-based debugging.

Example patterns:

- `tray.host_cpu`
- `sip{S}.io{I}.pcie_ep`
- `sip{S}.cube{C}.fabric`
- `sip{S}.cube{C}.pe{P}`
- `sip{S}.cube{C}.hbm_ctrl`

---

### 3.2 Link Specifications

A link MAY include:

- fixed latency (ns),
- bandwidth (GB/s) for serialization latency,
- optional capacity for contention modeling.

Topology builders MUST ensure:

- required links exist,
- link parameters are consistent with topology intent.

---

## 4. Output, Debuggability, and Diagrams

The simulator MUST provide:

- per-request hop-by-hop traces with timestamps,
- clear error messages for missing connectivity
  (e.g., "no link for A → B"),
- reproducible, inspectable representations of the modeled system.

Diagrams are **derived artifacts** of the simulator model:

- They MUST be generatable from the **compiled topology** and **distance metadata**
  used by execution and routing.
- Generation MAY be performed lazily or cached by the implementation,
  as long as outputs remain consistent with the compiled topology.

Diagram abstraction levels and distance-aware layout rules are defined in ADR-0005.
Automatic diagram generation and output conventions are defined in ADR-0006.

By default, generated diagrams are written under:

- `docs/diagrams/`

---

## 5. Non-Goals (for now)

The following are explicitly out of scope:

- cycle-accurate microarchitecture modeling,
- detailed cache coherence protocols,
- full PCIe / CXL protocol correctness.

These MAY be layered later via additional components and policies.

---

## 6. Decision Boundaries

- SPEC.md defines architectural intent and invariants.
- Code implements SPEC and MUST NOT introduce hidden invariants.
- Tests validate SPEC-defined behavior and MUST NOT encode fixed topology assumptions.
- ADRs record non-trivial architectural decisions and MUST be referenced when relevant.