paper(platform): edit-pass through §2 KernBench Platform done

- Reordered §2 subsections to follow the SIP → CUBE → PE → graph flow: Why KernBench → Device and execution model → Latency model → Modeled hardware configuration. Readers now meet the device hierarchy before the graph abstraction that re-uses it. - §2.2 Device and execution model: starts with the SIP/CUBE/PE hierarchy paragraphs (each anchoring fig:sip-arch, fig:cube-arch, fig:pe-arch); then the runtime-API/sim-engine/components bullet list; then the atomic-vs-composite command distinction (corrects the prior over-narrow framing that read every PE command as composite -- atomic single-engine commands exist too, and PE_CPU itself runs control-plane work directly). - §2.3 Latency model: opens with the four-contribution decomposition (per-node overhead, per-edge transmission, drain, queuing delay) and the latency_model schematic; retains existing The hardware as a graph / From graph to DES / Latency contributions / Congestion / Control-plane cost model / Accuracy paragraphs. Accuracy paragraph now closes on KernBench's sufficiency for *relative* HW/SW design trade-offs given analytic + external-simulator agreement. - New figures and assets: - figures/sip_architecture.pdf (SIP-level graph view) - figures/cube_architecture.pdf (CUBE-level zoom-in) - figures/latency_model.png (conceptual latency-model schematic with per-node / per-edge / drain / queuing-delay colour coding) - figures/pe_architecture.png (carried over) - Source-of-truth generator for the latency schematic: scripts/paper/paper_latency_model_diagram.py (a report-only harness under scripts/paper/ per the /paper isolation rule). - main.tex preamble: \usepackage{tikz} added (kept from prior sequence-diagram draft -- harmless now that the latency model is a PNG; left in to keep paragraph numbering stable). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-15 17:17:05 -07:00
parent 756680f4e6
commit 7f437a20bd
8 changed files with 679 additions and 51 deletions
@@ -15,6 +15,8 @@
 \usepackage{caption}
 \captionsetup{font=small,labelfont=bf}
 \usepackage{microtype}
+\usepackage{tikz}
+\usetikzlibrary{arrows.meta,positioning,calc,fit}

 \graphicspath{{figures/}}

@@ -3,11 +3,10 @@

 All results in this report are produced on \emph{KernBench}, a
 system-level discrete-event simulator for LLM kernels running on AHBM.
-This section explains why the platform exists, how it executes a
-kernel, how its latency model works---specifically how the hardware is
-viewed as a graph, how that graph is driven by a discrete-event engine,
-and how congestion is captured---and the concrete hardware configuration
-used for every experiment that follows.
+This section explains why the platform exists, the device and
+execution model it presents to a kernel writer, how its latency model
+turns that execution into a number, and the concrete hardware
+configuration used for every experiment that follows.

 \subsection{Why KernBench: source-level kernels without a software stack}
 \label{sec:why}
@@ -33,51 +32,140 @@ without the confound of compiler maturity or framework overhead. The
 cost is that KernBench numbers are \emph{not} E2E latencies; they are
 the achievable-kernel latencies an ideal software stack would expose.

-\subsection{Execution model}
+\subsection{Device and execution model}
 \label{sec:exec}

+KernBench models AHBM as a hierarchy of SIPs, CUBEs, and processing
+elements (PEs). At the system level, multiple SIPs are connected
+through inter-package links, while each SIP contains a collection of
+CUBEs joined by an on-package interconnect
+(Fig.~\ref{fig:sip-arch}).
+
+\begin{figure}[t]
+\centering
+\includegraphics[width=\linewidth]{sip_architecture.pdf}
+\caption{Modeled hardware graph at the \emph{SIP} level (one
+example configuration; specific parameters in \S\ref{sec:hw} /
+Table~\ref{tab:hw}). The SIP holds a $4{\times}4$ mesh of CUBEs and
+an IO chiplet, with each line a directed link labelled by its
+physical distance and bandwidth.}
+\label{fig:sip-arch}
+\end{figure}
+
+Each CUBE contains eight PEs, shared SRAM, HBM controllers, an
+\textsf{M\_CPU} control processor, and an intra-CUBE router mesh
+(Fig.~\ref{fig:cube-arch}). Together these components form the
+execution substrate for all kernels evaluated in this report.
+
+\begin{figure}[t]
+\centering
+\includegraphics[width=\linewidth]{cube_architecture.pdf}
+\caption{Modeled hardware graph at the \emph{CUBE} level---a
+zoom-in on one CUBE node from Fig.~\ref{fig:sip-arch}. The CUBE
+holds 8 PEs (each with its own HBM channels), the \textsf{M\_CPU}
+control processor, a shared SRAM, an intra-CUBE NoC router mesh, and
+UCIe links to neighbouring CUBEs.}
+\label{fig:cube-arch}
+\end{figure}
+
+Within a PE, commands are dispatched by \textsf{PE\_CPU} to
+\textsf{PE\_SCHED}, which routes work to specialized execution
+engines---DMA, FETCH/STORE, GEMM, vector-math, and IPCQ
+(Fig.~\ref{fig:pe-arch}). Commands come in two flavours. \emph{Atomic}
+commands target a single engine---a plain DMA read, a GEMM tile, a
+vector-math op, an IPCQ send/receive---and are the natural unit for
+short or special-purpose work; the \textsf{PE\_CPU} itself also runs
+control-plane work directly. \emph{Composite} commands, by contrast,
+carry an ordered pipeline of operations across multiple engines
+(\textsf{DMA\_READ} $\rightarrow$ \textsf{FETCH} $\rightarrow$
+\textsf{GEMM}/\textsf{MATH} $\rightarrow$ \textsf{STORE} $\rightarrow$
+\textsf{DMA\_WRITE}) that the \textsf{PE\_SCHED} tiles and streams
+without per-tile redispatch. The composite form is the substrate for
+the GEMM optimization (\S\ref{sec:gemm}) and, combined with on-PE
+collectives, for fused attention (\S\ref{sec:gqa}).
+
+\begin{figure*}[t]
+\centering
+\includegraphics[width=\linewidth]{pe_architecture.png}
+\caption{Modeled PE architecture used in this report---one example
+configuration whose specific parameters are listed in
+\S\ref{sec:hw} (Table~\ref{tab:hw}); KernBench is not tied to this
+particular decomposition and supports arbitrary PE block layouts
+provided each component is given a port and bandwidth model.
+\textsf{PE\_CPU} dispatches commands to the \textsf{PE\_SCHED}, which
+routes tile-token streams through the \textsf{PE\_DMA},
+\textsf{PE\_FETCH\_STORE}, and \textsf{GEMM}/\textsf{MATH} engines
+along on-chip links; the \textsf{PE\_IPCQ} provides the control plane
+for on-device collective communication.}
+\label{fig:pe-arch}
+\end{figure*}
+
 KernBench is layered along the flow of a request:

 \begin{itemize}
  \item The \textbf{runtime API} is host-facing and
  topology-agnostic---it deploys tensors and launches kernels but knows
  nothing about routing or interconnect.
-  \item The \textbf{simulation engine} schedules discrete events, routes
-  every request through the modeled graph, and tracks completion via
-  per-request correlation IDs.
+  \item The \textbf{simulation engine} schedules discrete events and
+  routes every request through the modeled graph.
  \item The \textbf{components} are device-side nodes that model
-  hardware behavior: the per-PE blocks (scheduler, DMA, GEMM and
+  hardware behaviour: the per-PE blocks (scheduler, DMA, GEMM and
  vector-math engines, TCM, IPCQ), the NoC routers, the HBM
  controllers, and the inter-chiplet links.
 \end{itemize}

-Within a PE, work is expressed as \emph{composite commands}: a single
-command carries an ordered pipeline of operations
-(\textsf{DMA\_READ} $\rightarrow$ \textsf{FETCH} $\rightarrow$
-\textsf{GEMM}/\textsf{MATH} $\rightarrow$ \textsf{STORE} $\rightarrow$
-\textsf{DMA\_WRITE}) that the PE scheduler tiles and streams. This
-composite mechanism is the substrate for the GEMM optimization
-(\S\ref{sec:gemm}) and, combined with on-PE collectives, for fused
-attention (\S\ref{sec:gqa}). Data and timing are handled in two passes,
-so that a kernel's numeric results and its latency are computed
-consistently but independently.
+Data and timing are handled in two passes, so that a kernel's numeric
+results and its latency are computed consistently but independently.

-\subsection{Latency model: a graph traversed by events}
+\subsection{Latency model: graph traversal and contention}
 \label{sec:latency}

-\paragraph{The hardware as a graph.} KernBench views the modeled
-hardware as a directed graph. \emph{Nodes} are the components listed
-above; \emph{edges} are the interconnect links between them, each
-carrying bandwidth (\si{\giga\byte\per\second}) and propagation
-(\si{\nano\second}) attributes. The topology is compiled once at
+The modeled hardware hierarchy described above is represented
+internally as a directed graph. Nodes correspond to hardware
+components---PEs, routers, memory controllers, SRAM blocks, IO
+chiplets---while edges represent communication links with associated
+bandwidth (\si{\giga\byte\per\second}) and propagation
+(\si{\nano\second}) attributes. Every operation in KernBench---DMA
+transfers, remote memory accesses, collective communication, command
+dispatch---is modelled as a traversal through this graph. End-to-end
+latency is decomposed into four contributions accumulated along the
+traversal path (Fig.~\ref{fig:latency-model}): \textbf{per-node
+overhead} at each component, \textbf{per-edge transmission} on each
+wire, \textbf{drain} (per-flit service occupancy) at the destination,
+and \textbf{queuing delay} at the shared FIFOs that the wire and the
+destination share between concurrent transactions.
+
+\begin{figure*}[t]
+\centering
+\includegraphics[width=\textwidth]{latency_model.png}
+\caption{Conceptual schematic of the latency model. Two source nodes
+(Requester A/B) inject flits through a chain of routers into a
+destination node; on the shared edge between routers, flits from the
+two transactions are interleaved flit-by-flit by the wire's FIFO
+arrival order. End-to-end latency is the sum of four contributions:
+\textbf{per-node overhead} (the switch's fixed processing cost,
+shown in light yellow---the same colour as the destination's
+processing-logic block), \textbf{per-edge transmission}
+($\textit{flit\_size}/\textit{BW}$ on each wire),
+\textbf{drain} (per-flit service occupancy at the destination's
+channel), and \textbf{queuing delay} (waiting in a FIFO when a
+shared resource is busy). The places where queuing actually
+accumulates are highlighted in green: the router output queue and
+the destination's input queue. This is the model, not a
+measurement; specific bandwidths and overheads are listed in
+\S\ref{sec:hw} (Table~\ref{tab:hw}).}
+\label{fig:latency-model}
+\end{figure*}
+
+\paragraph{The hardware as a graph.} The topology is compiled once at
 configuration time into this graph and is never mutated during a run.
-Every routed request---a DMA, a remote read, an IPCQ message, a
-kernel-launch command---is a \emph{traversal} of this graph from a
-source node to a destination service, hopping through routers and
-links along the way. There are no hidden shortcuts, implicit bypasses,
-or magic paths: if a request reaches its destination, the path it took
-is explicit in the graph, and the latency it incurred is the sum of
-the per-node and per-edge costs paid along that path.
+There are no hidden shortcuts, implicit bypasses, or magic paths: if a
+request reaches its destination, the path it took is explicit in the
+graph, and the latency it incurred is the sum of the per-node and
+per-edge costs paid along that path. The same graph representation
+applies recursively at every hierarchy level---system, SIP
+(Fig.~\ref{fig:sip-arch}), CUBE (Fig.~\ref{fig:cube-arch}), and PE
+(Fig.~\ref{fig:pe-arch}).

 \paragraph{From graph to discrete-event simulation.} The graph is
 driven by a discrete-event engine. Two kinds of events advance
@@ -85,14 +173,14 @@ simulation time: \emph{node events} (component switching overhead,
 service completions such as an HBM channel commit or a GEMM tile
 finish) and \emph{edge events} (the flit-by-flit serialization of a
 payload across a bandwidth-limited link). The engine maintains a
-priority queue of pending events ordered by their scheduled time,
-fires them one at a time, and treats ties under a deterministic
-ordering policy so that the same kernel on the same topology always
-yields the same trace. Per-request correlation IDs are stamped at
-injection and carried through every hop, so the trace is recoverable
-from injection to completion. Every nanosecond in a reported latency
-traces back to exactly one of these events on exactly one node or
-edge.
+priority queue of pending events ordered by their scheduled time and
+fires them one at a time, with ties broken under a deterministic
+policy so that the same kernel on the same topology always yields the
+same trace. Per-request correlation IDs are stamped at injection and
+carried through every hop, so the path from injection to completion
+is fully traceable. Every nanosecond in a reported latency
+corresponds to exactly one of these events on exactly one node or
+edge---there is no slack in the budget.

 \paragraph{Latency contributions.} Three kinds of latency accumulate
 along a traversal: (i) \emph{per-node fixed overhead}---each component
@@ -113,11 +201,11 @@ edges. \emph{Every directed edge has a FIFO}: an arriving flit takes
 its bandwidth-limited transfer time on top of whatever earlier flits
 are still being served, so a busy link queues later traffic behind
 earlier traffic rather than transferring everything at peak BW.
-\emph{HBM is modeled with per-pseudo-channel parallelism}: a stateless
-array of channel-availability timestamps with address-based channel
-selection captures the bank-level concurrency that real HBM exposes,
-so 64 channels per CUBE deliver real parallelism on uniform addresses
-but a hot channel surfaces as the bottleneck on skewed ones.
+\emph{HBM is modelled with per-pseudo-channel parallelism}: a
+stateless array of channel-availability timestamps with address-based
+channel selection captures the bank-level concurrency that real HBM
+exposes, so 64 channels per CUBE deliver real parallelism on uniform
+addresses but a hot channel surfaces as the bottleneck on skewed ones.
 \emph{Every component has a serial worker}: a router carrying two
 heavy streams interleaves them at flit granularity in arrival order
 rather than fanning out for free, so two concurrent collectives sharing
@@ -128,7 +216,7 @@ actually relieve them---which is exactly the question the codesign
 work in this report turns on.

 \paragraph{Control-plane (issue) cost model.} The cost of \emph{issuing}
-a command is modeled structurally rather than with a per-operation
+a command is modelled structurally rather than with a per-operation
 calibration table. The PE control processor charges, per command,
 \[
  d_{\text{cmd}} = \textsf{FIXED} + b_{\text{logical}} \cdot R,
@@ -171,10 +259,13 @@ than a model error. The known simplifications---FIFO router
 arbitration (instead of round-robin), HBM scheduler without
 write-buffer reordering, no bank conflict, no refresh or thermal
 effects, and no upstream backpressure---are the price of a
-deterministic, inspectable model. They bound the absolute accuracy but
-do not distort the \emph{relative} comparisons (tiling A vs.\ B,
-topology X vs.\ Y, with vs.\ without composite command, mesh vs.\
-torus) that this report is built on.
+deterministic, inspectable model. These simplifications bound the
+absolute accuracy, but the agreement with both analytic models and the
+external full-system simulator data above indicates that KernBench is
+sufficiently accurate for evaluating the \emph{relative}
+hardware--software design trade-offs (tiling A vs.\ B, topology X
+vs.\ Y, with vs.\ without composite command, mesh vs.\ torus) that
+are the primary objective of this work.

 \subsection{Modeled hardware configuration}
 \label{sec:hw}
@@ -0,0 +1,535 @@
+#!/usr/bin/env python3
+"""Conceptual latency-model diagram for the KernBench paper (v5).
+
+Generic naming (Requester Node A/B, Router, Destination Node), with
+Router internals visible (two input ports -> switch -> output queue ->
+output port) so the queuing delay can be located *at the out port*
+rather than at the box edge.  The Destination Node shows queue -> drain
+slot -> processing logic.
+
+Highlights:
+  * Same-size Routers, each with explicit switching logic + output queue.
+  * Edge labels removed -- the wires speak for themselves.
+  * Annotation lines are strictly vertical; arrow heads use a larger
+    mutation_scale so no thin line protrudes past the tip; `shrinkA`
+    pulls the tail away from the text so they no longer overlap.
+  * "flit-level interleaving on wires" (not "shared FIFO").
+  * Destination: queue -> drain -> processing logic (no "served").
+
+Output:
+    docs/report/1H-codesign-paper/figures/latency_model.png
+
+Per /paper isolation: this is a report-only harness under scripts/paper/.
+"""
+
+import math
+from pathlib import Path
+
+import matplotlib.patches as patches
+import matplotlib.pyplot as plt
+
+OUT = Path(
+    "/Users/ywkang/kernbench/docs/report/1H-codesign-paper/figures/latency_model.png"
+)
+OUT.parent.mkdir(parents=True, exist_ok=True)
+
+
+# --- Colours --------------------------------------------------------------
+C_A      = "#E07B5E"
+C_B      = "#5E9BD1"
+C_NODE   = "#FFFFFF"
+C_BORD   = "#222222"
+C_WIRE   = "#222222"
+C_ANN    = "#333333"
+C_DIM    = "#777777"
+C_SWITCH = "#FAFAFA"
+C_OQUEUE = "#EFEFEF"
+C_PROC   = "#F4ECDC"
+C_QUEUE  = "#3CB371"   # medium-sea-green: distinctive vs A/B flit colours
+
+
+# --- Canvas ---------------------------------------------------------------
+fig = plt.figure(figsize=(17.0, 5.0))
+ax = fig.add_subplot(111)
+ax.set_xlim(0, 33)
+ax.set_ylim(2.2, 11.2)
+ax.axis("off")
+
+
+# --- Top header: end-to-end latency formula -----------------------------
+ax.text(
+    16.5, 10.6,
+    r"End-to-end latency  =  $\Sigma$ per-node overhead  +  "
+    r"$\Sigma$ per-edge transmission  +  drain  +  "
+    r"queuing delay",
+    ha="center", fontsize=12, color=C_ANN, weight="bold",
+)
+ax.plot([1.5, 31.5], [10.05, 10.05], color=C_DIM, lw=0.6)
+
+
+# --- Helpers --------------------------------------------------------------
+def box(cx, cy, w, h, label, fs=11, fweight="normal"):
+    ax.add_patch(patches.FancyBboxPatch(
+        (cx - w / 2, cy - h / 2), w, h,
+        boxstyle="round,pad=0.05",
+        facecolor=C_NODE, edgecolor=C_BORD, linewidth=1.6,
+    ))
+    if label:
+        ax.text(cx, cy, label, ha="center", va="center",
+                fontsize=fs, weight=fweight)
+
+
+def draw_flit_aligned(cx, cy, angle_rad, w, h, color, label):
+    """Draw a flit polygon rotated to align with the wire angle."""
+    cos_a, sin_a = math.cos(angle_rad), math.sin(angle_rad)
+    corners = []
+    for dx, dy in [(-w / 2, -h / 2), (w / 2, -h / 2),
+                   (w / 2, h / 2), (-w / 2, h / 2)]:
+        rx = dx * cos_a - dy * sin_a
+        ry = dx * sin_a + dy * cos_a
+        corners.append((cx + rx, cy + ry))
+    ax.add_patch(patches.Polygon(
+        corners, facecolor=color, edgecolor="black", linewidth=0.5,
+    ))
+    ax.text(cx, cy, label, ha="center", va="center",
+            fontsize=9, color="white", weight="bold")
+
+
+def draw_wire_with_flits(x0, y0, x1, y1, n_flits, label_char,
+                         color=None, flit_w=0.66, flit_h=0.95, gap=0.10):
+    """Wire (line) + endpoint arrowhead + flits centred on the line.
+
+    Wires are drawn in the neutral C_WIRE colour: the wire itself is
+    the place where the *transmission* delay accumulates (flit_size /
+    BW), not where flits queue. Queuing happens before the wire, at
+    the FIFO at the egress side -- coloured separately.
+    """
+    head_back = 0.30
+    dx, dy = x1 - x0, y1 - y0
+    wire_len = math.hypot(dx, dy)
+    ux, uy = dx / wire_len, dy / wire_len
+    x_line_end = x1 - ux * head_back
+    y_line_end = y1 - uy * head_back
+    ax.plot([x0, x_line_end], [y0, y_line_end],
+            color=C_WIRE, lw=1.6, zorder=1)
+    head_len, head_half = 0.30, 0.16
+    bx = x1 - ux * head_len
+    by = y1 - uy * head_len
+    px, py = -uy, ux
+    tri = [
+        (x1, y1),
+        (bx + px * head_half, by + py * head_half),
+        (bx - px * head_half, by - py * head_half),
+    ]
+    ax.add_patch(patches.Polygon(tri, facecolor=C_WIRE,
+                                  edgecolor=C_WIRE, linewidth=0.0))
+    # Flit train
+    angle_rad = math.atan2(dy, dx)
+    train_len = n_flits * flit_w + (n_flits - 1) * gap
+    centre_p = 0.5
+    start_p  = centre_p - (train_len / 2) / wire_len
+    step_p   = (flit_w + gap) / wire_len
+    if isinstance(label_char, str):
+        labels = [label_char] * n_flits
+        cols   = [color] * n_flits
+    else:
+        labels = [c[0] for c in label_char]
+        cols   = [c[1] for c in label_char]
+    for i in range(n_flits):
+        p = start_p + (i + 0.5) * step_p
+        cx = x0 + p * dx
+        cy = y0 + p * dy
+        draw_flit_aligned(cx, cy, angle_rad,
+                          flit_w, flit_h, cols[i], labels[i])
+
+
+def draw_router(cx, cy, w, h, two_inputs=True):
+    """Same-size Router with explicit internal switching logic and an
+    output queue.  Returns the wire-attachment (x,y) for each input
+    port and the single output port:
+      ((in1_x, in1_y), (in2_x, in2_y) or None, (out_x, out_y))
+    """
+    # Outer box
+    box(cx, cy, w, h, "")
+    ax.text(cx, cy + h / 2 - 0.32, "Router",
+            ha="center", fontsize=10, weight="bold")
+
+    # Input ports (small circles on the left edge)
+    in_x = cx - w / 2
+    in_x_internal = in_x + 0.25
+    if two_inputs:
+        in_y_top = cy + 0.55
+        in_y_bot = cy - 0.55
+        for iy in (in_y_top, in_y_bot):
+            ax.add_patch(patches.Circle(
+                (in_x_internal, iy), 0.12,
+                facecolor="white", edgecolor=C_BORD, linewidth=0.9,
+            ))
+    else:
+        in_y_top = None
+        in_y_bot = cy
+        ax.add_patch(patches.Circle(
+            (in_x_internal, in_y_bot), 0.12,
+            facecolor="white", edgecolor=C_BORD, linewidth=0.9,
+        ))
+
+    # Switch (small box, centre-left-ish). sw_cy = cy so that the
+    # output queue and (single-input) input port are at the same y as
+    # the router centre -- this keeps all router-to-router edges
+    # strictly horizontal.
+    sw_w, sw_h = 0.85, 1.10
+    sw_cx = cx - 0.55
+    sw_cy = cy
+    # The switch is the router's processing logic -- colour it the
+    # same as the Destination Node's processing-logic block so the
+    # two read as the same "processing" concept.
+    ax.add_patch(patches.Rectangle(
+        (sw_cx - sw_w / 2, sw_cy - sw_h / 2), sw_w, sw_h,
+        facecolor=C_PROC, edgecolor=C_BORD, linewidth=0.8,
+    ))
+    ax.text(sw_cx, sw_cy, "switch", ha="center", va="center",
+            fontsize=7.5, style="italic")
+
+    # Short feeder lines (no arrowhead) from input ports to the
+    # switch.  We deliberately drop arrowheads here: the head + port
+    # circle were too small at this scale and read as an overlap.
+    sw_left_x = sw_cx - sw_w / 2
+    if two_inputs:
+        for iy in (in_y_top, in_y_bot):
+            ax.plot(
+                [in_x_internal + 0.12, sw_left_x],
+                [iy, sw_cy + 0.25 * ((iy - cy) / 0.55)],
+                color=C_DIM, lw=0.7, zorder=1,
+            )
+    else:
+        ax.plot(
+            [in_x_internal + 0.12, sw_left_x],
+            [in_y_bot, sw_cy],
+            color=C_DIM, lw=0.7, zorder=1,
+        )
+
+    # Output queue (small queue holding a couple of flits). This *is*
+    # a real queueing location, so its fill takes the C_QUEUE family.
+    oq_w, oq_h = 0.95, 0.55
+    oq_cx = cx + 0.55
+    oq_cy = sw_cy
+    ax.add_patch(patches.Rectangle(
+        (oq_cx - oq_w / 2, oq_cy - oq_h / 2), oq_w, oq_h,
+        facecolor="#D9F0E1", edgecolor=C_QUEUE, linewidth=0.9,
+    ))
+    ax.text(oq_cx, oq_cy + oq_h / 2 + 0.18, "FIFO",
+            ha="center", fontsize=7, color=C_DIM, style="italic")
+    # Two small flits inside (A and B) hinting at the in-flight contents
+    mini_w, mini_h = 0.22, 0.34
+    for i, (col, lab) in enumerate([(C_A, "A"), (C_B, "B")]):
+        mx = oq_cx - 0.30 + i * (mini_w + 0.06)
+        ax.add_patch(patches.Rectangle(
+            (mx, oq_cy - mini_h / 2), mini_w, mini_h,
+            facecolor=col, edgecolor="black", linewidth=0.3,
+        ))
+        ax.text(mx + mini_w / 2, oq_cy, lab,
+                ha="center", va="center",
+                fontsize=5.5, color="white", weight="bold")
+
+    # Short feeder line (no arrowhead) from switch into out queue
+    ax.plot(
+        [sw_cx + sw_w / 2, oq_cx - oq_w / 2],
+        [sw_cy, oq_cy],
+        color=C_DIM, lw=0.7, zorder=1,
+    )
+    # Short feeder line from out queue to out port
+    out_x_internal = cx + w / 2 - 0.25
+    ax.plot(
+        [oq_cx + oq_w / 2, out_x_internal - 0.12],
+        [oq_cy, oq_cy],
+        color=C_DIM, lw=0.7, zorder=1,
+    )
+    # Output port circle on the right edge (a port marker, not a
+    # queue -- the queue is on the wire that follows, not the port).
+    ax.add_patch(patches.Circle(
+        (out_x_internal, oq_cy), 0.12,
+        facecolor="white", edgecolor=C_BORD, linewidth=0.9,
+    ))
+
+    return (
+        (in_x_internal, in_y_top) if two_inputs else None,
+        (in_x_internal, in_y_bot),
+        (out_x_internal, oq_cy),
+        oq_cx,   # also return the out-queue centre so callouts can target it
+        oq_cy,
+    )
+
+
+# --- Box layout ---------------------------------------------------------
+# Requester centres set to match Router 1's input-port y so that
+# Edge 1A and Edge 1B are strictly horizontal. Box heights are kept
+# small enough that a visible gap separates the two Requester boxes.
+R1_TMP_CY = 7.0
+ReqA = (2.8, R1_TMP_CY + 0.55)   # = 7.55, matches in_y_top of R1
+ReqB = (2.8, R1_TMP_CY - 0.55)   # = 6.45, matches in_y_bot of R1
+box(*ReqA, 3.0, 0.85, "Requester\nNode A", fs=10)
+box(*ReqB, 3.0, 0.85, "Requester\nNode B", fs=10)
+
+R_W, R_H = 3.0, 2.8
+R1 = (10.5, 7.0)
+R2 = (20.0, 7.0)
+
+r1_in_top, r1_in_bot, r1_out, r1_oq_cx, r1_oq_cy = draw_router(
+    *R1, R_W, R_H, two_inputs=True,
+)
+_, r2_in_bot, r2_out, r2_oq_cx, r2_oq_cy = draw_router(
+    *R2, R_W, R_H, two_inputs=False,
+)
+
+# Destination Node (same height as Router; wide enough so queue +
+# drain + processing-logic all fit on a single horizontal row).
+Dst = (28.4, 7.0)
+Dst_W, Dst_H = 6.0, R_H   # match the Router height
+box(*Dst, Dst_W, Dst_H, "")
+ax.text(Dst[0], Dst[1] + Dst_H / 2 - 0.25, "Destination Node",
+        ha="center", fontsize=10, weight="bold")
+
+
+# --- Edges: requester -> router 1 (Edge 1A & 1B, with flits) ------------
+draw_wire_with_flits(
+    ReqA[0] + 1.6, ReqA[1],
+    r1_in_top[0] - 0.02, r1_in_top[1],
+    n_flits=4, color=C_A, label_char="A",
+)
+draw_wire_with_flits(
+    ReqB[0] + 1.6, ReqB[1],
+    r1_in_bot[0] - 0.02, r1_in_bot[1],
+    n_flits=4, color=C_B, label_char="B",
+)
+
+# Edge 2: router1 out -> router2 in (horizontal, interleaved)
+labels_e2 = [("A", C_A) if i % 2 == 0 else ("B", C_B) for i in range(8)]
+E2_y = r1_out[1]
+draw_wire_with_flits(
+    r1_out[0] + 0.02, r1_out[1],
+    r2_in_bot[0] - 0.02, r2_in_bot[1],
+    n_flits=8, label_char=labels_e2,
+)
+
+# Edge 3: router2 out -> destination (horizontal, interleaved)
+labels_e3 = [("A", C_A) if i % 2 == 0 else ("B", C_B) for i in range(4)]
+draw_wire_with_flits(
+    r2_out[0] + 0.02, r2_out[1],
+    Dst[0] - Dst_W / 2, r2_out[1],
+    n_flits=4, label_char=labels_e3,
+)
+
+
+# --- Destination internals: queue -> drain -> processing logic ----------
+# Light-green halo behind the queue area marks it as a queueing point.
+C_QUEUE_BG = "#D9F0E1"
+
+dy_main = Dst[1] - 0.10
+Dst_left   = Dst[0] - Dst_W / 2
+
+# Queue (4 flits, FIFO; rightmost is the next to drain, so the
+# order is set so the serve sequence after the in-flight A alternates
+# A (in drain) -> B -> A -> B -> A -- giving a clean BABA queue.
+qW, qH, qGap = 0.42, 0.62, 0.07
+q_labels = ["A", "B", "A", "B"]
+q_cols   = [C_A,  C_B,  C_A,  C_B]
+q_total  = len(q_labels) * qW + (len(q_labels) - 1) * qGap
+q_x_start = Dst_left + 0.35
+
+# Green halo behind the queue boxes (the queue itself is a queueing
+# location -- mark it with the C_QUEUE colour family)
+ax.add_patch(patches.FancyBboxPatch(
+    (q_x_start - 0.10, dy_main - qH / 2 - 0.08),
+    q_total + 0.20, qH + 0.16,
+    boxstyle="round,pad=0.02",
+    facecolor=C_QUEUE_BG, edgecolor=C_QUEUE,
+    linewidth=0.9, zorder=1.5,
+))
+
+for i, (lab, col) in enumerate(zip(q_labels, q_cols)):
+    qx = q_x_start + i * (qW + qGap)
+    ax.add_patch(patches.Rectangle(
+        (qx, dy_main - qH / 2), qW, qH,
+        facecolor=col, edgecolor="black", linewidth=0.4,
+        zorder=2,
+    ))
+    ax.text(qx + qW / 2, dy_main, lab,
+            ha="center", va="center",
+            fontsize=8, color="white", weight="bold", zorder=3)
+# Common y for the "queue" / "drain" labels. Matches the FIFO label
+# spacing inside the Router (0.18 above the box top) and uses the same
+# font size for visual consistency.
+DST_LABEL_Y = dy_main + qH / 2 + 0.18
+ax.text(q_x_start + q_total / 2, DST_LABEL_Y,
+        "queue",
+        ha="center", fontsize=7, style="italic", color=C_DIM)
+
+# Drain slot (height matched to queue boxes; font matched to FIFO/queue
+# labels for visual consistency)
+dr_W, dr_H = 0.85, qH
+dr_x = q_x_start + q_total + 0.35
+dr_cx = dr_x + dr_W / 2
+dr_cy = dy_main
+ax.add_patch(patches.FancyBboxPatch(
+    (dr_x, dr_cy - dr_H / 2), dr_W, dr_H,
+    boxstyle="round,pad=0.03",
+    facecolor=C_A, edgecolor="black", linewidth=1.0,
+))
+ax.text(dr_cx, dr_cy, "A", ha="center", va="center",
+        fontsize=9, color="white", weight="bold")
+ax.text(dr_cx, DST_LABEL_Y,
+        "drain",
+        ha="center", fontsize=7, style="italic", color=C_DIM)
+
+# Drain-time double-arrow underneath the drain slot. Move the "drain
+# time" text a touch further down so the descender does not collide
+# with the arrowhead.
+dt_y = dr_cy - dr_H / 2 - 0.28
+ax.annotate(
+    "", xy=(dr_x + dr_W, dt_y), xytext=(dr_x, dt_y),
+    arrowprops=dict(arrowstyle="<->", lw=0.9, color=C_ANN),
+)
+ax.text(dr_cx, dt_y - 0.42, "drain time",
+        ha="center", fontsize=8.5, color=C_ANN, style="italic")
+
+# Processing-logic block (after drain) — wider so the two-line text
+# does not collide with the box edges
+pl_W, pl_H = 1.75, 1.00
+pl_x = dr_x + dr_W + 0.40
+pl_cx = pl_x + pl_W / 2
+pl_cy = dr_cy
+ax.add_patch(patches.FancyBboxPatch(
+    (pl_x, pl_cy - pl_H / 2), pl_W, pl_H,
+    boxstyle="round,pad=0.03",
+    facecolor=C_PROC, edgecolor=C_BORD, linewidth=1.0,
+))
+ax.text(pl_cx, pl_cy, "processing\nlogic",
+        ha="center", va="center", fontsize=9)
+
+# Small gray arrows along the queue -> drain -> processing chain
+ax.annotate(
+    "", xy=(dr_x - 0.04, dr_cy),
+    xytext=(q_x_start + q_total + 0.13, dr_cy),
+    arrowprops=dict(arrowstyle="-|>", lw=0.7,
+                    color=C_DIM, mutation_scale=7),
+)
+ax.annotate(
+    "", xy=(pl_x - 0.04, dr_cy),
+    xytext=(dr_x + dr_W + 0.04, dr_cy),
+    arrowprops=dict(arrowstyle="-|>", lw=0.7,
+                    color=C_DIM, mutation_scale=7),
+)
+
+
+# --- Annotations (strictly vertical, no diagonals, no overlap) ---------
+
+def vcallout(text, xy, x_text_top_y, fs=10, clearance=0.55,
+             marker_color=None):
+    """Vertical dotted callout. Optional `marker_color` paints a small
+    coloured circle just left of the text -- used to tie the label to
+    a colour code in the diagram.
+    """
+    text_y = x_text_top_y
+    if text_y < xy[1]:
+        line_top_y = text_y + clearance
+    else:
+        line_top_y = text_y - clearance
+    ax.plot([xy[0], xy[0]], [xy[1], line_top_y],
+            color=C_ANN, lw=0.9, linestyle=":", zorder=2)
+    if marker_color is not None:
+        # Text-width estimate at fs=10 (~0.18 data units per char) so
+        # the marker is placed clearly to the left of the text.
+        text_w = len(text) * 0.18
+        marker_x = xy[0] - text_w / 2 - 0.30
+        ax.add_patch(patches.Circle(
+            (marker_x, text_y), 0.14,
+            facecolor=marker_color, edgecolor=C_BORD, linewidth=0.6,
+            zorder=3,
+        ))
+    ax.text(xy[0], text_y, text,
+            ha="center", va="center",
+            fontsize=fs, color=C_ANN, zorder=3)
+
+
+# Transmission delay -> flit on Edge 2 (text BELOW the wire).
+# Extra clearance ~ 2 x text height so the line stops well clear of
+# the label.
+e2_total = 8 * 0.66 + 7 * 0.10
+e2_centre = (r1_out[0] + r2_in_bot[0]) / 2
+e2_start = e2_centre - e2_total / 2
+trans_idx = 5
+trans_cx  = e2_start + (trans_idx + 0.5) * (0.66 + 0.10)
+vcallout(
+    "transmission delay  =  flit_size / BW",
+    xy=(trans_cx, E2_y - 0.55),
+    x_text_top_y=E2_y - 2.6,
+    clearance=0.45,
+)
+
+# Flit-level interleaving -> ABOVE the wire, anchored on a flit near
+# the centre/right of the wire. Text y matches the queuing-delay
+# callout so the two labels sit on the same horizontal row, separated
+# horizontally so they don't collide.
+int_idx = 4
+int_cx  = e2_start + (int_idx + 0.5) * (0.66 + 0.10)
+vcallout(
+    "flit-level interleaving on wires",
+    xy=(int_cx, E2_y + 0.55),
+    x_text_top_y=R1[1] + R_H / 2 + 0.85,
+)
+
+# Queuing delay -> at Router 1 out-queue. Text positioned just above
+# the Router-1 box.
+vcallout(
+    "queuing delay",
+    xy=(r1_oq_cx, r1_oq_cy + 0.30),
+    x_text_top_y=R1[1] + R_H / 2 + 0.85,
+    marker_color=C_QUEUE,
+)
+
+# Drain -> below the drain slot (text is below).
+# Extra clearance ~ 2.5 x text height so the line stops further from
+# the label.
+vcallout(
+    "drain  =  per-flit service occupancy",
+    xy=(dr_cx, dt_y - 0.66),   # just *inside* the Destination Node
+                                  # bottom edge -- the dotted line then
+                                  # penetrates the box slightly rather
+                                  # than hanging below it
+    x_text_top_y=E2_y - 2.6,
+    clearance=0.60,
+)
+
+# Per-node overhead -> points at Router 1's switch block (where the
+# component's fixed processing cost lives). Text on the same row as
+# transmission delay and drain so the three "below the wire" callouts
+# sit on one horizontal baseline.
+R1_SW_CX = R1[0] - 0.55         # sw_cx for Router 1
+R1_SW_BOTTOM_Y = R1[1] - 0.55   # bottom of sw box
+vcallout(
+    "per-node overhead",
+    xy=(R1_SW_CX, R1_SW_BOTTOM_Y),
+    x_text_top_y=E2_y - 2.6,
+    clearance=0.60,
+    marker_color=C_PROC,
+)
+
+
+# --- Legend (close to the diagram) --------------------------------------
+LX, LY = 1.0, 2.6
+ax.add_patch(patches.Rectangle((LX, LY), 0.7, 0.45,
+                                facecolor=C_A, edgecolor="black",
+                                linewidth=0.4))
+ax.text(LX + 0.95, LY + 0.22, "Transaction A flit",
+        va="center", fontsize=10)
+
+ax.add_patch(patches.Rectangle((LX + 4.8, LY), 0.7, 0.45,
+                                facecolor=C_B, edgecolor="black",
+                                linewidth=0.4))
+ax.text(LX + 5.75, LY + 0.22, "Transaction B flit",
+        va="center", fontsize=10)
+
+
+# --- Save ----------------------------------------------------------------
+fig.savefig(OUT, dpi=140, bbox_inches="tight", facecolor="white")
+print(f"Wrote {OUT}")