Memory-level parallelism (MLP), phases 1 and 2

AGENTS.md

@@ -34,8 +34,7 @@ This is the canonical entry point for **both human contributors and AI coding ag
 - [CONTRIBUTING.md](CONTRIBUTING.md) — public fork/PR contribution flow
 - [docs/bug_fixes.md](docs/bug_fixes.md) — bug-fix discipline (root-cause vs patch)
 - [docs/continuous_integration.md](docs/continuous_integration.md) — CI pipeline
-- [docs/proposals/](docs/proposals/) — design and migration proposals (drafts and in-progress)
-- [docs/designs/](docs/designs/) — accepted designs (post-proposal, post-implementation)
+- [docs/designs/](docs/designs/) — accepted designs (implemented)
 
 ---
 
@@ -154,19 +153,11 @@ See [docs/coding_guidelines_cpp.md](docs/coding_guidelines_cpp.md) and [docs/cod
 
 ---
 
-## 7. Proposals
-
-Design and migration proposals belong in [`docs/proposals/`](docs/proposals/) — never in build dirs or the repo root. A proposal captures the *why* of a non-trivial change before the code lands.
-
-Accepted designs that outlive their proposal phase live in [`docs/designs/`](docs/designs/).
-
----
-
-## 8. Living Document Policy
+## 7. Living Document Policy
 
 This file is versioned alongside the code and evolves with it. Both humans and agents are expected to update it:
 
 - **Add a rule** when you discover a footgun the next contributor will hit. Include the *why*.
 - **Update or remove a rule** that's gone stale. A stale rule is worse than no rule.
 - **Don't duplicate** content from the topic docs in `docs/`. AGENTS.md links; the topic doc explains.
-- **Branch-specific addenda are forbidden.** Branch context goes in a `docs/proposals/<feature>_proposal.md`, not in branch-local `AGENTS.md` variants. If the rule is general, promote it here; if it's a project-tracking detail, keep it in the proposal.
+- **Branch-specific addenda are forbidden.** No branch-local `AGENTS.md` variants: if a rule is general, promote it here; project-tracking details stay out of the repo docs.

CHANGELOG.md

@@ -12,6 +12,31 @@ follows the version pins recorded in [VERSION](VERSION) (`VORTEX_VERSION`,
 - **Hardware profiles.** `profiles/<name>.json` names a machine (`tinygpu`, `rtx1080`, `v100`, `a100`, `h100`, `rtx5080`) that every flow
   builds through `configure --profile=<name>` or `blackbox.sh --profile=<name>`; explicit `CONFIGS` win over the profile.
   See [docs/simulation.md](docs/simulation.md#hardware-profiles).
+- **Per-partition fetch (`VX_CFG_FETCH_WIDTH`, default 1).** The front end splits into FETCH_WIDTH slices, each with its own warp
+  scheduler, fetch, decode and icache port, serving the warps with `wid % FETCH_WIDTH == f`; CTA dispatch, barriers and warp control
+  stay core-wide. The core takes one of three structures from the fetch width, issue width and per-unit block counts; with one block
+  of every unit per issue slice and FETCH_WIDTH equal to ISSUE_WIDTH, each slice is a self-contained compute slice (the NVIDIA
+  processing-block organization). *Why:* sgemm at 16 warps x 16 threads, ISSUE_WIDTH 4: -31.6% cycles at FETCH_WIDTH 2.
+- **Hashed cache bank select (`VX_CFG_{DCACHE,L2}_BANK_HASH`).** The bank index is the raw bank bits XORed with a hash of the line
+  address above them (IPOLY(5) for 16 banks, IPOLY(37) for 32, an XOR fold otherwise), so power-of-two strides spread over the banks.
+  On by default for the L2 and for a dcache with 4 or more banks. *Why:* rtlsim sgemv 512x512 at 16 threads -32.8%, fp16 TCU SGEMM
+  on 2 cores + L2 at 16 threads -13.8%, WGMMA with DXA -13.5%.
+- **Dcache read merge (`VX_CFG_DCACHE_READ_MERGE`, on by default).** Plain reads of the same word from the ports of one LSU block in
+  the same cycle are served by one bank access, and the response is multicast to every port; each port's order into the bank is
+  unchanged. *Why:* uniform loads (kernel arguments, SGEMM's shared operand) no longer serialize on one bank: rtlsim sgemm -n128 at
+  16 threads -6.5%, -17.2% with an 8-byte dcache word.
+- **Write-allocate without fetch (`VX_CFG_{DCACHE,L2,L3}_LAZY_FETCH`, off by default).** A store miss in a write-back cache installs
+  the sector with its dirty-byte mask and a partial bit instead of fetching the line; a later load or AMO merges memory data under the
+  unwritten bytes, and evictions write back only dirty bytes. *Why:* streaming writes no longer read every line first: 1-core vecadd
+  DRAM reads -33%.
+- **MSHR line table (`VX_CFG_{DCACHE,L2,L3}_MSHR_LINES`, off by default).** Misses are tracked in a reference-counted table of
+  distinct {line, sector} entries, and request entries keep only an index into it, so the MSHR holds more requests without widening
+  its address compares. *Why:* at 400-cycle memory latency, 64 requests over 16 lines: sgemm -n128 -15.3%, raycast -8.1%.
+- **Command-processor DMA through the shared LLC (`VX_CFG_CP_LLC_CLIENT`, off by default).** When one LLC serves every core, the
+  CP's transfers enter it on their own client slot, so host transfers need only an L1 flush, not a walk of the whole LLC; transfers
+  of at least half the LLC write around it. A launch behind DMA writes waits for an L1-only flush token sent on the slot. The runtime
+  sees the new `DMA_LLC_CLIENT` CP capability. *Why:* rtlsim at 2 clusters + L2 + L3: dogfood -30%, multikernel -42.7%; dogfood
+  -64% with an 8 MB L3.
 
 ### Changed
 
@@ -23,6 +48,48 @@ follows the version pins recorded in [VERSION](VERSION) (`VORTEX_VERSION`,
   (JEDEC HBM2/HBM3/GDDR6 organization and nanosecond timing, 32 B bursts, permuted open-page bank interleave, per-channel
   responses), configured from data-sheet values: `VX_CFG_PLATFORM_MEMORY_{TYPE,DATA_RATE,LATENCY,QUEUE_SIZE}`. The default machine
   keeps its idle read latency (14/26 cycles open-row/conflict, was 13/27) and one-block-per-cycle peak.
+- **Cache flush architecture.** The CP starts every core's flush with one broadcast `CACHE_FLUSH` DCR write, then reads each core's
+  completion, so the flushes overlap; a flush read without a preceding write no longer starts one. A cache walks its banks, sends one
+  flush token down every memory port and answers once all ports have; every independently buffered write path (each dcache lane, each
+  OM input of the ocache) carries its own token. Read-only caches (icache, tcache, rtcache, rcache) no longer forward flushes.
+  `DBG_FLUSH_STALL` adds random input backpressure to the flush path for stress tests.
+- **L1-only flush between launches.** When a single LLC serves every memory client it is the point of coherence, so a launch
+  boundary cleans only the L1s: `CMD_CACHE_FLUSH` gains an L1-only flag, carried as a flush-token kind that a shared LLC answers
+  without a walk. Replicated LLCs and LLCs with peer clients (TEX, RTU, RASTER, OM, DXA) still walk. The runtime posts flushes
+  lazily and flushes fully before any device-memory transfer, and device busy stays high until every core write, uncached IO stores
+  included, is visible in memory. *Why:* per-launch flush cost at 4 cores + L2 drops from 2,750 to 180 cycles.
+- **Targeted fences.** A memory fence sends an L1-only flush token that the shared LLC answers without a walk. A fence that orders
+  only device (I/O) accesses decodes as its own LSU op, and its token makes each level drain to its memory ports without walking.
+  *Why:* printf, which fences after each device write, runs 13.6x faster on rtlsim at 2 cores + L2.
+- **LMEM DMA arbitration.** DXA writes into shared memory and the TCU's WGMMA operand reads now share the local-memory DMA port
+  round-robin. DXA had fixed priority and starved the TCU for about 510 cycles per tiled-scatter transfer. *Why:* rtlsim WGMMA with
+  DXA -10.2%, sparse WGMMA with DXA -11.6%.
+- **SimX timing model follows the RTL's structure.** Buffer capacities and registered ready at every pipeline boundary, per-slice
+  cache arbiter trees and network depths, registered cache core-response ports, flush and fence tokens end to end, and CTA launch onto
+  the nearest ready core with two launch credits per core. SimX's per-core cycle counter (`mcycle`) now advances only while the core
+  has work, as the RTL's does. rtlsim completes DRAM requests in per-bank arrival order, as the AXI platform does.
+
+### Fixed
+
+- **Global barriers across clusters.** Each cluster ran its own global-barrier unit that saw only its own cores, while
+  `vx_num_cores()`, and so `vortex::gbarrier`, counts the whole device. On SimX every global barrier hung with more than one
+  cluster; on the RTL, 2 clusters x 1 core timed out and 2 clusters x 2 cores released each cluster on its own arrivals, with no
+  ordering between clusters. One device-level unit now counts every core.
+- **Writes left dirty by a flush.** A cache forwarded its flush token on memory port 0 only, so writes queued on another port could
+  reach the next level after that level's walk and stay dirty there.
+- **Dirty sectors dropped on a sector refill.** With dirty-byte tracking and several sectors per line, installing a sector into a
+  resident line cleared the whole line's byte mask, so a dirty sibling sector wrote back with no bytes enabled and its data was lost
+  (mstress at 4 cores with a write-back, sectored L2).
+- **Local-memory LR/SC livelock at 16 threads.** One warp's per-lane LR burst used up a reservation's fixed hold budget of 7 refusals,
+  so contended LR/SC loops on shared memory stopped making progress. The budget now scales with the core's harts.
+- **`VX_stream_arb` trees with a partial last slice.** When the request count was not a whole number of slices, the last slice's
+  select was sized and placed as if it were full: the build failed lint, and with several outputs requests went to the wrong output.
+- **`-DSCOPE` builds.** The scope buses of fetch, issue, LSU, TCU, TEX, RASTER and OM were declared only under their `DBG_SCOPE_*`
+  taps, left unbound, or did not parse, so debug-scope builds failed to elaborate.
+- **mstress checked no data.** Its host reference was always 0, so the test passed with corrupted loads; it now checks every loaded
+  value exactly.
+- **FPGA run instructions.** The docs exported `TARGET=hw`, which `blackbox.sh` ignores, so runs fell back to xrtsim; they now pass
+  `--target=hw`.
 
 ## [3.0] — 2026-10-07
 

VX_config.toml

@@ -56,9 +56,11 @@ VX_CFG_MAX_BAR_EVENTS = 32
 VX_CFG_IBUF_SIZE = 4
 VX_CFG_DISPATCH_QUEUE_SIZE = 4
 VX_CFG_ISSUE_WIDTH = "expr: up($VX_CFG_NUM_WARPS / 16)"
+# Fetch slices, each scheduling and fetching its own warps on its own icache
+# port: a power of two that divides ISSUE_WIDTH.
+VX_CFG_FETCH_WIDTH = 1
 VX_CFG_SIMD_WIDTH = "expr: $VX_CFG_NUM_THREADS"
 VX_CFG_NUM_OPCS = "expr: up($VX_CFG_NUM_WARPS / (4 * $VX_CFG_ISSUE_WIDTH))"
-
 VX_CFG_NUM_GPR_BANKS = 4
 VX_CFG_NUM_VGPR_BANKS = 2
 
@@ -205,6 +207,13 @@ l3_is_llc = "expr: $VX_CFG_L3_ENABLED == 1"
 single_core = "expr: $VX_CFG_NUM_CORES == 1 and $VX_CFG_NUM_CLUSTERS == 1"
 single_cluster = "expr: $VX_CFG_NUM_CLUSTERS == 1"
 
+# The command processor's DMA becomes a client of the last-level cache, which
+# is then the point of coherence for host transfers: they need no cache flush.
+# Only a last-level cache that alone serves every core can take it (an L2 with
+# one cluster, or an L3); VX_CFG_CP_ON_LLC is set where it took effect.
+VX_CFG_CP_LLC_CLIENT = 0
+VX_CFG_CP_ON_LLC = "expr: int($VX_CFG_CP_LLC_CLIENT != 0 and (($l2_is_llc and $single_cluster) or $l3_is_llc))"
+
 [l1cache]
 VX_CFG_ICACHE_SIZE = 16384
 VX_CFG_ICACHE_NUM_WAYS = 4
@@ -215,6 +224,11 @@ VX_CFG_ICACHE_MRSQ_SIZE = 0
 VX_CFG_ICACHE_MREQ_SIZE = 0
 VX_CFG_ICACHE_CRSQ_SIZE = 0
 VX_CFG_ICACHE_MEM_PORTS = 1
+# banks behind the fetch-slice request ports
+VX_CFG_ICACHE_NUM_BANKS = "expr: pow(2, clog2($VX_CFG_FETCH_WIDTH))"
+# Merge same-line fetches from the core's slices in the same cycle into one
+# bank access; the response goes to every merged slice, each with its word.
+VX_CFG_ICACHE_READ_MERGE = 1
 VX_CFG_ICACHE_LATENCY = "expr: 2 + max(0, clog2($VX_CFG_ICACHE_SIZE) - clog2(16384))"
 
 VX_CFG_DCACHE_SIZE = 16384
@@ -224,9 +238,18 @@ VX_CFG_DCACHE_NUM_WAYS = 4
 # stays moderate: NT 1/2/4/8/16/32/64 -> word 4/8/8/16/16/32/32, banks 1/1/2/2/4/4/8.
 VX_CFG_DCACHE_WORD_SIZE = "expr: min($VX_CFG_L1_LINE_SIZE, min($VX_CFG_MEM_BLOCK_SIZE, ($VX_CFG_XLEN / 8) * pow(2, (clog2($VX_CFG_NUM_LSU_LANES) + 1) / 2)))"
 VX_CFG_DCACHE_WRITEBACK = "expr: int($dcache_is_llc and $single_core)"
-VX_CFG_DCACHE_DIRTYBYTES = 0
+# LAZY_FETCH (write-back LLC only): a store miss allocates its line without a
+# fill and a later read fetches and merges under the dirty bytes. It needs the
+# per-byte dirty masks, so it turns DIRTYBYTES on; DIRTYBYTES alone keeps
+# fetch-on-write. The same pair exists for L2 and L3.
+VX_CFG_DCACHE_DIRTYBYTES = "expr: $VX_CFG_DCACHE_LAZY_FETCH"
+VX_CFG_DCACHE_LAZY_FETCH = 0
 VX_CFG_DCACHE_REPL_POLICY = "expr: $__cache_repl_fifo"
 VX_CFG_DCACHE_MSHR_SIZE = 16
+# MSHR line entries: distinct {line, sector} misses a bank tracks, each naming
+# the chain of requests merged on it; MSHR_SIZE then sizes the request pool.
+# 0 keeps the address in every request entry (lines = requests).
+VX_CFG_DCACHE_MSHR_LINES = 0
 # dcache is unsectored: line = sector = mem transaction granule = L1_LINE (which
 # defaults to MEM_BLOCK but can be configured smaller). Banking comes from the
 # reduced WORD above (>= sector is clamped), not the line. Deriving from L1_LINE
@@ -243,31 +266,46 @@ VX_CFG_NUM_ICACHES = "expr: up($VX_CFG_SOCKET_SIZE / 4) if $VX_CFG_ICACHE_ENABLE
 VX_CFG_NUM_DCACHES = "expr: up($VX_CFG_SOCKET_SIZE / 4) if $VX_CFG_DCACHE_ENABLE else 0"
 
 VX_CFG_DCACHE_NUM_BANKS = "expr: pow(2, clog2(min($VX_CFG_DCACHE_NUM_REQS, 16)))"
+# XOR/IPOLY bank-select hash, so power-of-two strides spread over the banks.
+# With two banks it can only swap a line on parity, and the extra parallel
+# misses cost more in memory queuing than the spreading saves.
+VX_CFG_DCACHE_BANK_HASH = "expr: int($VX_CFG_DCACHE_NUM_BANKS >= 4)"
+# One bank access serves a warp's same-word reads arriving together on its
+# coalescer channels (uniform loads), and the response returns to all of
+# them in the same cycle.
+VX_CFG_DCACHE_READ_MERGE = 1
 
 VX_CFG_L1_MEM_PORTS = "expr: min($VX_CFG_DCACHE_NUM_BANKS, $VX_CFG_PLATFORM_MEMORY_NUM_BANKS) if ($VX_CFG_ICACHE_ENABLE or $VX_CFG_DCACHE_ENABLE) else min($VX_CFG_DCACHE_NUM_REQS, $VX_CFG_PLATFORM_MEMORY_NUM_BANKS)"
 
 [l2cache]
 VX_CFG_L2_SIZE = 1048576
 VX_CFG_L2_NUM_WAYS = 8
 VX_CFG_L2_WRITEBACK = "expr: int($l2_is_llc and $single_cluster)"
-VX_CFG_L2_DIRTYBYTES = 0
+VX_CFG_L2_DIRTYBYTES = "expr: $VX_CFG_L2_LAZY_FETCH"
+VX_CFG_L2_LAZY_FETCH = 0
 VX_CFG_L2_REPL_POLICY = "expr: $__cache_repl_fifo"
 VX_CFG_L2_MSHR_SIZE = 16
+VX_CFG_L2_MSHR_LINES = 0
 VX_CFG_L2_LATENCY = "expr: 4 + max(0, clog2($VX_CFG_L2_SIZE) - clog2(1048576))"
 VX_CFG_L2_MRSQ_SIZE = 4
 VX_CFG_L2_MREQ_SIZE = 0
 VX_CFG_L2_CRSQ_SIZE = 0
 
 VX_CFG_L2_NUM_BANKS = "expr: pow(2, clog2(min($VX_CFG_L2_NUM_REQS, 16)))"
+# On at every bank count: unlike the dcache, a 2-bank L2 measured neutral
+# (sgemm-mc-nt4 -0.6%, fp16-mc-nt4 +0.1%), and it gains from 4 banks up.
+VX_CFG_L2_BANK_HASH = 1
 VX_CFG_L2_MEM_PORTS = "expr: min($VX_CFG_L2_NUM_BANKS, $VX_CFG_PLATFORM_MEMORY_NUM_BANKS) if $VX_CFG_L2_ENABLE else min($VX_CFG_L2_NUM_REQS, $VX_CFG_PLATFORM_MEMORY_NUM_BANKS)"
 
 [l3cache]
 VX_CFG_L3_SIZE = 2097152
 VX_CFG_L3_NUM_WAYS = 8
 VX_CFG_L3_WRITEBACK = "expr: int($l3_is_llc)"
-VX_CFG_L3_DIRTYBYTES = 0
+VX_CFG_L3_DIRTYBYTES = "expr: $VX_CFG_L3_LAZY_FETCH"
+VX_CFG_L3_LAZY_FETCH = 0
 VX_CFG_L3_REPL_POLICY = "expr: $__cache_repl_fifo"
 VX_CFG_L3_MSHR_SIZE = 16
+VX_CFG_L3_MSHR_LINES = 0
 VX_CFG_L3_LATENCY = "expr: 4 + max(0, clog2($VX_CFG_L3_SIZE) - clog2(2097152))"
 VX_CFG_L3_MRSQ_SIZE = 4
 VX_CFG_L3_MREQ_SIZE = 0

VX_types.toml

@@ -76,6 +76,11 @@ VX_DCR_BASE_CACHE_FLUSH = 0x000
 VX_DCR_BASE_MPM_VALUE   = 0x001
 VX_DCR_BASE_STATE_END   = 0x002
 
+[cache_flush]
+# VX_DCR_BASE_CACHE_FLUSH write value: flush only the levels above the
+# last-level cache, which stays the point of coherence for the next kernel.
+VX_CACHE_FLUSH_L1_ONLY  = 0x10000
+
 [dcr_mmu]
 VX_DCR_MMU_STATE_BEGIN  = 0x004
 VX_DCR_MMU_SATP_LO      = 0x004