Caching Strategies for Heterogeneous Memory Systems
1. Introduction
Modern systems no longer have one kind of memory. A server might have local DDR5 DRAM, a CXL-attached memory expander two hops away on the fabric, HBM stacked on an accelerator, and a GPU’s own VRAM that the CPU can address but never wants to touch directly. Each of these tiers has a different latency, a different bandwidth ceiling, and — critically for this article — a different relationship to the CPU cache hierarchy. Some tiers are fully cache-coherent. Some are coherent but slow enough that treating them like DRAM is a performance bug. Some are not coherent with the CPU at all, and reading or writing them incorrectly produces silently wrong results instead of a crash.
This article is about the layer of software that sits between “the CPU’s normal cache hierarchy” and “whatever this new memory tier actually is”: a software-managed cache. We will build one, break it twice under real tools, fix it, and measure it. The mental model is simple — near memory is fast and CPU-cached, far memory is slow and may not be — but the implementation detail is where production systems succeed or quietly corrupt data.
2. Historical Background
Hardware cache coherence protocols (MESI and its descendants) were built on an assumption: there is one pool of DRAM, and every core’s cache is kept consistent with it by snooping or directory lookups. That assumption held for decades because it was cheaper to add more coherent DRAM than to manage two kinds of memory in software.
Two pressures broke it. First, DRAM scaling slowed while core counts kept climbing, so systems started attaching memory over PCIe/CXL links rather than only on local memory channels — that link’s latency is too large to treat as uniform-cost DRAM even though it is coherent. Second, accelerators (GPUs, smart NICs, FPGAs) came with their own local memory that the host CPU could map but not cheaply keep coherent with, because full hardware coherence across a PCIe link at CPU-cache granularity is either unavailable or too costly in practice. The kernel’s answer has been incremental: NUMA gave the scheduler and allocator a notion of memory “distance,” mempolicy and later kernel memory tiering (and DAMON, covered in an earlier article in this series) gave it a notion of hot/cold pages to migrate between tiers, and CXL 3.0’s HDM-DB gave hardware a coherence model for dynamically attached capacity. None of that solves the userspace problem this article covers: even when the kernel places pages correctly, the application still decides how to move bytes between a fast local buffer and a slow or non-coherent one, and that decision is where cache-management bugs live.
The instruction-level tools this article relies on are themselves an older lineage than CXL. Non-temporal stores and the write-combining (WC) memory type were introduced with SSE, originally to let software stream large amounts of data — video frame buffers, graphics textures — out to a device without thrashing the CPU cache. CLFLUSH is older still. What changed is not the instructions but the reason to reach for them: a decade ago they were a niche optimization for a narrow set of multimedia and driver code; today, with far-memory tiers a routine part of server design, knowing exactly which store instruction is being issued and what it guarantees has become a mainstream systems-programming skill rather than a specialist one.
3. Systems-Level Problem
Here is the concrete problem: you have a working set that is written frequently but only a fraction of it is hot at any moment. Writing every byte directly to far memory pays that tier’s latency on every access. Copying the whole thing into DRAM defeats the purpose of having a large, cheap far tier at all. The standard answer is a cache: keep hot lines in near memory, mark them dirty, and lazily write them back to far memory. That is a write-back cache, and it is precisely what a CPU does for you with DRAM — except now you are building it in software for a tier the CPU won’t manage automatically.
Two things make this harder than an ordinary in-memory cache:
Concurrency. Multiple threads hit the same near-memory line for different far-memory addresses. A lookup-then-claim sequence without atomicity produces two writers on one cache line.
Store ordering across a non-coherent or weakly-ordered boundary. If you use non-temporal (write-combining) stores to avoid polluting the CPU cache with far-memory writes — which you often want to, since that data will not be re-read locally — those stores are, by design, not ordered the way normal stores are. A consumer outside the CPU’s cache-coherence domain (a DMA engine, an accelerator polling a doorbell) can observe “this data is ready” before the data itself is globally visible, unless you fence explicitly.
Both of these are real bugs we will reproduce below, not hypotheticals.
There is a design question underneath both bugs that is worth stating explicitly: a software cache for heterogeneous memory has to make the same three decisions any cache makes — placement (direct-mapped, set-associative, or fully associative), write policy (write-back vs. write-through), and eviction policy (LRU, clock, random) — but each decision now has a heterogeneous-memory-specific cost attached to getting it wrong. A direct-mapped cache is simple and lock-friendly (one lock per line, as built here) but suffers more conflict misses than set-associative designs; that trade is more expensive here than in a hardware cache because a “miss” against far memory can mean microseconds, not nanoseconds. Write-back defers cost to eviction, which is exactly what you want when far-memory latency is high, but it means dirty data can be lost on a crash unless the eviction and durability story is designed together — the same tension this series’ NVM persistency article covered from the opposite direction (durability first, performance second). This article picks the simplest point in that space (direct-mapped, write-back, periodic writeback rather than LRU-driven eviction) deliberately, so the two real bugs are visible without a more complex design obscuring them.
4. Linux Kernel Architecture
