Data Allocation and Placement in Complex Heterogeneous Memory Systems
1. Introduction
Modern servers no longer have “memory” in the singular sense. A single machine can expose DRAM attached to multiple NUMA nodes, high-bandwidth memory (HBM) stacked on a GPU or accelerator package, persistent memory (PMEM) modules on the DDR bus, and — increasingly — CXL-attached memory expanders sitting one hop further away than local DRAM. Each tier has a distinct latency, bandwidth, capacity, and durability profile, and the Linux kernel has to decide, for every page it hands out, which physical memory backs that page.
This lesson is about that decision: how Linux models heterogeneous memory, how it decides where a page should live, how it migrates pages between tiers under pressure, and how a systems engineer observes and tunes that behavior in production. This is not a NUMA tutorial in the introductory sense — we assume you already understand what a NUMA node is. Instead we go one level deeper, into the tiered-memory and CXL-memory machinery that has landed in the kernel over the last several releases (node_states, memory tiers, demotion, numa_balancing, mempolicy, and the DAX/kmem driver path that turns PMEM or CXL capacity into ordinary zoned memory).
