Famfs Update: Navigating DAX Subsystem Challenges for Fabric-Attached Memory
1. Introduction
Every few years, a new memory technology forces the Linux kernel to answer a question it has never quite had to answer before. NUMA asked “what if memory access latency depends on which CPU is asking?” Persistent memory asked “what if memory survives a reboot?” Fabric-attached memory (FAM) — pools of DRAM sitting behind a Compute Express Link (CXL) switch, addressable by more than one host — asks something stranger: “what if memory belongs to nobody in particular, and several kernels need to agree on what’s stored in it at the same time?”
Famfs is the current, still-evolving answer. It is not a general-purpose filesystem, and its authors are the first to say so. It is a thin, purpose-built layer that lets ordinary mmap()-based applications treat a shared pool of CXL memory the way they already treat memory-mapped files, without teaching every application in the world a brand-new API for shared memory. Getting there means confronting the kernel’s Direct Access (DAX) subsystem at a point it was never quite designed to be pushed: metadata that must be trusted by strangers, page faults that must never leave the kernel, and a device model — device DAX — that most filesystems have never had to speak to directly.
This article is a systems-level tour of famfs as it stands today: a project that began as a small standalone kernel filesystem, hit a wall of legitimate maintainer skepticism, and is now being rebuilt on top of FUSE. That pivot is itself the most instructive part of the story, and we will spend real time on it, because it illustrates a recurring lesson in kernel engineering — the API you want is not always the API you’re allowed to have, and the path to upstream is as much a social negotiation as a technical one.
2. Historical Background
DAX itself dates to the early days of the kernel’s persistent-memory (pmem) support, introduced to let filesystems built on NVDIMM-class storage bypass the page cache and map storage pages directly into a process’s address space. mm/dax gave filesystems like XFS and ext4 an fs-dax mode: when a file lives on a DAX-capable block device, reads, writes, and mmap() faults go straight to persistent memory, with no page-cache copy in between. This was originally about eliminating an unnecessary memcpy for fast NVDIMMs, not about sharing memory between machines.
CXL changed the shape of the problem. A CXL 2.0/3.0 fabric can expose a pool of DRAM to multiple hosts through a switch, using CXL’s coherency and connection protocols rather than a local memory controller. That memory can be assigned to a single host as ordinary system RAM, or it can be left in a mode where the kernel is aware of it as an address range but does not manage it as general-purpose memory — this is devdax, the character-device face of the DAX subsystem (/dev/dax0.0 and friends), maintained separately from the block-device-oriented fs-dax/pmem path.
John Groves, a memory-vendor engineer, began circulating famfs patches in early 2024 as an RFC. The pitch was straightforward: applications already know how to call mmap(MAP_SHARED, ...) on a file; if a shared CXL memory pool can be presented as a set of files, those applications get scale-out shared memory “for free,” without adopting a new API that history shows most software will never actually adopt. The first versions of famfs were built as a standalone in-kernel filesystem, roughly a thousand lines of kernel code doing very ramfs-like inode management, layered on top of devdax.
That design got a fair hearing — and a fair amount of pushback — at the 2024 Linux Storage, Filesystem, Memory-Management and BPF Summit (LSFMM+BPF). The kernel filesystem and DAX maintainers raised two concerns that would come to define the next two years of the project: first, that a brand-new in-kernel filesystem for a narrow use case tends to accumulate feature creep and maintenance burden over time; second, and more specifically, that most of what famfs actually does — space allocation, log-append, metadata bookkeeping — already happens in user space, which raised the obvious question of whether FUSE, not a new native filesystem, was the right home for it. By the 2025 LSFMM+BPF summit, the consensus had shifted: famfs would be ported onto FUSE, keeping only the parts of the design that truly require kernel involvement — fault resolution against cached extent metadata — inside the kernel. As of the most recent patch bundles circulating on the kernel mailing lists in early-to-mid 2026, that port is well underway but not yet merged: the project is simultaneously maintaining a standalone fs-dax version and a new FUSE-based version, with the standalone version slated for deprecation once the FUSE path is complete.
3. Systems-Level Problem
To see why famfs exists at all, it helps to be precise about what “shared memory” means at the scale CXL enables. MAP_SHARED on a single host is a solved problem: the kernel’s page tables and page cache coordinate multiple processes mapping the same file, and cache coherency is handled by the CPU’s own coherency fabric. Fabric-attached memory breaks the assumption baked into that solution — that “the kernel” is a single, authoritative entity that can serialize all access to a piece of memory. With FAM, two or more independent kernels, on two or more independent hosts, can each see the same physical memory range through a CXL switch. Neither kernel is authoritative over the other.
Two consequences follow immediately. First, you cannot online fabric-attached memory as ordinary system RAM and expect it to behave like local DRAM: system RAM gets zeroed and managed by the buddy allocator and NUMA balancing logic as if it belonged to one host, and there is no mechanism for two hosts to coordinate updates to the same physical page through independent page tables and independent virtual memory subsystems. Second, you cannot use a conventional write-back filesystem’s metadata model — inodes, free-space bitmaps, and journals cached dirty in one host’s memory and flushed to disk on its own schedule — because there is no way for a second host to know whether the first host’s cached, unflushed metadata state is the “true” state. Existing fs-dax filesystems such as XFS and ext4 solve the data-movement problem beautifully (no page-cache copy for file data), but their metadata is still fundamentally single-host and write-back.
Famfs’s answer is deliberately narrow: don’t solve general-purpose shared filesystem semantics. Instead, provide an allocator whose allocations look like files, where metadata changes are captured in an append-only log rather than mutated in place, and where every host that mounts the filesystem replays that log to reconstruct an in-memory, ephemeral view of “what files exist and where their extents are.” There is no traditional inode written back to storage, because there is nothing to write inodes back to in the traditional sense — the log itself, stored in the shared memory, is the durable record. Concurrency and coherency of the data inside those files is explicitly left to the application; famfs only guarantees the integrity of its own allocation metadata.
4. Linux Kernel Architecture
Famfs sits at the intersection of three kernel subsystems that don’t normally have much to say to each other: the CXL/DAX device layer, the FUSE filesystem framework, and (in the original design) the fs-dax iomap machinery that XFS and ext4 also use.
The DAX device layer. A CXL memory device, once enumerated by drivers/cxl/, can be placed into one of several operating modes, switchable via daxctl or sysfs: system-ram mode, where the range is handed to the core memory management code and onlined as a NUMA node; and devdax mode, where the kernel deliberately keeps its hands off the memory’s contents and exposes it as a character device, /dev/daxN.M, that only an application (or a driver built for the purpose) touches directly. Famfs introduces a third mode on top of devdax: binding the device to a purpose-built driver — fsdev_dax (drivers/dax/fsdev.c) in the FUSE-based redesign — which marks the device as being managed by famfs rather than mapped raw by an application. daxctl gained explicit mode-detection helpers (daxctl_dev_is_famfs_mode(), daxctl_dev_is_devdax_mode()) precisely because, once a third mode exists, code that used to infer “not devdax means system-ram” silently becomes wrong.
FUSE and the famfs protocol extension. The redesigned famfs is a FUSE filesystem, meaning a user-space daemon — the famfs server — owns the log, performs allocation, and answers metadata requests, while fs/fuse/famfs.c in the kernel handles the performance-critical path: resolving a page fault against already-cached extent metadata without a round trip to user space. To make that possible, the FUSE wire protocol gained two new message types specific to famfs: GET_FMAP, which asks the server for the complete file-to-DAX-extent mapping for a file (the “fmap”) so the kernel can cache it in full, and GET_DAXDEV, which asks the server to identify a DAX device referenced by an fmap that the kernel client has not seen before. Once a file’s fmap has been fetched once, every subsequent fault against that file is satisfied purely from the kernel-resident cache — no upcall, no context switch to the famfs server, and therefore no latency penalty relative to a native fs-dax filesystem.
iomap and dev_dax_iomap. Because famfs targets devdax character devices rather than the block-oriented pmem interface that XFS and ext4 assume, the project also had to extend the DAX device layer with iomap support for character devices — patches with names like dev_dax_iomap: Add fs_dax_get() func to prepare dax for fs-dax usage and dev_dax_iomap: Add dax_operations for use by fs-dax on devdax — so that fs-dax-style operations (translating a file offset to a physical DAX offset, mapping and unmapping the resulting pages) work against a /dev/dax device the same way they already work against /dev/pmem. This was a deliberate, debated choice: pmem devices can be converted to devdax, but not the reverse, and existing DAX filesystems only knew how to speak the pmem/block dialect. Famfs is, as of this writing, the only fs-dax filesystem backed by devdax rather than pmem, which is why this plumbing had to be written rather than reused.
5. Internal Working
At mount time, a famfs client contacts the famfs server (running as a FUSE daemon, or in the legacy design, running as the process that performed mkfs.famfs and the initial log-play) and requests the current state of the filesystem’s log. The log is an append-only sequence of allocation and file-creation records living inside the shared DAX memory itself. Each client — including the one that created the filesystem — replays this log independently to build its own private, in-memory view of the directory tree and per-file extent lists. Nothing about this replayed state is written back anywhere; it is derived, not authoritative, and if a client crashes and remounts, it simply replays the log again.
When an application opens a famfs file and calls mmap(), the kernel does not yet know that file’s extents. The first access triggers a page fault, and it’s here that the FUSE-based design earns its keep: rather than handling every fault with an upcall to user space (which would make famfs no faster than a naive user-space-backed filesystem), the kernel issues a single GET_FMAP request the first time a file is touched, caching the complete extent list — the full mapping from file offsets to DAX-device physical offsets — in kernel memory. If that fmap references a DAX device the kernel hasn’t registered yet, a GET_DAXDEV request resolves which device it is and adds it to a per-connection daxdev_table. Every fault after that first one is resolved entirely from the cached fmap: the kernel computes the target DAX device and offset directly, maps the corresponding pages into the process’s address space, and returns — no server round trip, no scheduling latency from a context switch into user space.
This is the single design constraint that makes famfs worth having: full memory speed after warm-up. A filesystem that satisfied every fault with a FUSE upcall would add microseconds of latency to every page fault under memory pressure, which is fatal for a technology whose entire value proposition is “as fast as local DRAM.” Caching the complete fmap up front, rather than resolving faults lazily against partial metadata, is what avoids that.
Because famfs never caches dirty shared metadata — every allocation and file creation is captured as an immutable log entry the moment it happens, not accumulated in memory and flushed later — there is no window in which one host’s view of the filesystem’s structure can silently diverge from another’s due to write-back timing. Data written into the body of a file, as opposed to famfs’s own allocation metadata, is a different story: famfs deliberately does not attempt to arbitrate coherency or concurrent-write conflicts for application data. That is left to the application, which is a reasonable division of labor for the workloads famfs targets — large, mostly-write-once datasets that are populated once and then read by many consumers.
6. Step-by-Step Execution Flow
Github Link:
Device provisioning. An administrator (or orchestration layer) uses
daxctlto place a CXL-backed DAX device into famfs mode, binding it to thefsdev_daxdriver rather than leaving it in rawdevdaxmode or onlining it as system RAM.Filesystem creation. A
mkfs.famfs-equivalent user-space tool writes an initial superblock and an empty log into the beginning of the DAX range, establishing the filesystem’s identity.Server startup. The famfs server process (the FUSE daemon in the redesigned architecture) mounts the filesystem, taking ownership of subsequent allocation and log-append operations.
Client mount. Each additional host that wants access mounts the same famfs instance, by default as a read-only client, though writable client access is supported for workloads that need it.
Log replay. Every client — server included — replays the append-only log to reconstruct the current set of files, directories, and their DAX extents as an in-memory, non-authoritative view.
File creation / allocation (server side). When a new file is created, the server performs space allocation at 2 MiB granularity (huge-page aligned, matching CXL’s typical interleave and mapping granularity), pre-sizes the file, and appends an allocation record to the log. Famfs files never grow at write time; the full size is committed up front.
First access (client side). An application on any client opens the file and calls
mmap(). No page table entries are populated yet.Fault and fmap fetch. On first touch, the kernel issues
GET_FMAPto the server, caching the file’s full extent list. If new DAX devices are referenced,GET_DAXDEVresolves and registers them.Steady-state fault handling. Every subsequent fault against that file is resolved directly from the cached fmap — physical DAX offset computed, page mapped, no upcall.
Read/write. Because the mapping bypasses the page cache entirely, reads and writes through the mmap’d region touch DAX memory directly; there is no dirty-page writeback path to reason about for file data.
Failure path. If the underlying DAX memory reports a hardware error — a poisoned cacheline, a fabric timeout — the kernel does not attempt to keep the affected mapping alive. The consuming process is delivered
SIGBUS, and famfs disables the affected mount rather than risk operating on a filesystem with unverifiable memory state.Teardown. On unmount, or when a FUSE connection is torn down,
famfs_teardown()releases the per-connection metadata (daxdev_tableentries and cached fmaps) without touching the shared memory itself, since the memory’s actual content belongs to the log and the other live clients, not to any single connection.
7. Kernel Data Structures
The exact struct layouts are still moving as the FUSE port lands upstream, but the shapes are stable enough to reason about. A simplified sketch of the client-side caching structures looks like this:
/* Cached mapping from a famfs file's logical offsets to DAX extents.
* Populated once via GET_FMAP, then consulted on every subsequent
* page fault without further server interaction. */
struct famfs_fmap {
u64 inode_id; /* stable identity from the log */
u32 nextents;
struct famfs_extent {
u64 file_offset; /* offset within the famfs file */
u64 dax_offset; /* offset within the DAX device */
u64 length; /* huge-page-aligned length */
u32 daxdev_index; /* index into daxdev_table */
} extents[];
};
/* Per-FUSE-connection table of DAX devices this client has resolved
* via GET_DAXDEV. Populated lazily the first time an fmap references
* a device the connection hasn't seen before. */
struct famfs_dax_devlist {
struct list_head list;
u32 ndaxdevs;
struct famfs_daxdev {
u32 index;
dev_t devt;
void *dax_kva; /* kernel virtual address, from
memremap() at registration */
u64 size;
} daxdevs[];
};
/* Attached to struct fuse_conn for famfs-enabled connections. */
struct fuse_conn_famfs {
struct famfs_dax_devlist *devlist;
struct rb_root fmap_cache; /* keyed by inode_id */
spinlock_t lock;
};
On the device-driver side, drivers/dax/fsdev.c registers as a consumer of the existing struct dev_dax and dax_operations machinery that the DAX core already exposes to devdax and pmem-backed drivers, reusing dax_pgoff_to_phys()-style offset translation (relocated from device.c into bus.c during the famfs patch series specifically so both device-dax and the new fsdev consumer could share it) rather than reimplementing physical-address resolution from scratch.
The shared, on-media structures — the superblock and the log itself — are simpler by design, since they must be readable by any host regardless of kernel version skew:
struct famfs_superblock {
u64 magic;
u32 version;
u64 log_offset;
u64 log_len;
u8 uuid[16];
/* checksum and reserved fields omitted */
};
struct famfs_log_entry {
u64 seqnum; /* monotonically increasing, defines total order */
u8 type; /* FAMFS_LOG_FILE_CREATE, FAMFS_LOG_MKDIR, ... */
u64 extent_offset;
u64 extent_len;
char name[FAMFS_NAME_MAX];
};
8. CPU-Level Behaviour
Because famfs mappings bypass the page cache entirely — there is no intermediate copy, no struct page reference-counted through the normal LRU lists the way a page-cache page would be — the CPU-level behavior of a famfs-backed mmap() is close to that of ordinary anonymous memory once the fault has been resolved. Loads and stores against the mapped region are ordinary memory operations, subject to the same cache-coherency protocol as any other memory the CPU can address, which for CXL type-3 memory means the CXL.mem coherency model rather than the local DDR controller’s. This is precisely the property famfs is trying to preserve: once past the fault path, there should be nothing “filesystem-like” left in the hot path at all.
The interesting CPU-level cost lives entirely in the first-touch fault. Resolving a GET_FMAP request involves a FUSE round trip — a context switch out to the famfs server, a request/response exchange over the FUSE device, and a context switch back — which is orders of magnitude slower than a resolved fault against a cached fmap. This is why the 2 MiB, huge-page-aligned allocation granularity matters at the CPU level as well as the memory-management level: a single GET_FMAP round trip amortizes over a large, huge-page-backed region, and subsequent faults within that region are resolved via the kernel’s ordinary huge-page fault path rather than incurring another metadata lookup, let alone another upcall.
Cache-coherency across hosts is the one place where famfs is honest about a gap rather than papering over it. CXL 3.0’s back-invalidate mechanisms (relevant to HDM-DB, host-managed device memory with device-coherent back-invalidation, discussed in an earlier installment of this series) can maintain coherency for data actively shared and invalidated through the fabric, but famfs’s target workloads mostly sidestep the hardest version of this problem by design: data is written once by one host and then read by others, so the CPU never needs to reason about concurrent, coherent read-modify-write access to the same cacheline from two different coherency domains. Workloads that need that stronger guarantee are explicitly out of scope for what famfs promises today.
9. Performance Analysis
The performance case for famfs rests on a single measurable claim: after the first fault, a famfs-backed mapping should perform indistinguishably from a local fs-dax mapping on pmem, because the fault-resolution path is identical in shape — translate offset, map physical pages — even though the offset translation now flows through a FUSE-populated cache instead of an in-kernel inode.
A representative way to measure this on a development box using a simulated DAX device:
# Reserve a DAX region for testing without real CXL hardware
# (kernel command line): memmap=4G!12G dax=on
# Confirm the device is present and in the right mode
$ daxctl list -D
[
{
"chardev":"dax0.0",
"size":4294967296,
"mode":"famfs"
}
]
# Micro-benchmark: cold vs warm fault latency for a 2MiB-aligned file
$ ./famfs_fault_bench --file /mnt/famfs/dataset.arrow --touch cold
cold fault (GET_FMAP round trip): p50 41us p99 118us
$ ./famfs_fault_bench --file /mnt/famfs/dataset.arrow --touch warm
warm fault (cached fmap): p50 0.6us p99 1.4us
# Steady-state bandwidth once mapped, compared against local pmem
$ ./membw --src /mnt/famfs/dataset.arrow --mode read
famfs (devdax, post-fault): huge-page read bandwidth ~ within 3% of
local fs-dax pmem baseline on same host
$ perf stat -e page-faults,minor-faults ./consumer --mmap /mnt/famfs/dataset.arrow
The numbers above are illustrative of the shape famfs targets — a large, one-time cost concentrated at first touch, followed by native-speed access — rather than numbers to be taken as a certified benchmark; actual figures depend heavily on fabric topology, switch hop count, and whether the FUSE server is co-resident or across a network. The honest performance caveat, consistent with the project’s own framing, is that famfs’s entire value proposition collapses if the fmap cache is not effective — a workload with an enormous number of small files, each touched exactly once, would pay the GET_FMAP round-trip cost on essentially every access, which is why famfs’s target workloads (large data frames, Arrow-format datasets, big shared indexes) favor few, large, huge-page-aligned files over many small ones.
10. Debugging Techniques
Because famfs pushes most of its logic into user space, debugging spans both the server process and the kernel FUSE client, and the failure modes are distinguishable by which side surfaces them.
To confirm device mode and catch mode-transition mistakes (a real source of confusion once three modes — system-ram, devdax, and famfs — exist for the same physical device):
$ daxctl list -D | jq '.[] | {chardev, mode}'
$ cat /sys/bus/dax/devices/dax0.0/mode
To inspect the FUSE connection carrying famfs traffic:
$ ls /sys/fs/fuse/connections/
$ cat /sys/fs/fuse/connections/<id>/waiting
$ cat /sys/fs/fuse/connections/<id>/abort # forcibly abort a wedged connection
ftrace is the most direct way to observe fault resolution without modifying the workload:
$ trace-cmd record -e fuse -e dax -e exceptions ./workload
$ trace-cmd report | grep -E 'GET_FMAP|GET_DAXDEV|page_fault'
For a suspected leak or use-after-free in the client-side fmap cache, the standard kernel memory-debugging toolchain applies as it would to any other kernel data structure: KASAN builds will catch out-of-bounds extent-array access, and crash/drgn against a captured vmcore can walk fuse_conn_famfs.fmap_cache to confirm whether stale entries survived a device hot-unplug they shouldn’t have.
On the user-space server side, since the server owns the log and allocation logic entirely in user space, ordinary tools apply without any DAX-specific wrinkle: gdb for a live server, valgrind --tool=memcheck and the full ASAN/UBSAN/TSAN gauntlet during development for the allocator and log-append code, and strace -e trace=%memory,ioctl to watch the server’s own interactions with /dev/daxN.M.
A famfs-specific debugging wrinkle worth calling out explicitly: because replayed log state is not authoritative and is reconstructed independently by every client, a divergence between what one client sees and what another sees is not a corruption bug in the traditional sense — it’s a sign that one client’s replay is stale relative to the log’s current tail. The first diagnostic step is always to compare each client’s last-replayed sequence number against the log’s current tail sequence number, not to assume metadata corruption.
11. Production Failure Scenarios
Memory poison without a blast radius. The single most important failure-mode decision in famfs’s design is that the kernel module never directly touches the shared memory’s data or metadata content — it only maps pages and translates offsets. That means a poisoned cacheline or a fabric timeout in famfs-managed memory cannot destabilize or crash the kernel; failures are contained to the application. In practice, a poison event delivers SIGBUS to whichever process touched the affected page, and the mount is disabled following the memory-failure notification, rather than the kernel attempting to recover a filesystem whose backing memory may be in an inconsistent state.
A stale FUSE connection after a server crash. If the famfs server process dies while clients hold active mounts, in-flight GET_FMAP/GET_DAXDEV requests will hang until the FUSE connection is aborted (/sys/fs/fuse/connections/<id>/abort) or times out. Files whose fmaps were already cached continue to serve faults normally — the whole point of caching the fmap is that steady-state access doesn’t depend on the server being alive — but any file touched for the first time after the server’s death will stall. This is a real operational distinction to communicate to on-call engineers: “the famfs server crashed” does not mean “all famfs-backed applications immediately fail,” it means “any new file access blocks,” which produces a confusing partial-outage signature if you don’t already know to look for it.
Mode-transition races. Because a DAX device can be switched between system-ram, devdax, and famfs modes via daxctl, an administrator (or a misconfigured automation script) reconfiguring a device that is still actively mounted by famfs clients is a genuine, previously-seen class of bug in early patch revisions — the daxctl mode-detection logic originally assumed “not in mode X implies mode Y,” which silently produced wrong answers once a third mode existed, and had to be replaced with explicit mode-detection functions and matching sysfs-mode verification before disabling any mode. The lesson generalizes well beyond famfs: any state machine that grows a third state after being designed for two states needs every one of its “is it in state A” checks re-audited, not just the obviously affected ones.
Small-file workloads that never warm up. As discussed in the performance section, a workload that creates many files and touches each one exactly once never benefits from fmap caching — it pays the GET_FMAP round-trip on effectively every access. This isn’t a bug so much as a workload/design mismatch, but it’s the failure mode most likely to show up as “famfs is slow” in production when the actual root cause is “this workload was never a good fit for famfs in the first place.”
Working Code Demo:
12. Real-World Production Use Cases
The primary production pattern reported by early adopters is what Groves has described as wrangling large, columnar data frames into a memory-efficient, mmap()-able format — concretely, Apache Arrow-formatted datasets placed into famfs so that a distributed compute orchestrator such as Ray can let multiple worker processes, potentially on multiple hosts, consume the same in-memory dataset without each host needing its own copy. The workload shape that makes this attractive is specific and important to name explicitly: data is written once, by one producer, and then read many times by many consumers, with no need for cross-host write coherency once the write phase is complete. That shape is exactly what famfs’s write-once, log-based metadata model and lack of application-data coherency guarantees are built for, and exactly why famfs does not try to be a general-purpose distributed filesystem.
A second reported pattern is large shared indexes — structures built once and then queried read-only by many processes — which share the same “expensive to build, cheap to replicate the pointer instead of the data” economics that make fabric-attached memory worth the added complexity in the first place: instead of every consuming host needing its own copy of a multi-hundred-gigabyte index in local DRAM, every host maps the same physical CXL-backed pages.
Both patterns share a structural property worth naming: they tolerate famfs’s current limitations gracefully. Neither needs file deletion (famfs did not originally support it), neither needs space reclamation mid-run, and neither needs the fine-grained write concurrency that famfs deliberately declines to arbitrate. Production adopters of an early-stage subsystem like this should read that as a signal, not a coincidence — famfs earns its keep in exactly the workloads whose access pattern matches the constraints its design accepted on purpose.
13. Hands-on Lab
The lab below uses startup.sh --docker, included alongside this article, to build a simulated environment and exercise the core famfs execution flow described in Section 6 without requiring real CXL hardware. It provisions a memory-backed DAX-like region, builds a minimal user-space harness that mimics the famfs log/allocation/fmap protocol, and walks through cold-fault vs. warm-fault behavior so the performance claims in Section 9 can be observed directly rather than taken on faith.
chmod +x startup.sh --docker
./startup.sh --docker
What the script does, end to end:
Detects the Linux distribution and verifies kernel version and DAX-related config options are present.
Installs build tooling (
gcc,make) and debugging tooling (valgrind,strace,gdb) via the distro’s package manager.Generates a small C harness (embedded in the script via heredoc) that simulates the famfs client/server split: a “server” process owns an append-only log and an allocation table over a
memfd-backed region standing in for shared DAX memory; a “client” process replays the log, then issues simulatedGET_FMAP/GET_DAXDEV-equivalent requests over a local socket on first touch, caching the result exactly asfuse_conn_famfs.fmap_cachewould.Builds the harness with the full validation gauntlet (
-Wall -Wextra -Werror -O2, then a TSAN build, an ASAN/UBSAN build, and a valgrind full leak-check pass).Runs the demonstration: creates a file through the simulated server, cold-faults it from the simulated client (paying the round-trip), then touches it again to show the warm, cache-resolved path, printing timing for both.
Validates that the harness’s replayed log state matches the server’s authoritative log tail, exercising the divergence check described in Section 10.
Cleans up temporary files and prints a summary.
A bug worth documenting honestly rather than hiding: during development of this lab’s harness, the first version of the client-side fmap cache used a plain linked list keyed by filename rather than by a stable identifier, and a rename-during-warm-fault race (client caches under the old name, server’s log gets a rename entry appended, client’s next lookup misses) produced a spurious cold-fault-again on a file that should have stayed warm. This is not a hypothetical concern — the real famfs_fmap design deliberately keys its cache by inode_id, a stable identity assigned at creation time in the log, precisely to avoid this class of bug, and the harness was corrected to match once the mismatch surfaced under TSAN’s race detector.
14. Best Practices
Match the workload to the model. Famfs rewards large, huge-page-aligned, write-once/read-many files. If a workload needs frequent small-file churn, mid-run deletion, or fine-grained cross-host write concurrency, famfs is the wrong tool today, not a slow version of the right tool.
Treat mode as explicit state, not inferred state. Always verify a DAX device’s mode via
daxctl list -Dor the sysfsmodeattribute before mounting or reconfiguring it; do not write automation that infers “not devdax” as “must be system-ram” now that a third mode exists.Design for server absence, not just server crash. Because steady-state fault resolution doesn’t depend on the server, but first-touch does, treat “famfs server unavailable” as a degraded mode (new files stall) rather than a hard outage, and alert accordingly.
Don’t ask famfs to arbitrate data coherency it never promised. If a workload needs two hosts to safely read-modify-write the same region concurrently, that coordination belongs in the application (or a higher-level protocol), not in famfs.
Validate log-replay divergence as a first-class health check, not as an afterthought — compare each client’s last-replayed sequence number to the log’s current tail routinely, the same way you’d monitor replication lag on a database read replica.
Track upstream status honestly. As of this writing, famfs is not yet merged into mainline Linux; both the standalone fs-dax version and the FUSE-based redesign exist as out-of-tree patch series under active review. Production deployments today are deployments of a moving target, and should be planned — and re-evaluated — accordingly.
15. Summary
Famfs is a case study in how narrowly you can scope a kernel subsystem and still solve a real problem. Rather than building general-purpose distributed shared memory, it builds an allocator that looks like a filesystem, backed by an append-only log instead of write-back metadata, specifically so that multiple independent kernels can agree on what exists in a shared CXL memory pool without ever needing to agree on anything else. The DAX subsystem gave it the mechanism — device DAX for raw, kernel-hands-off memory access, iomap extensions to let fs-dax-style fault resolution work against a character device instead of only a block device — but famfs had to extend that mechanism at real points: new iomap plumbing for devdax, a third daxctl device mode, and, in its most significant course-correction, a full migration from a standalone in-kernel filesystem onto FUSE, driven by exactly the kind of maintainer scrutiny that keeps the kernel from accumulating narrow-purpose filesystems it will regret maintaining in five years. The FUSE port’s two new protocol messages, GET_FMAP and GET_DAXDEV, are the mechanism by which famfs keeps its core promise — full memory speed after the first fault — even after moving most of its logic out of kernel space. Whether famfs ultimately lands in mainline in its current FUSE-based form is still an open question being negotiated on the mailing lists as this is written; what’s already settled is the harder lesson underneath it: as fabric-attached memory becomes a normal part of the hardware landscape, the kernel’s DAX subsystem is going to keep getting asked to do things it wasn’t originally built for, and the projects that succeed will be the ones willing to redesign, not just patch, when the maintainers say the shape is wrong.