The Kernel Watches Itself: Inside Soft Lockup, Hard Lockup, and Hung Task Detection
A userspace watchdog answers a simple question: Is my application still running? It cannot answer a far more important one: Is the kernel itself still making forward progress? Once the scheduler stalls or a CPU stops servicing interrupts, userspace loses the ability to observe the system. Linux solves this by implementing multiple watchdog mechanisms inside the kernel, each designed to detect a different class of failure before a partially responsive system turns into a complete outage.
The Naïve Approach
A common assumption is that periodically writing to /dev/watchdog guarantees system health. It doesn’t.
Imagine a storage driver spinning forever while holding a spinlock with interrupts disabled. The watchdog daemon continues executing on another CPU, successfully resetting the hardware watchdog, while every task waiting for that lock blocks indefinitely.
The observable symptoms are deceptive:
SSH connections stop responding.
Processes accumulate in uninterruptible sleep (
Dstate).One CPU remains pinned at nearly 100% system time.
System load rises even though overall CPU utilization appears low.
The machine is not completely dead—it has simply lost forward progress on part of the kernel. Userspace monitoring cannot distinguish this from a healthy system.
