Configuring the Linux Kernel for Low-Latency Real-Time Applications

This article is written from a production debugging perspective. The concepts build on each other in order — read through once before running any code.

---

Part I

---

You’ve set your thread to SCHED_FIFO at priority 99. It wakes up on time 99% of the time. Then, mid-demo, a 700-microsecond spike appears in your latency histogram. You didn’t change the code. The kernel did something to your thread.

This article is about what that something is, and how to systematically eliminate it.

---

The Real Problem Isn’t Priority

Most engineers stop at chrt -f 99 ./app and call it real-time. That’s not real-time — that’s wishful scheduling. On a stock kernel, a SCHED_FIFO thread at priority 99 can still be delayed by: a hardirq handler on its CPU, a softirq (ksoftirqd) running to completion, an RCU grace-period callback, a page fault in the hot path, and a scheduler tick from CONFIG_HZ_1000 firing every millisecond.

Priority determines who runs next. It says nothing about what’s already running on the CPU when your thread wakes up.

---