Embedded Linux and real-time Linux

Most robots larger than a vacuum cleaner run Linux somewhere. It hosts ROS 2, perception, planning, networking and user interfaces, and on many robots it also runs control loops. Standard Linux was designed for throughput and fairness, not deadlines, and an unmodified kernel can occasionally delay a high-priority thread by milliseconds. Since Linux 6.12, released in November 2024, the PREEMPT_RT patches that make the kernel fully preemptible are part of the mainline kernel for x86, 64-bit Arm and RISC-V, which has made real-time Linux much easier to adopt. This lesson covers why standard Linux has latency spikes, what PREEMPT_RT changes, how to configure a real-time thread correctly, how to measure the result, how to isolate cores, and how ROS 2 and micro-ROS fit into a real-time design.

Why standard Linux is not real-time

A general-purpose kernel aims to keep every core busy and share time fairly. Several of its mechanisms add unbounded or poorly bounded delay to a waking thread: long stretches of kernel code that cannot be preempted, interrupt handlers and softirqs (deferred interrupt work) that run ahead of all threads, spinlocks held with preemption disabled, page faults when touching memory that is not yet mapped, and power management that lets cores sleep in deep states that take time to exit. Under heavy disk, network or graphics load, a thread at the highest priority can wake hundreds of microseconds or a few milliseconds late. For a 1 kHz control loop, a single 3 ms delay means three missed cycles.

Linux offers several preemption models. The standard server and desktop configurations allow preemption only at certain points. PREEMPT_RT goes much further: it converts most spinlocks into sleeping locks with priority inheritance, runs most interrupt handlers as kernel threads with their own priorities, and makes almost all kernel code preemptible. A high-priority user thread can then preempt nearly everything, including most interrupt handling, and worst-case wake-up latency on suitable hardware falls to tens of microseconds.

Real-time scheduling policies

Linux has several scheduling classes. Ordinary threads use the default fair scheduler (SCHED_OTHER, implemented by CFS and, since Linux 6.6, by the EEVDF scheduler). Real-time threads use SCHED_FIFO or SCHED_RR, fixed-priority policies with priorities from 1 to 99 that always run ahead of ordinary threads: SCHED_FIFO runs a thread until it blocks or a higher-priority thread wakes, while SCHED_RR adds time slicing among equal priorities. SCHED_DEADLINE, available since Linux 3.14, implements earliest deadline first with a budget: a thread declares a runtime, deadline and period, and the kernel admits it only if the total fits, then guarantees and enforces the budget. These map directly onto the fixed-priority and EDF theory of lesson 5.

By default, Linux throttles real-time threads so they cannot use more than 95 % of each second (the sched_rt_runtime_us setting of 950,000 out of 1,000,000 µs), leaving ordinary threads a chance to run if a real-time thread loops forever. This safety net is useful during development, but it means a misbehaving real-time thread stalls for 50 ms every second, which looks like a mysterious periodic glitch.

A correctly configured real-time thread

Getting real-time behaviour from Linux requires several steps in the application, not just a real-time kernel.

• Lock memory: call mlockall(MCL_CURRENT | MCL_FUTURE) so that no page of the process can be paged out, then pre-fault the stack and any buffers by touching them at start-up, so that no page fault occurs in the control loop.
• Set the policy and priority: SCHED_FIFO at a priority above other application threads but normally below kernel threads that must serve it, such as the interrupt thread of its network or EtherCAT interface.
• Use absolute-time periodic waits: clock_nanosleep with CLOCK_MONOTONIC and TIMER_ABSTIME, advancing the target by the period each cycle.
• Avoid anything unbounded in the loop: no memory allocation, no blocking system calls such as file or console I/O, no locks shared with non-real-time threads unless they use priority inheritance.

#define _GNU_SOURCE
#include <pthread.h>
#include <sched.h>
#include <sys/mman.h>
#include <time.h>

static void *control_loop(void *arg)
{
    struct timespec next;
    clock_gettime(CLOCK_MONOTONIC, &next);
    for (;;) {
        next.tv_nsec += 1000000;                      /* 1 ms period      */
        if (next.tv_nsec >= 1000000000) { next.tv_nsec -= 1000000000; next.tv_sec++; }
        clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next, NULL);
        read_sensors();                               /* no malloc, no printf */
        compute_control();
        write_actuators();
    }
    return NULL;
}

int main(void)
{
    mlockall(MCL_CURRENT | MCL_FUTURE);               /* no paging         */
    prefault_stack_and_buffers();
    pthread_attr_t a;  struct sched_param p = { .sched_priority = 80 };
    pthread_attr_init(&a);
    pthread_attr_setinheritsched(&a, PTHREAD_EXPLICIT_SCHED);
    pthread_attr_setschedpolicy(&a, SCHED_FIFO);
    pthread_attr_setschedparam(&a, &p);
    cpu_set_t cpus;  CPU_ZERO(&cpus);  CPU_SET(3, &cpus);   /* isolated core */
    pthread_attr_setaffinity_np(&a, sizeof cpus, &cpus);
    pthread_t t;  pthread_create(&t, &a, control_loop, NULL);
    pthread_join(t, NULL);
}

The process needs permission to use real-time priorities and lock memory, granted by running with the CAP_SYS_NICE and CAP_IPC_LOCK capabilities or by limits in /etc/security/limits.conf; in containers these must be passed through explicitly. Forgetting PTHREAD_EXPLICIT_SCHED is a classic bug: without it, the new thread silently inherits the creator's ordinary policy and all the other settings are ignored.

Measuring latency with cyclictest

The standard measurement is cyclictest from the rt-tests suite. It runs one or more high-priority threads that sleep until absolute times and records how late each wake-up is, reporting minimum, average and maximum latency and optionally a histogram. A run is meaningful only under realistic stress: heavy network traffic, disk I/O, GPU work and memory pressure, typically generated with tools such as stress-ng, and for long enough to catch rare events, commonly several hours to a day.

import random, math
random.seed(11)
def run(kind, n=3_600_000):            # one hour at 1 kHz
    worst, over100, over1000 = 0.0, 0, 0
    for i in range(n):
        x = random.lognormvariate(math.log(6), 0.45)       # typical wake-up, us
        if kind == "standard" and random.random() < 3e-4:
            x += random.expovariate(1 / 250)               # non-preemptible kernel paths
        if kind == "rt" and random.random() < 3e-4:
            x += random.expovariate(1 / 8)                 # short residual sections
        worst = max(worst, x)
        over100 += x > 100
        over1000 += x > 1000
    return worst, over100, over1000
for kind in ("standard", "rt"):
    w, a, b = run(kind)
    print("%-8s max %7.1f us   >100 us: %5d   >1 ms: %d" % (kind, w, a, b))

standard max  1835.0 us   >100 us:   746   >1 ms: 30
rt       max    55.7 us   >100 us:     0   >1 ms: 0

The numbers in this model are illustrative, but the shape matches real measurements: on a standard kernel under load, rare wake-ups are hundreds of microseconds to milliseconds late, while on a well-tuned PREEMPT_RT system the worst case stays in the tens of microseconds. Real results depend heavily on hardware. Firmware features such as system management interrupts on x86 machines can stall every core for hundreds of microseconds regardless of the kernel, which is why real-time robot computers are chosen and validated with cyclictest before purchase, not after.

Core isolation and interrupt affinity

On a multicore processor, the most effective step after PREEMPT_RT is to dedicate cores to real-time work. The kernel's isolcpus or the newer cpuset and nohz_full boot options keep the scheduler from placing ordinary threads on chosen cores and stop the periodic scheduler tick there when only one thread is running. Interrupt affinity settings (under /proc/irq) steer device interrupts away from those cores, except the ones the real-time thread needs. The control thread is pinned to an isolated core with CPU affinity, so its caches stay warm and nothing else competes for that core. Frequency scaling and deep idle states should be disabled on real-time cores, because changing frequency or waking from deep sleep adds latency.

ROS 2 and real time

ROS 2 was designed with real-time use in mind, but a ROS 2 node is real-time only if its whole chain is: the operating system, the middleware, the executor and the user code. The default executors are not deterministic, because they service ready callbacks in an order that depends on arrival and implementation details. Practical patterns are to create a separate executor in a dedicated SCHED_FIFO thread for the control callbacks, to use callback groups so that slow callbacks cannot block fast ones, to avoid memory allocation by using preallocated messages and loaned messages, and to choose middleware QoS settings (lesson 10) that never block the publisher. The ros2_control framework, which most ROS 2 arms and mobile bases use, runs its controller manager in a real-time loop for exactly this reason, reading hardware interfaces, updating controllers and writing commands at a fixed rate, while the rest of ROS 2 communicates with it through non-blocking buffers.

For the hardest loops, many robots put the control on a microcontroller instead. micro-ROS brings ROS 2 concepts (nodes, publishers, subscribers, services) to microcontrollers running FreeRTOS, Zephyr or NuttX, connecting to the main computer through an agent over serial, USB, UDP or CAN. The microcontroller runs the 1 to 10 kHz loops with bare-metal or RTOS timing guarantees, while the Linux side handles planning and perception at lower rates.

// ROS 2 (rclcpp): run the control node in its own real-time executor thread
auto control_node = std::make_shared<JointController>();      // preallocates buffers
rclcpp::executors::SingleThreadedExecutor rt_exec;
rt_exec.add_node(control_node);

std::thread rt_thread([&] {
    sched_param p{};  p.sched_priority = 80;
    pthread_setschedparam(pthread_self(), SCHED_FIFO, &p);  // real-time policy
    rt_exec.spin();                                         // control callbacks only
});
rclcpp::executors::MultiThreadedExecutor other_exec;       // logging, parameters, UI
other_exec.add_node(diagnostics_node);
other_exec.spin();

Hypervisors and asymmetric multiprocessing

Many robot processors now combine large application cores with small real-time cores on one chip, for example Cortex-A cores running Linux beside Cortex-M or Cortex-R cores running an RTOS. In this asymmetric arrangement the RTOS core handles hard real-time loops and safety monitoring with microcontroller-style determinism, while Linux handles everything else; the cores communicate through shared memory with a framework such as OpenAMP and RPMsg. A related approach uses a real-time hypervisor to run Linux and an RTOS side by side on separate cores of the same processor, with hardware partitioning of memory and devices. Both approaches give a single board the predictability of a microcontroller and the software ecosystem of Linux.