Reliability, testing and functional safety

The previous lessons showed how to design embedded software that meets its deadlines when everything works. This lesson is about what happens when something does not: a sensor fails, a task overruns, memory is corrupted, a cosmic ray flips a bit, or a bug that survived every test appears in the field. Robots work near people and expensive equipment, so their embedded software must detect faults and reach a safe state reliably, and safety-related functions must be developed to recognized standards. The lesson covers watchdogs and fault handling, testing strategies from unit tests to hardware-in-the-loop, static analysis, an awareness-level tour of the functional safety standards that apply to robots, and the security measures that now sit alongside safety.

Faults, errors and failures

Reliability engineering separates three ideas. A fault is a defect or abnormal condition: a software bug, a broken wire, a bit flip. An error is the incorrect internal state a fault produces: a wrong joint target in memory. A failure is when the system's delivered behavior deviates from what was required: the arm moves where it should not. Good real-time design breaks the chain at every link: prevent faults through careful development, detect errors before they become failures, and contain failures so that the system degrades or stops safely rather than behaving dangerously. Faults can be systematic, built in by design or coding mistakes and present in every unit, or random, arising from hardware wear-out and environmental effects. Testing and process address systematic faults; redundancy and diagnostics address random ones.

Watchdogs

A watchdog timer is a hardware counter that resets the processor, or triggers a safe-state output, unless software refreshes it before it expires. It is the last line of defense against software that has hung or run away. A watchdog is only useful if it is refreshed in the right place: refreshing it from a timer interrupt proves only that the interrupt still runs, not that the control task does. The better pattern is a supervisor that checks that every critical task has reported progress within its deadline and only then refreshes the watchdog. A window watchdog adds a lower limit too: refreshing too early is also treated as a fault, which catches software stuck in a tight loop that keeps refreshing.

/* Supervisor: refresh the window watchdog only if all critical tasks checked in */
static volatile uint32_t heartbeat;            /* each task sets its own bit  */
#define HB_JOINT   (1u << 0)
#define HB_SAFETY  (1u << 1)
#define HB_COMMS   (1u << 2)
#define HB_ALL     (HB_JOINT | HB_SAFETY | HB_COMMS)

void supervisor_task(void *arg)                /* runs every 10 ms             */
{
    TickType_t last = xTaskGetTickCount();
    for (;;) {
        xTaskDelayUntil(&last, pdMS_TO_TICKS(10));
        uint32_t seen = __atomic_exchange_n(&heartbeat, 0, __ATOMIC_SEQ_CST);
        if (seen == HB_ALL) {
            WWDG->CR = WWDG_RELOAD;            /* refresh inside the window    */
        } else {
            enter_safe_state(seen ^ HB_ALL);   /* report which task is missing */
        }
    }
}

Fault handling and safe states

Every robot needs a defined safe state for each failure, and the right one depends on the machine. For many arms it is removing motor power and applying brakes (a category 0 stop) or decelerating under control and then removing power (category 1); for some, controlled stopping with power maintained (category 2). For a drone, cutting power is the worst option in flight. Fault handlers should be designed with the same care as normal paths: a hard fault handler on a Cortex-M should log the faulting address and registers to a reserved memory area, put outputs into their safe condition and then reset, so that the cause can be diagnosed afterwards. Common detection mechanisms include range and plausibility checks on sensor values, comparing redundant sensors, deadline and overrun monitoring (lesson 4), stack and memory protection faults (lesson 8), CRCs on messages and stored data, and periodic self-tests of RAM, flash and the processor.

Testing real-time software

Testing real-time software combines ordinary software testing with tests of timing and fault behavior.

• Unit tests check functions on a host computer, with hardware access replaced by test doubles. Control algorithms, filters and state machines can be tested thoroughly this way, with coverage measured.
• Integration tests on the target exercise tasks together, with the real RTOS and drivers.
• Hardware-in-the-loop (HIL) testing connects the real controller to a real-time simulation of the robot and its environment, so that the firmware experiences realistic sensor signals and its outputs drive a simulated plant. HIL makes it possible to test dangerous faults, such as a broken encoder at full speed, safely and repeatably.
• Timing tests run long soaks under maximum load while recording maximum response times, overruns and stack high-water marks (lessons 2 and 8).
• Fault-injection tests deliberately cause the faults the design claims to handle: disconnect sensors, corrupt messages, stall tasks, fill queues, and check that each is detected and handled within its required time.

# Fault-injection test of a joint supervisor state machine (host-side)
class Supervisor:
    def __init__(self):
        self.state, self.missed = "RUN", 0
    def step(self, encoder_ok, cmd_age_ms, overrun):
        if self.state == "SAFE":
            return self.state
        if not encoder_ok:
            self.state = "SAFE"                      # immediate stop
        elif cmd_age_ms > 20:
            self.state = "STOPPING"                  # decelerate on stale command
        elif overrun:
            self.missed += 1
            if self.missed >= 3:
                self.state = "SAFE"                  # repeated overruns
        else:
            self.missed = 0
            if self.state == "STOPPING":
                self.state = "RUN"
        return self.state

tests = {
    "encoder fails":       [(True, 1, False)] * 3 + [(False, 1, False)],
    "commands stop":       [(True, a, False) for a in (1, 10, 25, 30)],
    "one overrun":         [(True, 1, x) for x in (False, True, False, False)],
    "three overruns":      [(True, 1, True)] * 3,
    "stale then resumes":  [(True, 30, False), (True, 2, False)],
}
expected = {"encoder fails": "SAFE", "commands stop": "STOPPING", "one overrun": "RUN",
            "three overruns": "SAFE", "stale then resumes": "RUN"}
fails = 0
for name, seq in tests.items():
    s = Supervisor()
    for inputs in seq:
        final = s.step(*inputs)
    ok = final == expected[name]
    fails += not ok
    print("%-20s -> %-8s %s" % (name, final, "pass" if ok else "FAIL"))
print("failures:", fails)

encoder fails        -> SAFE     pass
commands stop        -> STOPPING pass
one overrun          -> RUN      pass
three overruns       -> SAFE     pass
stale then resumes   -> RUN      pass
failures: 0

Tests like these are cheap to run on every commit in continuous integration. The same scenarios are then repeated on HIL rigs with real timing, because a state machine that reacts correctly on a host may still react too slowly on the target.

Static analysis and code review

Static analysis tools examine source code without running it. Coding-standard checkers enforce MISRA C or similar rules (lesson 8); bug finders detect null dereferences, buffer overruns, uninitialized variables and data races; and sound analyzers based on abstract interpretation can prove the absence of whole classes of run-time errors, such as division by zero or array out-of-bounds, for every possible input. Worst-case stack and execution-time analyzers belong to the same family. Combined with peer code review and compiler warnings set to fail the build, static analysis catches many faults that testing would find late or never.

Functional safety standards

Functional safety means freedom from unacceptable risk due to a system failing to perform its safety functions correctly. It is governed by standards that set requirements for the whole life cycle, from hazard analysis to decommissioning, with more rigor required for higher risk. This course gives an awareness-level map only; safety-related robot software must be developed by people trained in the applicable standard, and certification is done with accredited assessment bodies.

• IEC 61508 is the generic standard for electrical, electronic and programmable safety systems. It defines four safety integrity levels, SIL 1 to SIL 4, each with targets for the probability of dangerous failure and requirements for development techniques, and part 3 covers software.
• ISO 13849-1, revised in 2023, covers safety-related parts of machine control systems. It rates safety functions by performance level, PL a to PL e, based on architecture category, component reliability, diagnostic coverage and common-cause failure measures. IEC 62061 is the machinery alternative based on SIL.
• ISO 10218-1 and -2, revised in 2025, are the safety standards for industrial robots and robot systems; the 2025 edition absorbed the collaborative-robot guidance of ISO/TS 15066 and is generally read as requiring PL d with category 3 architecture for key safety functions unless risk assessment justifies otherwise. ISO 13482 covers personal care robots and ISO 3691-4 driverless industrial trucks and mobile robots.
• ISO 26262 adapts IEC 61508 to road vehicles with automotive safety integrity levels, ASIL A to ASIL D, and is relevant to autonomous vehicles and many robot components derived from automotive parts.
• National adoption differs. In Canada, CSA Z434 adopts ISO 10218 with Canadian deviations, and provincial occupational health and safety rules and the Canadian Electrical Code apply to installations. In the United States, ANSI/A3 R15.06 (a 2025 edition adopting ISO 10218) is the national robot standard, while OSHA's general duty clause and machine guarding rules and the National Electrical Code apply to installations.

In software terms, higher integrity levels demand more: formal requirements and traceability, restricted language subsets, defensive programming, full structural test coverage, independent review, qualified tools, and freedom from interference, meaning evidence that less critical software cannot disturb safety functions through memory, timing or communication. The memory protection of lesson 8, the timing analysis of lesson 5 and the supervisors of this lesson are the technical means of providing that evidence. A common architecture keeps safety functions on a separate, certified safety controller or safety-rated drive functions such as safe torque off, so that the large non-safety software stack does not need certification at all.

Security and safety together

A robot connected to a network is open to attack, and an attacker who can change firmware or commands can defeat every safety measure that depends on software. Security is now treated alongside safety. IEC 62443 is the industrial automation security series, defining security levels and requirements for components and systems. Practical measures for embedded robot controllers include secure boot, which verifies a cryptographic signature on firmware before running it; signed and encrypted firmware updates with rollback protection; disabling or locking debug ports in production; authenticated and integrity-checked messages on networks that cross trust boundaries; and keeping safety functions on hardware that does not accept remote software changes at all. Regulation is tightening: the European Union's Cyber Resilience Act and Machinery Regulation, both phasing in during 2026 and 2027, add cybersecurity obligations for products with digital elements and for machinery sold in Europe, which affects Canadian and US robot makers who export.