Testing robot software

Testing is how a team turns hope into evidence. For robot software it is harder than for most software: behavior depends on time, on noisy sensors and on the physical world, and the most important failures happen rarely. Yet the same principles apply, and a well-designed test suite lets a team change code with confidence, catch regressions within minutes and show customers and regulators that requirements are met. This lesson covers the levels of testing, the test pyramid, techniques for testing code that talks to hardware and time, ROS 2 testing tools, coverage and the problem of flaky tests. Simulation-based testing is developed further in lessons 5, 6 and 8.

Levels of testing

Unit tests check a small piece of code, a function or class, in isolation, quickly and deterministically: does the inverse kinematics function return the right joint angles for a known pose, does the velocity limiter clamp correctly at the boundary? Integration tests check that several components work together: does the localization node, given a recorded scan and odometry, publish poses in the right frame at the right rate? System tests exercise the whole robot or its full software stack, usually in simulation or on a test track, against system requirements: does the robot navigate a 200 m route without collisions? Acceptance tests are system tests agreed with the customer, run against the acceptance criteria in the contract.

Each level catches different defects. Unit tests catch logic errors where they are cheapest to diagnose. Integration tests catch interface mismatches: units, frames, timing, QoS incompatibilities, message ordering. System tests catch emergent problems that only appear when everything runs together: CPU overload, latency chains, interactions between behaviors. Field tests catch what the simulation and the test track did not model.

The test pyramid

The test pyramid is a guideline for proportions: many fast, cheap unit tests at the base, fewer integration tests in the middle and a small number of slow, expensive system and field tests at the top. The reasoning is economic. A unit test runs in milliseconds and points directly at the broken function; a full simulation test may take minutes, may fail for many reasons and needs investigation to locate the fault. A suite shaped like an inverted pyramid, mostly slow end-to-end tests, is slow to run, flaky and hard to diagnose.

Unit testing robot code

Robot code is easiest to unit test when algorithms are separated from middleware. Put the maths of a controller, a planner or a filter in plain functions or classes that take inputs and return outputs, and keep the ROS 2 node as a thin wrapper that converts messages and calls them. The core can then be tested with ordinary frameworks, GoogleTest for C++ and pytest for Python, both of which integrate with colcon test and the ament build tools.

Good unit tests are fast, deterministic, independent of each other and focused on one behavior each. They test boundaries and failure cases, not just typical inputs: zero, negative and maximum values, empty point clouds, NaN readings, angles near ±π. Floating-point results are compared with tolerances. Each test name states the behavior checked, for example test_velocity_limiter_clamps_at_max_speed, so a failure report is self-explanatory.

import math, random
def wrap(a):                      # wrap an angle to (-pi, pi]
    a = math.fmod(a + math.pi, 2 * math.pi)
    if a <= 0: a += 2 * math.pi
    return a - math.pi
def limit(v, vmax):               # clamp a velocity command
    return max(-vmax, min(vmax, v))
# example-based tests
assert abs(wrap(3 * math.pi) - math.pi) < 1e-12
assert abs(wrap(-math.pi / 2) + math.pi / 2) < 1e-12
assert limit(2.0, 1.2) == 1.2 and limit(-5, 1.2) == -1.2
# property-based tests: check invariants on many random inputs (seeded)
rnd = random.Random(7); n = 0
for _ in range(10000):
    a = rnd.uniform(-1e4, 1e4)
    w = wrap(a)
    assert -math.pi < w <= math.pi                         # range invariant
    assert abs(math.sin(w) - math.sin(a)) < 1e-6           # same direction
    v, vmax = rnd.uniform(-10, 10), rnd.uniform(0.1, 3)
    assert abs(limit(v, vmax)) <= vmax                     # never exceeds the limit
    n += 1
print(f"{n} random cases passed all invariants")

10000 random cases passed all invariants

The second half of that listing is property-based testing: instead of choosing examples by hand, state invariants that must hold for every input and check them against many generated inputs. Libraries such as Hypothesis for Python and RapidCheck for C++ automate the generation and, when a property fails, shrink the input to a minimal counterexample. Property tests are excellent for geometry, transforms, filters and planners, where invariants such as 'a wrapped angle stays in range' or 'a planned path never enters an obstacle' are easy to state.

Mocks, fakes and hardware interfaces

Code that talks to hardware, the network or the clock is hard to test directly. The solution is to depend on interfaces and substitute test doubles. A fake is a working but simplified implementation, such as a simulated motor that integrates commanded velocity into position. A mock is a test double that records how it was called, so the test can check, for example, that the driver sent a stop command when the watchdog expired. A stub returns canned answers. With ros2_control, a mock hardware component lets the controller manager and controllers be tested without any real motors.

Time deserves special care. Code that calls the system clock directly cannot be tested deterministically. ROS 2 nodes use a clock abstraction that can follow simulated time (the use_sim_time parameter), so a test can advance time explicitly and check behavior at each step, such as a timeout firing after exactly 0.5 s of simulated time regardless of how fast the test machine is.

Integration testing in ROS 2

ROS 2 provides launch_testing, which starts a set of nodes from a launch description and then runs test cases against the live graph: checking that a node publishes on a topic within a time limit, that its outputs match expected values, that processes exit cleanly. Recorded data replayed from rosbag2 files is a powerful integration input: a bag from a real robot run, replayed into the localization or perception stack, tests the code against real sensor data repeatably. Teams build up a library of 'golden' bags covering difficult cases (glare, crowds, featureless corridors) and run them on every change, comparing outputs against stored reference results with tolerances.

# test_localization_launch.py  (listing; launch_testing)
import unittest, launch, launch_ros.actions, launch_testing.actions, pytest
@pytest.mark.launch_test
def generate_test_description():
    loc = launch_ros.actions.Node(package="my_loc", executable="localizer",
                                  parameters=[{"use_sim_time": True}])
    bag = launch.actions.ExecuteProcess(cmd=["ros2", "bag", "play", "golden/aisle_glare", "--clock"])
    return launch.LaunchDescription([loc, bag, launch_testing.actions.ReadyToTest()])
class TestPoseOutput(unittest.TestCase):
    def test_pose_published_within_5s(self, proc_output):
        # in a real test: subscribe to /pose and assert a message arrives in 5 s
        pass

Coverage

Code coverage measures which parts of the code the tests execute: statement (line) coverage, branch coverage (whether each decision has gone both ways) and, for safety-critical code, modified condition/decision coverage (MC/DC), which requires showing that each condition in a decision independently affects the outcome. Tools include gcov and lcov for C++ and coverage.py for Python, and colcon can aggregate results. Coverage reveals untested code, which is valuable; it does not show that the tested code is correct, because a test can execute a line without checking its result.

# Branch coverage by hand: which of the 2*3 = 6 branch outcomes do the tests exercise?
def speed_cmd(dist_m, person_seen, battery_low):
    v = 1.2
    if person_seen:            # decision 1
        v = 0.3
    if dist_m < 0.5:           # decision 2
        v = 0.0
    if battery_low:            # decision 3
        v = min(v, 0.6)
    return v
tests = [(5.0, False, False), (5.0, True, False), (0.2, False, True)]
hit = set()
for d, p, b in tests:
    hit |= {("person", p), ("close", d < 0.5), ("battery", b)}
    print(f"speed_cmd{(d, p, b)} = {speed_cmd(d, p, b)}")
total = {(k, v) for k in ("person", "close", "battery") for v in (True, False)}
print(f"branch outcomes covered: {len(hit)}/{len(total)}; missing: {sorted(total - hit)}")

speed_cmd(5.0, False, False) = 1.2
speed_cmd(5.0, True, False) = 0.3
speed_cmd(0.2, False, True) = 0.0
branch outcomes covered: 6/6; missing: []

Here three tests happen to cover all six branch outcomes, yet they never check a person seen at close range with a low battery, the combination that matters most for safety. Coverage tools would report 100 % branch coverage. This is why coverage is a floor, not a goal: requirements, scenarios and risk analysis decide which combinations must be tested.

Flaky tests

A flaky test sometimes passes and sometimes fails without any change to the code. Robot test suites are prone to flakiness because of timing (a message arrives a few milliseconds late on a busy CI machine), concurrency, random seeds, network ports shared between tests and simulation nondeterminism. Flakiness is corrosive: when failures are often false alarms, engineers stop trusting the suite and real regressions slip through.

# If each of N independent tests is flaky with probability p, how often does a clean
# commit fail CI?  P(at least one failure) = 1 - (1 - p)^N
for N, p in ((100, 0.001), (500, 0.001), (500, 0.005), (2000, 0.001)):
    print(f"N={N:5d}, p={p:.3f}: spurious CI failure rate = {1 - (1 - p) ** N:.1%}")
# Retrying a failed test once reduces each test's effective flake rate to p^2:
p = 0.005
print(f"with one retry, N=500, p={p}: {1 - (1 - p**2) ** 500:.2%}")

N=  100, p=0.001: spurious CI failure rate = 9.5%
N=  500, p=0.001: spurious CI failure rate = 39.4%
N=  500, p=0.005: spurious CI failure rate = 91.8%
N= 2000, p=0.001: spurious CI failure rate = 86.5%
with one retry, N=500, p=0.005: 1.24%

Retries hide flakiness rather than fix it, and can also hide real intermittent bugs, which in robots are often race conditions that will eventually appear in the field. Better practices: use simulated time instead of wall-clock sleeps, wait for conditions rather than fixed delays, seed all random generators, isolate tests with separate ROS domain IDs, quarantine a flaky test with an issue to fix it, and track the flake rate of each test over time.