Reward design and safety in reinforcement learning

A reinforcement learning agent optimizes exactly what it is rewarded for, not what its designers meant. That simple fact explains a large share of failed robot learning projects. A reward that seems obviously right can lead to behaviour that technically scores well while missing the point entirely, and an agent that explores freely can damage itself, its surroundings or people. This lesson covers how to design rewards that produce the intended behaviour, how to recognize and prevent reward hacking, and how to keep learning robots safe during both training and deployment.

What a reward has to do

A reward function turns the designer's intentions into a number at every step. It has to do three jobs at once. It must define success correctly, so that the best-scoring behaviour is the behaviour you want. It should provide enough signal for learning to make progress, rather than leaving the agent wandering with no feedback. And it should discourage side effects that the designer cares about but did not think to mention, such as excessive force, jerky motion, energy waste or knocking other objects over.

Those goals pull against each other. A sparse reward of +1 for task success and 0 otherwise defines success precisely but gives almost no learning signal, as lesson 4 showed for random exploration. A dense reward with many terms gives plenty of signal but creates many opportunities for unintended trade-offs. Good reward design is the craft of finding the simplest reward that is both correct and learnable.

Reward shaping

Shaping means adding intermediate rewards to guide learning. The danger is that the shaping terms change which behaviour is optimal. A classic illustration comes from simulated soccer: rewarding a robot for touching the ball produced an agent that vibrated against the ball to collect touch rewards rather than scoring. In manipulation, rewarding proximity to an object can produce an agent that hovers just above the object, because closing the gripper risks pushing the object away and losing proximity reward.

Potential-based shaping avoids this. Choose a potential function \( \Phi(s) \) that estimates how close a state is to success, such as the negative distance from gripper to object, and add the shaping term \( F(s, s') = \gamma \Phi(s') - \Phi(s) \) to the original reward. Over any trajectory, these terms telescope: their discounted sum depends only on the start and end states, not on the path. As a result, potential-based shaping provably leaves the optimal policy unchanged while giving the agent a helpful gradient towards success.

# Potential-based shaping keeps the optimal policy; a naive bonus does not
import numpy as np
# Chain of 6 states, start at 0, goal at 5 (+10). Actions: 0 = left, 1 = right.
# State 2 has a "bonus" for naive shaping. Step cost -0.1.
N, gamma = 6, 0.95
def solve(shape):
    V = np.zeros(N)
    for _ in range(500):
        Vn = V.copy()
        for s in range(N - 1):
            qs = []
            for a in (0, 1):
                s2 = max(0, s - 1) if a == 0 else s + 1
                r = -0.1 + (10.0 if s2 == N - 1 else 0.0)
                if shape == 'naive' and s2 == 2:
                    r += 2.0                                     # bonus every visit to state 2
                if shape == 'potential':
                    phi = lambda x: 2.0 * x                      # closer to goal = higher potential
                    r += gamma * (0 if s2 == N - 1 else phi(s2)) - phi(s)
                qs.append(r + gamma * (0 if s2 == N - 1 else V[s2]))
            Vn[s] = max(qs)
        V = Vn
    pol = []
    for s in range(N - 1):
        qs = []
        for a in (0, 1):
            s2 = max(0, s - 1) if a == 0 else s + 1
            r = -0.1 + (10.0 if s2 == N - 1 else 0.0)
            if shape == 'naive' and s2 == 2: r += 2.0
            if shape == 'potential':
                r += gamma * (0 if s2 == N - 1 else 2.0 * s2) - 2.0 * s
            qs.append(r + gamma * (0 if s2 == N - 1 else V[s2]))
        pol.append('R' if qs[1] >= qs[0] else 'L')
    return ''.join(pol)
for shape in ('none', 'potential', 'naive'):
    print(f"{shape:9s} optimal actions in states 0-4: {solve(shape)}")

none      optimal actions in states 0-4: RRRRR
potential optimal actions in states 0-4: RRRRR
naive     optimal actions in states 0-4: RRRLL

Reward hacking and specification gaming

Reward hacking, also called specification gaming, happens when an agent finds a way to score highly that the designer did not intend. In simulation it is common and sometimes spectacular: agents exploit physics-engine bugs to move faster than intended, flip objects so that a sensor reading looks like success, or learn that ending an episode early avoids penalties. On real robots, hacking tends to be subtler: a policy that lifts the object briefly to trigger the success detector and then drops it, or one that pushes the object into the camera's blind spot where a detector reports it as removed.

Several habits catch hacking early. Watch videos of the learned behaviour regularly, not just the reward curve, because a rising return can hide degenerate behaviour. Make success detectors robust, for example by requiring an object to stay lifted for a few seconds and checking it with two independent signals. Add terms for side effects you care about, such as penalties on contact forces above a threshold or on large changes in action. Keep the simulator honest by randomizing physics, as lesson 7 describes, so that exploits that depend on exact parameters stop working. And evaluate in conditions that differ from training, since a policy that relies on a quirk will usually fail when the quirk is absent.

Combining multiple objectives

Most real tasks have several objectives: complete the task, do it quickly, use little energy, keep forces low, move smoothly and avoid collisions. A weighted sum of terms is the usual approach, \( r = w_1 r_{\text{task}} - w_2 \|\tau\|^2 - w_3 \|a_t - a_{t-1}\|^2 - w_4 \cdot \text{collision} \), but the weights are hard to set. Too much weight on smoothness and the policy may freeze; too little and it may jitter in ways that wear out gearboxes. A practical procedure is to start with only the task term until the policy can sometimes succeed, then add penalty terms one at a time, checking behaviour after each. Normalizing each term by a typical value makes weights more interpretable.

When some requirements are hard limits rather than preferences, such as maximum contact force or minimum distance to a person, it is often better to express them as constraints rather than as penalties. Constrained RL methods optimize the task reward subject to the expected cost of constraint violations staying below a threshold. A common approach adds a Lagrange multiplier that grows automatically while the constraint is being violated and shrinks when it is satisfied, so the agent is pushed harder towards compliance only when needed.

Safe exploration on real robots

During training, the agent must try actions whose consequences it does not yet know. On a real robot, the cost of a bad try can be a broken gripper, a damaged part or a person in harm's way. Safe exploration combines several layers, and no single one is enough.

Hardware and environment measures come first: compliant or low-inertia arms, torque and speed limits set in the robot controller, padded or enclosed training cells, objects that are cheap and unbreakable, and staff trained to use the emergency stop. Safety-rated functions, such as speed and separation monitoring or power and force limiting for collaborative robots described in ISO 10218 and ISO/TS 15066, sit outside the learning system entirely and cannot be overridden by it.

Algorithmic measures restrict what the learner can command. Action limits and rate limits stop large jumps. A safety filter checks each proposed action against a model of the robot and its surroundings and replaces it with the nearest safe action when needed; control barrier functions are one formal version of this idea. Residual learning around a safe base controller, covered in lesson 1, keeps behaviour close to something known to be safe. Starting from a demonstration-trained policy concentrates exploration near reasonable behaviour.

Procedural measures complete the picture: train in simulation first and transfer, pilot each new policy at reduced speed, increase autonomy gradually, log every episode, and stop training automatically when monitored signals such as contact forces or joint temperatures exceed limits.

# A simple safety filter: clip a proposed velocity so the arm can stop before a wall
import numpy as np
x_wall = 0.60          # wall position along the axis (m)
a_max = 2.0            # maximum braking deceleration (m/s^2)
margin = 0.02          # extra clearance (m)
def safe_velocity(x, v_proposed):
    gap = x_wall - margin - x
    if gap <= 0:
        return min(v_proposed, 0.0)                     # only allow moving away
    v_limit = np.sqrt(2 * a_max * gap)                  # stopping distance v^2 / (2a) <= gap
    return min(v_proposed, v_limit)
for x, v in ((0.30, 0.8), (0.50, 0.8), (0.55, 0.8), (0.58, 0.8), (0.59, -0.2)):
    print(f"x={x:.2f} m, proposed {v:+.2f} m/s -> allowed {safe_velocity(x, v):+.3f} m/s")

x=0.30 m, proposed +0.80 m/s -> allowed +0.800 m/s
x=0.50 m, proposed +0.80 m/s -> allowed +0.566 m/s
x=0.55 m, proposed +0.80 m/s -> allowed +0.346 m/s
x=0.58 m, proposed +0.80 m/s -> allowed +0.000 m/s
x=0.59 m, proposed -0.20 m/s -> allowed -0.200 m/s

Safety after deployment

A policy that behaved well in evaluation can still meet situations it never saw. Deployment safety relies on monitoring and fallbacks. Out-of-distribution detection flags observations unlike those in training, for example by measuring the disagreement of an ensemble of models or the reconstruction error of an autoencoder, and hands control to a conservative fallback or a person. Runtime monitors check physical quantities, such as forces, speeds and distances, against limits. Every intervention and near miss should be logged and reviewed, both to fix the policy and to update the risk assessment required by occupational health and safety rules in both Canada and the United States, which treat a learning robot like any other machine: the employer must identify hazards and control them, regardless of how the robot's behaviour was produced.