Human factors: workload, situation awareness and Fitts' law

Every robot that works with people is part of a system whose weakest link is often human attention, not robot hardware. Human factors, also called ergonomics, is the discipline that studies human capabilities and limits and applies them to the design of tools, tasks and environments. This lesson introduces the human factors ideas used most often in HRI: how people perceive and attend, how mental workload is defined and measured, what situation awareness is and how it fails, how quickly people can point and choose, and how errors arise. Each idea comes with a formula or a measurement method that you will use again in the user-study lessons later in the course.

Perception, attention and multiple resources

People take in information through several channels at once, but attention is limited. Christopher Wickens' multiple resource theory proposes that tasks interfere with each other more when they draw on the same resources: the same sensory modality (visual or auditory), the same processing code (spatial or verbal), or the same response type (manual or vocal). Driving while listening to a podcast is easier than driving while reading, because listening uses an auditory and verbal channel while driving is visual and spatial. For HRI this means that an operator watching a camera feed and steering a robot with a joystick will cope better with spoken alerts than with text messages popping up on the same screen.

Attention also has a spotlight character. People notice what they expect and what they are looking for, and can miss surprising events in plain view, a phenomenon called inattentional blindness. Change blindness is the related failure to notice that something has changed, especially if the change happens during a glance away or a screen refresh. In a control room supervising a fleet of robots, a single robot's status icon turning from green to amber may go unseen for minutes. Good interface design uses motion, sound and position to direct attention to what matters, and avoids crying wolf with frequent unimportant alerts that train operators to ignore the channel.

Mental workload

Mental workload is the portion of a person's limited capacity that a task demands. Too little workload leads to boredom and loss of vigilance; too much leads to errors, slowed responses and stress. Performance is often best at a moderate level, an idea popularly linked to the Yerkes-Dodson law, although the original 1908 study concerned arousal in mice and the inverted-U picture is a simplification. In HRI, workload depends on the number of robots supervised, the frequency of interventions, the quality of the interface, time pressure and the consequences of error.

Workload is measured in three main ways. Performance measures look at how well the person does the main task, or a secondary task added to probe spare capacity, such as responding to a tone. Physiological measures include heart rate variability, pupil diameter, skin conductance and brain activity; they are continuous but noisy and affected by many factors besides workload. Subjective measures ask people to rate their workload, and are cheap, sensitive and widely used. The most common subjective measure is the NASA Task Load Index, or NASA-TLX, developed by Sandra Hart and Lowell Staveland at NASA Ames and published in 1988.

In the original procedure, each person also completes 15 pairwise comparisons, choosing which of each pair of subscales contributed more to workload in this task. The number of times a subscale is chosen, from 0 to 5, becomes its weight, and the weights sum to 15. The weighted workload score is

Many studies skip the comparisons and report the raw TLX, the simple mean of the six ratings. Raw and weighted scores are highly correlated in most data sets, and the raw version saves time, but a study should say which it used and report the subscales as well, since two tasks with the same total can differ in where the load comes from.

# NASA-TLX: raw and weighted scores for a small group, two interface conditions
import numpy as np
rng = np.random.default_rng(430)
scales = ["mental", "physical", "temporal", "performance", "effort", "frustration"]
base = np.array([60, 20, 50, 35, 60, 45])
def ratings(shift):
    r = base + shift + rng.normal(0, 10, size=(8, 6))
    return np.clip(np.round(r / 5) * 5, 0, 100)          # 0 to 100 in steps of 5
joystick = ratings(np.array([10, 5, 10, 5, 10, 10]))
assisted = ratings(np.array([-10, 0, -5, -5, -10, -10]))
w = np.array([5, 0, 3, 2, 4, 1])                         # one weight set, sums to 15
for name, r in (("joystick", joystick), ("assisted", assisted)):
    raw = r.mean(axis=1); weighted = r @ w / 15
    print(f"{name:9s} raw TLX mean {raw.mean():5.1f}  weighted TLX mean {weighted.mean():5.1f}")
print("subscale means (joystick):", {k: float(v) for k, v in zip(scales, joystick.mean(axis=0).round(1))})

joystick  raw TLX mean  55.6  weighted TLX mean  65.6
assisted  raw TLX mean  36.6  weighted TLX mean  43.1
subscale means (joystick): {'mental': 71.2, 'physical': 25.0, 'temporal': 60.6, 'performance': 44.4, 'effort': 75.0, 'frustration': 57.5}

Situation awareness

Mica Endsley defined situation awareness, SA, as the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future. Level 1 SA is perceiving: noticing that a robot has stopped. Level 2 is comprehending: understanding that it stopped because its path is blocked by a pallet. Level 3 is projecting: realizing that if it stays stopped, three other robots queued behind it will miss their pick deadline. Errors at level 1 are the most common in studies of aviation incidents, which is why interfaces must make critical information visible, not merely available somewhere in a menu.

Situation awareness can be measured with the Situation Awareness Global Assessment Technique, SAGAT, in which a simulation is frozen at random moments, displays are blanked, and the participant answers questions about the current state. Answers are compared with the true state. SAGAT is direct but requires a simulator and interrupts the task. Alternatives include probe questions asked without freezing (the situation present assessment method, SPAM) and subjective ratings such as the Situation Awareness Rating Technique, SART. In teleoperation, poor SA appears as robots driving into obstacles that were off-camera, operators losing track of which way the robot is facing, or operators not noticing that a robot has tipped.

Pointing: Fitts' law

Many robot interfaces involve pointing: tapping a target on a tablet map, moving a cursor to select an object for a robot to pick, or guiding an end effector to a part by hand. Paul Fitts showed in 1954 that the time to move to a target grows with the ratio of distance to target width. The form most used today, proposed by Scott MacKenzie, is

The logarithm is called the index of difficulty, ID, measured in bits. The constants \( a \) and \( b \) are found by regression from experimental data and depend on the device and the person. Throughput, ID divided by MT, measured in bits per second, summarizes the performance of a pointing device. Fitts' law tells designers that small, distant buttons are slow; that frequently used controls should be large and close; and that screen edges and corners, which act as targets of effectively infinite width for a mouse, are fast to hit. For direct physical guidance of a robot arm, studies have found that Fitts-like relationships hold for the human-robot pair, with constants that depend on the robot's admittance or impedance settings.

# Fitting Fitts' law to synthetic pointing data
import numpy as np
rng = np.random.default_rng(430)
D = np.repeat([64, 128, 256, 512], 3 * 10).astype(float)          # mm
W = np.tile(np.repeat([8, 16, 32], 10), 4).astype(float)            # mm
ID = np.log2(D / W + 1)
MT = 0.15 + 0.12 * ID + rng.normal(0, 0.05, ID.size)                 # s
b, a = np.polyfit(ID, MT, 1)
pred = a + b * ID
r2 = 1 - np.sum((MT - pred) ** 2) / np.sum((MT - MT.mean()) ** 2)
tp = np.mean(ID / MT)
print(f"a = {a:.3f} s, b = {b:.3f} s/bit, R^2 = {r2:.3f}")
print(f"ID range {ID.min():.2f} to {ID.max():.2f} bits; mean throughput {tp:.2f} bits/s")

a = 0.170 s, b = 0.115 s/bit, R^2 = 0.866
ID range 1.58 to 6.02 bits; mean throughput 6.10 bits/s

Choosing: the Hick-Hyman law

When a person must choose among several equally likely options, decision time grows with the logarithm of the number of options: \( RT = a + b \log_2(n + 1) \), the Hick-Hyman law. Doubling the number of commands on a robot's menu adds a roughly constant amount of time rather than doubling it, but the law holds only for familiar, well-practised choices. For HRI interfaces, the practical lessons are to group commands, put the most frequent ones first and avoid making operators scan long unfamiliar lists during time-critical moments.

Human error

James Reason classified unsafe acts into slips, lapses and mistakes. A slip is an action carried out wrongly, such as pressing the button next to the intended one. A lapse is a memory failure, such as forgetting to re-enable a safety scanner after maintenance. A mistake is a wrong plan carried out correctly, such as choosing the wrong recovery procedure because of a misdiagnosis. Violations, deliberate departures from rules, are a separate category, often driven by production pressure or badly designed procedures. Reason's Swiss cheese model pictures defences as slices with holes; accidents happen when holes in several layers line up.

For robot systems, error-tolerant design follows from these categories. Slips are reduced by spacing and shaping controls so they cannot be confused, and by confirming irreversible actions. Lapses are reduced by checklists and interlocks that make a forgotten step impossible to skip. Mistakes are reduced by training, clear status displays and diagnostic support. Violations are reduced by making the safe way also the convenient way: if bypassing a light curtain saves a worker ten minutes per shift, someone will eventually bypass it.

Physical ergonomics

Human factors also covers bodies. Collaborative workstations should keep frequently handled parts within easy reach, generally the area reachable with the elbows near the body, and avoid repeated reaching above the shoulder or bending below the knee. Working height depends on the task: precision work higher, heavy work lower. Anthropometric tables give body dimensions across populations, and designers commonly aim to accommodate from the 5th percentile female to the 95th percentile male for key dimensions, although that rule of thumb still excludes people and adjustable designs are better. Lesson 4 returns to workstation layout for collaborative robots, and Lesson 10 discusses designing for people with disabilities.