What manipulation is and why it is hard

Manipulation is the part of robotics where a robot changes the world on purpose by touching it. A mobile robot that drives through a warehouse moves itself; a manipulator that picks a box from a shelf, threads a cable through a clip or folds a towel moves something else. That difference sounds small, but it changes almost everything about the problem. The robot must now reason about objects it did not design, contacts it cannot see directly, forces that appear and disappear in milliseconds, and uncertainty in where things are and how they will react. This course builds the tools to handle those problems, starting in this lesson with what manipulation is, how tasks are classified, and why a task a child does without thinking can defeat a well-funded engineering team.

Earlier courses gave you the pieces this one assembles. ROB-210 covered kinematics and the Jacobian, ROB-300 covered dynamics and force and impedance control, ROB-330 covered computer vision and pose estimation, and ROB-350 covered motion planning for arms. Here we focus on what happens at and around the contact: how fingers and suction cups hold objects, how to decide where to grasp, how to size an end effector, and how all of that fits into a working system.

A working definition

In this course, manipulation means purposeful physical interaction between a robot and objects in its environment, usually through an end effector, to change the objects' position, orientation, shape or state. That definition covers picking up a part, but also pushing a box across a table, turning a valve, opening a drawer, inserting a peg, wiping a surface, cutting food and tying a knot. Some of those tasks involve a stable grasp in which the object moves rigidly with the hand. Others, called non-prehensile manipulation, move objects without grasping them at all, by pushing, tilting, toppling or throwing.

It helps to separate three questions that every manipulation system has to answer. First, where is the object and what is it like: its pose, shape, mass, friction and fragility. Second, how should the robot make and manage contact: which end effector, where to place the contacts, how much force to apply. Third, how should the robot move before, during and after contact so the task succeeds and nothing is damaged. Perception answers the first, grasp analysis and synthesis the second, and planning and control the third. Real systems interleave all three, because new sensing changes the plan and contact changes what the robot senses.

A taxonomy of tasks

Classifying tasks is not an academic exercise. The category tells you which hardware, sensing and algorithms are likely to work, and it tells you where the hard parts will be.

Pick and place is the most common industrial task: take an object from one location and put it in another, usually with a defined final pose. Order fulfillment, machine tending, palletizing and kitting are all pick and place. The hard parts are perception (finding the object, especially when objects are piled in a bin), choosing a reliable grasp, and handling the long tail of objects that are shiny, transparent, floppy or heavier than expected.

Assembly and insertion bring two parts together with tight tolerances: a peg into a hole, a connector into a socket, a gear onto a shaft. Position control alone usually fails, because the tolerance is smaller than the robot's absolute accuracy and the perception error. Successful systems use compliance, either passive (a flexible wrist) or active (the impedance and force control of ROB-300), and search strategies that let contact forces guide the final motion.

Tool use means grasping one object to act on another: a screwdriver, a spatula, a sponge, a knife. The robot must grasp the tool in a way that suits the task, not just in a way that holds it, and must control forces at the tool tip rather than at the fingers.

In-hand manipulation reorients or repositions an object within the hand without putting it down, the way a person rolls a pen between their fingers. It is the most dexterous category and remains largely a research topic, covered in lesson 9.

Non-prehensile manipulation includes pushing an object into a corner so it can be grasped, sliding a thin book to the table edge before picking it up, or tilting a box to get fingers under it. Humans do this constantly. Robots that can only grasp from above are surprisingly limited by comparison.

Deformable and articulated objects form their own group because the usual rigid-body assumptions fail. Cloth, cables, food, bags and soft packaging change shape when touched. Doors, drawers, lids and scissors move along joints that the robot must discover or respect. Lesson 11 covers these.

Why manipulation is hard

Manipulation has been studied since the first industrial robots of the 1960s, yet general-purpose manipulation in unstructured settings is still unsolved. Several difficulties combine.

Contact is discontinuous. A finger is either touching an object or not, and when it touches, the contact may stick or slip. The equations of motion change abruptly at each transition. Small errors in position can therefore cause large changes in outcome: a gripper that closes two millimeters off-center may push the object away instead of grasping it. Planning and control methods that assume smooth dynamics struggle with this.

Perception is partial and noisy. Cameras see surfaces, not mass or friction. The hand and the object itself hide the contact region at exactly the moment it matters most. Shiny metal, glass and black plastic defeat many depth sensors. A pose estimate that is good enough to drive toward an object is often not good enough to insert it.

Objects vary without limit. A warehouse may hold hundreds of thousands of different items, many of which the robot has never seen. Even a single product category varies: one box is sealed, the next has a flap open; one bag is full, the next is half-empty. Systems tuned for a few objects often fail on the next one.

Physical properties are uncertain. The coefficient of friction between a rubber fingertip and a cardboard box might be 0.5 on a dry day and 0.3 if the box is dusty. Mass may be unknown until the object is lifted. Center-of-mass location affects whether a grasp will twist. These parameters are hard to estimate before contact and matter a great deal during it.

Failures are costly and varied. A dropped object may be damaged, may land somewhere unreachable, or may jam a machine. A grasp that succeeds 98 % of the time sounds good until you count that a high-throughput cell may attempt thousands of picks per shift, so even small failure rates create frequent human interventions.

Finally, the search space is large. A seven-joint arm with a two-finger gripper has many ways to grasp even a simple object, and each grasp leads to different reachable placements. Choosing well requires reasoning across perception, grasping and motion together.

The manipulation pipeline

Most practical manipulation systems follow a similar sequence, even when the internal methods differ. The robot senses the scene, identifies the object or a region to grasp, chooses a grasp, plans an approach, executes the approach with appropriate control, closes the gripper or activates suction, verifies the grasp, transports the object, places or uses it, releases, and retreats. Each stage can fail, so robust systems check outcomes and loop back when needed.

Verification deserves special attention because it is often omitted in early prototypes. A grasp can be verified by gripper finger position (did the fingers stop at the expected width?), by motor current or force (is the gripper actually squeezing something?), by vacuum level for suction (did the pressure drop as expected?), by a wrist force-torque sensor (did the measured weight increase?), or by vision. Combining two signals is common, because each can fool you on its own.

# Expected attempts and interventions for a pick cell with retries
import math
p = 0.95            # probability a single grasp attempt succeeds
max_tries = 3       # attempts before asking a human for help
picks_per_hour = 600
p_fail_all = (1 - p) ** max_tries
exp_attempts = sum((1 - p) ** k for k in range(max_tries))   # 1 + q + q^2
print(f"probability all {max_tries} attempts fail: {p_fail_all:.5f}")
print(f"expected attempts per pick: {exp_attempts:.3f}")
print(f"human interventions per 8 h shift: {p_fail_all * picks_per_hour * 8:.1f}")
for p2 in (0.90, 0.95, 0.99):
    q = 1 - p2
    print(f"p={p2:.2f}: interventions per shift = {q**max_tries * picks_per_hour * 8:6.2f}")

probability all 3 attempts fail: 0.00013
expected attempts per pick: 1.052
human interventions per 8 h shift: 0.6
p=0.90: interventions per shift =   4.80
p=0.95: interventions per shift =   0.60
p=0.99: interventions per shift =   0.00

The independence assumption in that example is the weak point, and it is a useful lesson in itself. If an item is transparent and the depth camera cannot see it, the second and third attempts will fail for the same reason as the first. Good recovery behaviors change something: they take a new image from another viewpoint, choose a different grasp type, switch from suction to fingers, or push the object to a better position.

Measuring manipulation performance

Because manipulation results are noisy, they must be measured over many trials. Common metrics include grasp success rate (fraction of attempts that lift and hold the object), task success rate (fraction of complete tasks finished without help), mean picks per hour, mean time between interventions, damage rate, and for assembly, insertion success and peak contact force. Benchmarks such as the YCB object set, a standard collection of everyday objects with published models, help researchers compare methods on the same items. Industrial buyers care most about throughput, intervention rate and damage, measured on their own products.

A success rate measured on 50 trials has wide uncertainty. As a rough guide, the standard error of a proportion is \( \sqrt{p(1-p)/n} \), so 47 successes in 50 trials (94 %) gives a standard error of about 3.4 percentage points, and a 95 % interval of roughly 87 % to 100 %. Claims of small improvements need hundreds or thousands of trials. The rule of three from ROB-360 also applies: zero failures in \( n \) trials supports a failure rate below about \( 3/n \) with 95 % confidence.

Where manipulation is used today

Industrial robot arms have handled welding, painting and machine tending for decades, in structured cells where parts arrive in known poses. The newer frontier is unstructured or semi-structured manipulation: picking mixed items in e-commerce warehouses, depalletizing boxes of varying sizes, sorting recycling, harvesting fruit, handling laboratory samples, preparing food, assisting in hospitals and homes. Humanoid robots aimed at factory and logistics work have attracted major investment, and much of their value depends on manipulation rather than walking. At the time of writing (October 2026), reliable general-purpose manipulation in homes remains a research goal, while warehouse picking of many item types has reached commercial deployment at scale.

The economics follow the pipeline. Where objects are uniform and presented consistently, simple grippers and fixed programs work and payback is fast. Where objects vary, the cost shifts to perception, grasp planning, recovery and the engineering effort to reach a high enough success rate. The capstone in lesson 12 works through that trade-off for a bin-picking cell.

How this course is organized

Lessons 2 to 4 build the physical foundations: end effectors, contact mechanics and friction, and grasp analysis. Lessons 5 and 6 cover how to choose grasps and how perception supplies the information needed. Lesson 7 turns the physics into sizing calculations for grippers and suction cups. Lessons 8 to 11 widen the scope to planning whole manipulation tasks, dexterous hands, mobile manipulation, and deformable and articulated objects, with a preview of learned policies that ROB-410 develops. Lesson 12 brings everything together in a cell design.