Capstone: a perception pipeline for a pick-and-place robot

This capstone brings the course together in one design. A collaborative robot arm must pick mixed parts from a moving conveyor and place them into fixtures. Parts arrive in random positions and orientations, sometimes touching, under factory lighting that changes through the day. The robot needs, for every part, its type, its 6-D pose in the robot base frame, and a time stamp, delivered early enough to plan and execute a grasp before the part leaves reach. You will work through the design as a perception engineer would: write requirements, choose and place the camera, plan calibration, build the detection and pose pipeline, budget latency and error, plan the evaluation, and address safety and privacy. The numbers are illustrative but realistic; the method is what transfers to any robot vision task.

Requirements

Good perception design starts from what the robot must achieve, not from algorithms. The team agrees the following requirements, each testable.

Two requirements deserve comment. P2 is about the whole chain, not just the camera: calibration, detection, pose estimation, time synchronization and the robot's own accuracy all contribute, so it must be split into an error budget. P8 encodes a design principle from every lesson: perception outputs must carry confidence, and the system must have a safe default when confidence is low.

Camera choice and placement

Parts lie flat on the belt, so a camera looking straight down on a section of conveyor upstream of the robot is natural: the belt is a known plane (lesson 5), which constrains the geometry, and parts are seen without occlusion by the arm. The field of view must cover the belt width of 400 mm plus margin, and the image must resolve P2.

A global-shutter sensor is required: at 0.25 m/s a rolling shutter that reads out over 20 ms would skew parts by 5 mm (ROB-220). The exposure must also freeze motion. At 0.25 m/s and 0.2 mm per pixel, the belt moves 1.25 px per millisecond, so an exposure of 0.4 ms keeps blur to 0.5 px, which needs bright, even lighting: a diffuse LED ring or dome light strobed in sync with exposure, which also makes images consistent through the day and suppresses glare from shiny parts. Fixing exposure and gain, rather than leaving auto-exposure on, keeps detection behaviour stable. A depth camera is not needed for flat parts on a known plane, but height must be known per part type for the z coordinate, and a check for stacked or upturned parts is wise: a simple laser line or a low-cost depth sensor can flag parts whose height does not match their type.

Calibration plan

Three calibrations link pixels to robot coordinates (lesson 3).

• Intrinsics and distortion: a ChArUco board, 25 to 40 images covering the whole field with strong tilts, focus and aperture locked; accept only if RMS reprojection error is below 0.3 px and parameter uncertainties are small.
• Camera to belt plane: the homography or plane pose from a flat target lying on the belt.
• Camera to robot base: the robot touches the corners of a target on the belt with a calibrated pointer tool, or picks up a target and presents it to the camera in several poses. Verification: the robot touches ten independently measured points, and the residual must be within the budget.
• Belt motion: an encoder on the conveyor gives belt position, so each image is tagged with the encoder count at exposure and the part's position can be predicted at any later time. The encoder's scale (millimetres per count) is itself calibrated by tracking a marker over a measured distance.

A fixed fiducial at the edge of the field of view is checked every few minutes in production; if its measured position drifts by more than 0.3 mm, the cell raises an alarm for recalibration.

Detection and pose pipeline

The pipeline follows the structure of lesson 1, with each stage chosen for this task.

1. Acquire: strobed global-shutter image with time stamp and encoder count, triggered every 120 mm of belt travel so each part is seen at least twice.

2. Detect and segment: a small instance segmentation network (lesson 10) gives class, mask and confidence per part; touching parts are separated by their masks. It is trained on images from this cell, with labels bootstrapped by a foundation segmentation model and checked by people, augmented for lighting changes and blur, and split by production day for validation.

3. Estimate pose: for each mask, a keypoint head predicts four to eight known points on the part; their pixel coordinates are undistorted and, since the part lies on the known belt plane at a known height, PnP (lesson 11) gives position and orientation, refined by minimizing reprojection error. For symmetric parts, the orientation is reduced to the symmetry-distinct range.

4. Check: reject a part if detection confidence is below threshold, if reprojection error exceeds 1.5 px, if its measured outline does not match the model, or if consecutive observations disagree by more than 1 mm; these feed requirement P8.

5. Transform and predict: map the pose into the robot base frame with the calibrated extrinsics, and add the belt displacement since exposure using the encoder.

6. Track: associate observations of the same part across frames (lesson 11) using the predicted belt motion, averaging poses to reduce noise.

7. Log: store images of low-confidence parts and failed picks for review and retraining, addressing distribution shift (lesson 9).

Latency and motion compensation

Error budget

Independent error sources combine as the root sum of squares (ROB-310). The team lists 1σ contributions to position error in x and y and checks the total against P2.

import math

budget_mm = {                       # 1-sigma contributions, x-y position in the robot frame
    "feature location (0.5 px at 0.2 mm/px)": 0.10,
    "camera intrinsics and distortion":        0.15,
    "camera-to-robot extrinsics":              0.35,
    "belt plane and part height":              0.15,
    "time stamp and encoder":                  0.15,
    "robot positioning repeatability":         0.05,
    "robot absolute accuracy in the cell":     0.40,
    "part-to-model variation":                 0.20,
}
total = math.sqrt(sum(v ** 2 for v in budget_mm.values()))
for k, v in sorted(budget_mm.items(), key=lambda kv: -kv[1]):
    print("%-42s %.2f mm  (%4.1f %% of variance)" % (k, v, 100 * v ** 2 / total ** 2))
print("combined 1-sigma: %.2f mm, 95 %% (about 2 sigma): %.2f mm, requirement 2.0 mm" % (total, 2 * total))

# latency and throughput check
stages = [9, 6, 4, 38, 18, 3, 2]
print("latency: %d ms of 120 ms" % sum(stages))
picks_per_min, cycle_s = 20, 60 / 20
print("time per pick available: %.1f s" % cycle_s)

robot absolute accuracy in the cell        0.40 mm  (39.8 % of variance)
camera-to-robot extrinsics                 0.35 mm  (30.4 % of variance)
part-to-model variation                    0.20 mm  ( 9.9 % of variance)
camera intrinsics and distortion           0.15 mm  ( 5.6 % of variance)
belt plane and part height                 0.15 mm  ( 5.6 % of variance)
time stamp and encoder                     0.15 mm  ( 5.6 % of variance)
feature location (0.5 px at 0.2 mm/px)     0.10 mm  ( 2.5 % of variance)
robot positioning repeatability            0.05 mm  ( 0.6 % of variance)
combined 1-sigma: 0.63 mm, 95 % (about 2 sigma): 1.27 mm, requirement 2.0 mm
latency: 80 ms of 120 ms
time per pick available: 3.0 s

The budget meets P2 with margin, and it shows where to spend effort: robot absolute accuracy and the camera-to-robot calibration dominate, so improving the camera itself would hardly help, while a better hand-eye calibration or calibrating the robot's kinematics in the cell would. This is the usual lesson of error budgets: the weakest link is rarely the one engineers find most interesting.

Evaluation and acceptance

Requirement P7 asks for 99 % pick success. How many trials does it take to show that? With n trials and zero failures, the 95 % upper confidence bound on the failure rate is about \( 3/n \) (the 'rule of three'), so showing a failure rate below 1 % with no failures needs about 300 picks; allowing a few failures needs more.

The test plan also covers more than the average: each part type and orientation, touching parts, worst-case belt speed, lighting extremes, a dirty lens, and deliberately confusing items such as an unknown part or a stacked pair, which must be skipped and flagged rather than picked (P8). Perception metrics (detection AP, pose ADD, calibration residuals) are tracked alongside the system-level pick success rate, so a regression can be traced to its stage. After deployment, the same metrics are monitored continuously from logs.

Safety, privacy and documentation

The cell is a collaborative application, so its safety functions follow the robot safety standards of ROB-100 and ROB-450 (ISO 10218 and, in Canada and the US respectively, CSA Z434 and ANSI/A3 R15.06) and are implemented with certified sensors such as safety laser scanners or light curtains, not with the vision system. The camera looks down at the belt, but operators may lean into view; images are processed on the cell computer, logged only when a pick fails, cropped to the belt area, and kept for a limited period under the plant's privacy policy (lesson 1).

Finally, the design is documented: requirements and their tests, the camera and lighting specification, calibration procedures and acceptance limits, the model's training data and version, the error and latency budgets, and the procedure for recalibration and retraining. That documentation is what lets someone else maintain the system, and what an auditor or customer will ask for.