Learning-based perception II: detection and segmentation

A classifier says what is in an image; a robot usually needs to know where. Object detection finds every instance of the classes of interest and puts a bounding box around each, with a class label and a confidence score. Segmentation goes further and labels individual pixels: semantic segmentation gives every pixel a class (floor, wall, person, crop, weed), and instance segmentation separates individual objects with their own masks, which is what a bin-picking robot needs to tell touching parts apart. This lesson explains how modern detectors and segmenters work, how they are evaluated, how foundation models are changing practice, and how to turn their outputs into the 3-D information a robot acts on.

Bounding boxes and intersection over union

A bounding box is usually written as \( (x_1, y_1, x_2, y_2) \), its top-left and bottom-right corners in pixels, or as centre, width and height. The basic measure of how well a predicted box matches a ground-truth box is intersection over union (IoU), the area of their overlap divided by the area of their union:

IoU runs from 0 (no overlap) to 1 (identical boxes). A detection usually counts as correct if its IoU with an unmatched ground-truth box of the same class exceeds a threshold, traditionally 0.5. The same definition applied to pixel sets gives mask IoU for segmentation.

How detectors work

Modern detectors share a structure: a backbone network (lesson 9) extracts feature maps, a neck such as a feature pyramid network combines features from several scales so small and large objects can both be found, and a head predicts boxes and classes.

• Two-stage detectors, led by Faster R-CNN (2015), first propose candidate regions likely to contain objects, then classify each region and refine its box. They are accurate and remain common in research and high-accuracy applications.
• One-stage detectors predict boxes directly from every location of the feature maps in a single pass. The YOLO (You Only Look Once) family, started in 2015 and developed through many versions by several groups since, divides the image into a grid and predicts boxes and class scores for each cell; SSD and RetinaNet are other examples. Their speed made them the default for robots and video, and at the time of writing several YOLO versions are actively maintained.
• Transformer detectors, starting with DETR (2020), treat detection as predicting a set of boxes directly, using attention between learned object queries and image features, and avoid hand-tuned components such as anchors and non-maximum suppression. Real-time variants such as RT-DETR now compete with YOLO models on speed.

Many detectors predict boxes relative to anchors, a fixed set of reference boxes of different sizes and aspect ratios at each location; newer anchor-free designs predict the box edges or centre directly.

Non-maximum suppression

A one-stage detector typically fires several overlapping boxes on the same object, from neighbouring grid cells or anchors. Non-maximum suppression (NMS) cleans this up: sort detections by confidence, take the highest, remove every other box of the same class whose IoU with it exceeds a threshold (often 0.45 to 0.7), and repeat with the next remaining box. NMS has a known weakness in crowds and clutter: two genuinely different objects that overlap heavily, such as stacked parcels or people walking side by side, can have one suppressed. Soft-NMS, which lowers rather than removes overlapping scores, and set-prediction detectors such as DETR reduce this problem.

Evaluating detectors: precision, recall and mAP

Detection outputs a variable number of scored boxes, so evaluation sweeps the confidence threshold. At each threshold, matched detections are true positives, unmatched detections are false positives, and unmatched ground-truth objects are false negatives, giving one precision and recall pair (lesson 9). Plotting precision against recall over all thresholds gives a precision-recall curve, and the area under it is the average precision (AP) for that class. The mean over classes is mAP. The COCO benchmark reports AP averaged over IoU thresholds from 0.50 to 0.95 in steps of 0.05, written AP@[.5:.95], which rewards tight boxes; AP50 uses the single threshold of 0.5.

import numpy as np

def iou(a, b):
    x1, y1 = max(a[0], b[0]), max(a[1], b[1])
    x2, y2 = min(a[2], b[2]), min(a[3], b[3])
    inter = max(0, x2 - x1) * max(0, y2 - y1)
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)

def nms(boxes, scores, thr=0.5):
    order = list(np.argsort(scores)[::-1]); keep = []
    while order:
        i = int(order.pop(0)); keep.append(i)
        order = [j for j in order if iou(boxes[i], boxes[j]) < thr]
    return keep

gt = [(100, 100, 200, 200), (300, 120, 380, 220), (500, 300, 560, 360)]
det = [(102, 98, 205, 203), (110, 105, 210, 196), (305, 125, 385, 225),
       (40, 40, 90, 90), (495, 310, 555, 365), (290, 110, 370, 215)]
score = np.array([0.95, 0.80, 0.90, 0.60, 0.55, 0.40])

keep = nms(det, score, 0.5)
print("kept after NMS:", keep)

# greedy matching at IoU 0.5, in score order, then precision-recall and AP
used = set(); tp = []
for i in sorted(keep, key=lambda k: -score[k]):
    m = max(range(len(gt)), key=lambda g: iou(det[i], gt[g]) if g not in used else -1)
    ok = m not in used and iou(det[i], gt[m]) >= 0.5
    if ok: used.add(m)
    tp.append(ok)
tp = np.array(tp); ctp = np.cumsum(tp)
prec = ctp / np.arange(1, len(tp) + 1); rec = ctp / len(gt)
for p, r, t in zip(prec, rec, tp):
    print("  %s  precision %.2f  recall %.2f" % ("TP" if t else "FP", p, r))
ap = sum((rec[k] - (rec[k - 1] if k else 0)) * prec[k:].max() for k in range(len(rec)))
print("AP50 = %.3f" % ap)

kept after NMS: [0, 2, 3, 4]
  TP  precision 1.00  recall 0.33
  TP  precision 1.00  recall 0.67
  FP  precision 0.67  recall 0.67
  TP  precision 0.75  recall 1.00
AP50 = 0.917

Non-maximum suppression removed box 1, a duplicate of box 0, and box 5, a duplicate of box 2. Of the four survivors, three match real objects and one (box 3, in empty space) is a false positive, so recall reaches 1.0 with final precision 0.75, and the area under the stepped precision-recall curve is 0.917.

Semantic and instance segmentation

Semantic segmentation networks, such as fully convolutional networks, U-Net (2015) and DeepLab, use an encoder that shrinks the image into rich low-resolution features and a decoder that upsamples them back to full resolution, with skip connections carrying fine detail across, so every pixel receives a class. Mobile robots use them to find drivable ground and sidewalks, agricultural robots to separate crop from weed, and inspection robots to outline cracks and corrosion. Instance segmentation adds object identity: Mask R-CNN (2017) extends Faster R-CNN with a small mask head inside each detected box, and YOLO-style models now offer segmentation heads too. Panoptic segmentation combines both, giving instance masks for countable 'things' and class labels for 'stuff' such as road or sky. Segmentation quality is measured with mask IoU, averaged over classes as mean IoU (mIoU) for semantic segmentation and with mask AP for instances.

Foundation models and open-vocabulary perception

At the time of writing, large pre-trained 'foundation' models are changing how robot perception is built. The Segment Anything Model (SAM), released by Meta in 2023 with later versions extending it to video, produces high-quality masks for any object from a prompt such as a point or a box, without being trained on that object class. Open-vocabulary detectors such as Grounding DINO and OWL-ViT find objects described by free text ('the blue mug', 'a torn bag'), drawing on vision-language training similar to CLIP. These models let a robot handle new objects without collecting and labelling a dataset, and they make excellent labelling assistants. Their costs are size and speed (many are too slow for a control loop on embedded hardware), less predictable failure modes, and licensing terms that must be checked before commercial use. A common pattern is to use a foundation model offline to label data, then train a small, fast task-specific detector for deployment.

From 2-D detections to 3-D actions

A box or mask in an image is not yet a grasp. To act, the robot lifts the detection into 3-D. With a depth camera, take the depth pixels inside the mask (not the box, which includes background), back-project them to a point cloud (lesson 8), reject outliers, and compute a centroid, principal axes or a fitted shape; this gives an object position and rough orientation in the camera frame, which the extrinsic calibration transforms into the robot frame. Without depth, a known object size, a known supporting plane, or stereo can provide the missing distance. For precise grasps, 6-D pose estimation from keypoints or models (lesson 11) is needed. Throughout, carry uncertainty forward: a low-confidence detection or a mask that contains few valid depth pixels should lead to a cautious action, a second look from another viewpoint, or a request for help.