Computational depth, bokeh and portrait modes

Shallow depth of field, a sharp subject against a softly blurred background, is one of the most recognizable looks in portrait photography. With a large sensor and a fast lens it happens optically. A phone's tiny sensor and short focal length make it almost impossible, so phones simulate it: they estimate how far away every part of the scene is, segment the subject, and blur the background by an amount that depends on depth. This lesson explains the optics that phones imitate, the ways phones measure depth (stereo cameras, dual-pixel sensors, time-of-flight and LiDAR sensors, and neural networks that estimate depth from a single image), how synthetic blur is rendered, why hair and glasses still cause trouble, and what else depth maps make possible, from refocusing after capture to relighting and cinematic video.

The optics being imitated

In a real lens, a point at the focus distance is imaged as a point; points nearer or farther are imaged as discs, called circles of confusion. The size of the blur disc grows with the aperture diameter and with how far the point is from the focus distance. For a lens of focal length \( f \) at f-number \( N \) focused at distance \( s \), a point at distance \( d \) produces a blur disc on the sensor of approximate diameter:

Because \( f \) appears squared, a short focal length gives tiny blur. A phone main camera with an actual focal length of about 6 to 7 mm produces blur discs a small fraction of those from an 85 mm portrait lens at the same field of view and f-number, once both are scaled to the same print size. This is the physical reason phone backgrounds stay sharp. The equation also shows that blur rises quickly behind the subject and then levels off toward a maximum for very distant backgrounds, a shape synthetic bokeh must reproduce to look convincing.

import numpy as np
def blur(f_mm, N, s_m, d_m):
    f, s, d = f_mm / 1000, s_m, d_m
    return f ** 2 / (N * (s - f)) * abs(d - s) / d * 1000   # mm on the sensor
s = 2.0                                                      # subject at 2 m
for d in (2.5, 4, 8, 50):
    ff = blur(85, 1.8, s, d)                    # full frame, 36 mm wide sensor
    ph = blur(6.9, 1.8, s, d)                   # phone main camera, about 9.8 mm wide sensor
    ff_pct = ff / 36 * 100; ph_pct = ph / 9.8 * 100
    print(f"background at {d:4.1f} m: full frame {ff_pct:5.2f} % of frame width, phone {ph_pct:5.3f} %")

background at  2.5 m: full frame  1.16 % of frame width, phone 0.027 %
background at  4.0 m: full frame  2.91 % of frame width, phone 0.068 %
background at  8.0 m: full frame  4.37 % of frame width, phone 0.102 %
background at 50.0 m: full frame  5.59 % of frame width, phone 0.130 %

Expressed as a share of the frame width, the full-frame blur is roughly 40 times larger. (The two set-ups do not have the same field of view, because the phone lens is a wide angle, so the comparison is simplified; at matched framing the phone still blurs far less.) To reproduce the full-frame look, a phone must add almost all of the blur in software, which requires knowing depth.

Measuring depth: stereo and disparity

Two cameras a known distance apart see the scene from slightly different viewpoints. A nearby object shifts more between the two images than a distant one. This shift, measured in pixels, is called disparity, and it is inversely proportional to depth:

Here \( Z \) is depth, \( f_{\text{px}} \) is the focal length expressed in pixels, \( B \) is the baseline between the cameras and \( d \) is the disparity. Phones with two rear cameras can use them as a stereo pair. Finding disparity means matching each patch in one image to the corresponding patch in the other along the same row, much like the alignment search in lesson 3. Because the baseline on a phone is only about 1 to 2 cm, disparities are small and depth precision falls off quickly with distance: a one-pixel error in disparity corresponds to a large depth error for distant objects.

f_px, B = 2800, 0.012
for Z in (0.5, 1, 2, 4, 8):
    d = f_px * B / Z
    z_hi = f_px * B / (d - 1) if d > 1 else float("inf")
    z_lo = f_px * B / (d + 1)
    print(f"depth {Z:4.1f} m: disparity {d:5.1f} px, a 1 px error gives {z_lo:5.2f} to {z_hi:6.2f} m")

depth  0.5 m: disparity  67.2 px, a 1 px error gives  0.49 to   0.51 m
depth  1.0 m: disparity  33.6 px, a 1 px error gives  0.97 to   1.03 m
depth  2.0 m: disparity  16.8 px, a 1 px error gives  1.89 to   2.13 m
depth  4.0 m: disparity   8.4 px, a 1 px error gives  3.57 to   4.54 m
depth  8.0 m: disparity   4.2 px, a 1 px error gives  6.46 to  10.50 m

Dual-pixel and other depth sensors

Many phone and camera sensors use dual-pixel autofocus: each pixel is split into left and right halves under one microlens, so each half sees light from one side of the lens. The two half-images form a tiny stereo pair with a baseline equal to a fraction of the lens aperture, about a millimeter or less on a phone. Google's Pixel phones used dual-pixel data, combined with machine learning, to compute depth for Portrait mode from a single camera, a method described in research papers from 2018 onward. The disparities are tiny, so the depth is coarse, but combined with semantic segmentation of people it is enough to separate subject from background.

Active sensors measure depth directly. Time-of-flight (ToF) sensors emit infrared light and measure how long it takes to return. LiDAR scanners on some phones emit a pattern of laser pulses and measure their return times at many points. Active sensors work in darkness and on textureless surfaces where stereo fails, but they have low resolution and limited range, typically a few meters. Phones fuse them with camera images, using the camera's high resolution to sharpen the depth map's edges.

Monocular depth and segmentation

Neural networks trained on millions of images can estimate relative depth from a single photograph by learning cues humans also use: perspective, occlusion, texture gradients, familiar object sizes and haze. Research models such as MiDaS and Depth Anything, and the models built into phones and editing software, produce convincing depth maps for ordinary scenes. They are guesses, however. A network has no measurement of distance; it infers depth from what similar scenes usually look like, and it can be fooled by photographs of photographs, unusual scales or mirrors. In practice, phones combine learned depth with whatever measurement they have, and add a segmentation network that outputs a matte: a per-pixel estimate of how much each pixel belongs to the subject, including fractional values at soft edges such as hair.

Rendering synthetic bokeh

With a depth map, the pipeline must blur each pixel according to its distance from the chosen focal plane. The blur radius is usually computed from a lens model like the equation at the start of this lesson, scaled to imitate a chosen virtual aperture:

This simplified form, with \( K \) a constant set by the virtual aperture, follows from the thin lens model when the focal length is small compared with the distances. Blur is proportional to the difference in inverse depth, which is why disparity, itself proportional to inverse depth, is convenient. Rendering is harder than it looks. A naive approach blurs each pixel with its own radius, but real defocus involves light from background points spreading behind the edges of foreground objects, and foreground blur spreading over the subject. Good renderers treat the scene as layers by depth and composite them from back to front, or "scatter" each pixel's light as a disc onto the output. They also shape the blur kernel like a real aperture, add bright, crisp-edged discs for highlights (requiring HDR values so that light sources stay bright after blurring), and sometimes imitate optical quirks such as cat's-eye shapes near the corners.

import numpy as np
s = 2.0                                        # focus on the subject at 2 m
for N, K in ((1.8, 40.0), (4.0, 18.0)):          # K scales with virtual aperture size
    radii = [K * abs(1 / s - 1 / d) for d in (1.5, 2.0, 2.5, 4.0, 10.0, 1000.0)]
    print(f"f/{N}: " + ", ".join(f"{r:4.1f}" for r in radii) + "  px at 1.5, 2, 2.5, 4, 10 m and infinity")

f/1.8:  6.7,  0.0,  4.0, 10.0, 16.0, 20.0  px at 1.5, 2, 2.5, 4, 10 m and infinity
f/4.0:  3.0,  0.0,  1.8,  4.5,  7.2,  9.0  px at 1.5, 2, 2.5, 4, 10 m and infinity

The output shows blur growing behind the subject toward a maximum at infinity, about half the radius at 4 m already, and growing faster in front of the subject. A smaller virtual aperture scales all radii down. Portrait modes typically let the user pick a virtual f-number after capture and move the focus point, because the depth map is saved with the image.

Where portrait modes fail

Depth and segmentation errors produce characteristic artifacts. Fine hair may be cut off sharply or blurred into the background. Transparent and reflective objects, such as glasses, wine glasses and windows, confuse both stereo and networks. Gaps between arms and the body may stay sharp when they should be blurred. Objects at the same depth as the subject, such as the chair they sit on, may be blurred if segmentation labeled only the person. Ground planes may show an abrupt transition instead of a gradual increase in blur. Highlights in the background may look like soft blobs rather than bright discs if the image was tone mapped before blurring. Manufacturers have improved all of these with better mattes and models, but critical viewers can still spot them, especially in large prints.

Beyond bokeh: what depth enables

Depth maps make other effects possible. Refocusing after capture lets the user move the focal plane to a different subject. Portrait lighting effects simulate studio lights by estimating surface normals of a face and relighting it. Background replacement uses mattes to swap or remove backgrounds in video calls. Cinematic video modes apply synthetic depth of field to video, with focus racking between subjects chosen automatically or by tapping, and let the focus be changed in editing. Augmented reality places virtual objects behind real ones. Spatial photos and videos for headsets use depth or stereo capture to create three-dimensional images. In each case, the depth information is partly measured and partly inferred, and the effect is a rendering rather than an optical record.

The last point bears emphasis. A phone that labels its effect "f/2" is describing an imitation calibrated by the manufacturer, usually to resemble a full-frame lens at that f-number, not the phone lens's own aperture. For honest communication with clients, describe phone portrait blur as synthetic.