Burst photography and multi-frame alignment

The single most important idea in phone photography is the burst: instead of one exposure, capture many short ones and merge them. A burst gathers more light than any one frame without the motion blur of a long exposure, because each short frame freezes movement and the merging happens in software. But frames taken by a moving hand of a moving world do not line up. The hard part of burst photography is alignment and robust merging: matching each frame to a reference, and deciding pixel by pixel which information to trust. This lesson works through the mathematics of frame averaging, shows how alignment is done with image pyramids and tiles, and explains how robust merging avoids the ghosts that appear when people, leaves or water move between frames. The same techniques underlie HDR, night modes and super-resolution in the lessons that follow.

Why bursts work

Suppose a scene is static and the camera is perfectly still. Each frame records the same signal plus independent random noise. Averaging \( n \) frames leaves the signal unchanged but reduces the noise standard deviation by \( \sqrt{n} \), because independent noise partly cancels:

This is the same improvement as an exposure \( n \) times longer, apart from read noise, which each frame adds separately (lesson 2). Short frames have a further advantage: each can be exposed to protect highlights, so the merged result has both low noise and unclipped highlights. Google's HDR+ pipeline, described by Samuel Hasinoff and colleagues in 2016, made this its central design: capture a burst of identically and deliberately underexposed frames, align and merge them, then brighten the shadows in tone mapping. Underexposing every frame equally, rather than bracketing, makes alignment easier because the frames look alike.

import numpy as np
rng = np.random.default_rng(420)
truth = np.full((200, 200), 0.2)            # a flat patch, linear value 0.2
sigma = 0.05                                 # per-frame noise
for n in (1, 2, 4, 8, 16, 32):
    frames = truth + rng.normal(0, sigma, (n, 200, 200))
    merged = frames.mean(axis=0)
    measured = merged.std()
    print(f"{n:2d} frames: noise {measured:.4f}  predicted {sigma / np.sqrt(n):.4f}  SNR {0.2 / measured:5.1f}")

 1 frames: noise 0.0495  predicted 0.0500  SNR   4.0
 2 frames: noise 0.0355  predicted 0.0354  SNR   5.6
 4 frames: noise 0.0251  predicted 0.0250  SNR   8.0
 8 frames: noise 0.0176  predicted 0.0177  SNR  11.3
16 frames: noise 0.0125  predicted 0.0125  SNR  16.0
32 frames: noise 0.0089  predicted 0.0088  SNR  22.5

The measured noise follows the prediction closely. Note the diminishing returns: going from 1 to 4 frames halves the noise, but going from 16 to 32 improves it by only about 30 percent. Because alignment, memory and time costs grow with each frame, pipelines choose the burst length based on light level, typically a handful of frames in daylight and many more in darkness.

The alignment problem

Real bursts are never perfectly aligned. Hand tremor shifts the camera by several pixels between frames and rotates it slightly. Subjects move: people blink, leaves flutter, cars pass. Alignment means estimating, for each frame, the transformation that maps it onto a chosen reference frame. The reference is usually the sharpest frame or the one closest to the shutter press, chosen by measuring gradient energy as a sharpness score.

Global methods estimate one transformation for the whole frame, such as a translation, a rotation or a homography (a projective transform that handles camera rotation). Global alignment is fast and robust but cannot handle parallax, where near and far objects shift by different amounts, or independent subject motion. Local methods divide the image into tiles and estimate a displacement for each tile, which handles parallax and some motion. The densest methods compute optical flow, a motion vector for every pixel, which is most flexible but expensive and prone to errors in textureless areas.

Coarse-to-fine alignment with pyramids

Searching every possible shift at full resolution would be slow. Pipelines instead build a Gaussian pyramid: the image is repeatedly blurred and downsampled by two, producing a stack of progressively smaller versions. A shift of 32 pixels at full resolution becomes 4 pixels at the level that is one-eighth the size, where it can be found by a small search. That estimate is passed down, doubled, and refined with another small search at the next level, and so on to full resolution. For each tile, the search minimizes a difference measure between the reference tile and shifted versions of the alternate frame's tile, commonly the sum of absolute differences (L1) or squared differences (L2), sometimes refined to sub-pixel precision by fitting a quadratic to the error surface around the minimum.

import numpy as np
rng = np.random.default_rng(420)
base = rng.random((80, 80))
# smooth it so it looks like an image rather than pure noise
for _ in range(3):
    base = (base + np.roll(base, 1, 0) + np.roll(base, -1, 0) + np.roll(base, 1, 1) + np.roll(base, -1, 1)) / 5
true_shift = (3, -5)                                   # rows, columns
alt = np.roll(base, true_shift, axis=(0, 1)) + rng.normal(0, 0.01, base.shape)
ref_tile = base[24:56, 24:56]
best = None
for dy in range(-8, 9):
    for dx in range(-8, 9):
        tile = alt[24 + dy:56 + dy, 24 + dx:56 + dx]
        ssd = np.sum((tile - ref_tile) ** 2)
        if best is None or ssd < best[0]:
            best = (ssd, dy, dx)
print(f"true shift {true_shift}, estimated shift ({best[1]}, {best[2]}), SSD {best[0]:.4f}")

true shift (3, -5), estimated shift (3, -5), SSD 0.1092

The search recovers the shift exactly even with noise, because the smoothed texture gives a clear minimum. In flat, textureless tiles such as clear sky, many shifts give nearly the same error, so the estimate is unreliable; fortunately misalignment in flat areas is also nearly invisible. Repeating texture, such as fences and tiles, can produce false minima, a known cause of merging artifacts.

Robust merging and ghosting

Even after alignment, some pixels will not match: a person moved their arm, a wave broke, or alignment simply failed in a tile. Plain averaging would blend the moving object at several positions into semi-transparent copies, called ghosts. Robust merging prevents this by comparing each aligned frame to the reference and reducing the weight of pixels that differ by more than noise can explain. The noise model from lesson 2 tells the algorithm how much difference to expect from noise alone at each brightness level. Differences much larger than that are treated as motion or misalignment and mostly ignored.

HDR+ performs this merge in the frequency domain on tiles, using a pairwise Wiener-filter style weighting: for each frequency component of each tile, the contribution of an alternate frame is shrunk toward the reference in proportion to how much it differs relative to the expected noise. Other pipelines use per-pixel weights. The cost of robustness is that areas with motion receive fewer frames and therefore remain noisier. You can sometimes see this in phone night shots: a moving person is noisier or softer than the static background around them.

import numpy as np
rng = np.random.default_rng(420)
n, sigma = 8, 0.04
scene = np.full((40, 40), 0.3)
frames = []
for k in range(n):
    f = scene.copy()
    col = 5 + 4 * k                      # a bright bar moves 4 px per frame
    f[10:30, col:col + 3] = 0.9
    frames.append(f + rng.normal(0, sigma, f.shape))
frames = np.array(frames)
ref = frames[0]
plain = frames.mean(axis=0)
d = np.abs(frames - ref)
w = np.clip(1 - (d - 2 * sigma) / (2 * sigma), 0, 1)   # full weight within 2 sigma, zero beyond 4 sigma
robust = (w * frames).sum(axis=0) / w.sum(axis=0)
truth_ref = scene.copy(); truth_ref[10:30, 5:8] = 0.9
bg = (slice(10, 30), slice(9, 36))        # where the bar was in later frames but not in the reference
print(f"mean ghost brightness along the path, plain average: {(plain[bg] - 0.3).mean():+.3f}")
print(f"mean ghost brightness along the path, robust merge:  {(robust[bg] - 0.3).mean():+.3f}")
print(f"static area noise, plain {plain[32:, :].std():.4f}, robust {robust[32:, :].std():.4f}, single frame {ref[32:, :].std():.4f}")

mean ghost brightness along the path, plain average: +0.058
mean ghost brightness along the path, robust merge:  +0.000
static area noise, plain 0.0141, robust 0.0156, single frame 0.0397

The plain average leaves faint copies of the bar along its path, while the robust merge rejects them and keeps the static background almost as clean as the plain average. Robust merging therefore gives most of the noise benefit without ghosts, at the price of more noise where motion occurred.

Zero shutter lag and frame selection

Burst pipelines also solve shutter lag. Because the camera continuously fills a ring buffer with frames while the viewfinder is active, pressing the shutter can select frames already captured at that instant, known as zero shutter lag (ZSL). The pipeline can choose the sharpest frames from the buffer, discard frames blurred by a hand jolt, and select a reference in which faces have open eyes. Some phones capture one longer frame after the press as well, to add shadow detail, and merge it with the buffered short frames. Group shots benefit from per-face selection: a few pipelines and editing apps replace a blinking face with the same face from another frame of the burst. That crosses from merging into compositing, a distinction that matters in documentary contexts.

Bursts with dedicated cameras

Dedicated cameras can use the same techniques. Many offer high-speed bursts with electronic shutters, and software such as stacking tools for astrophotography or noise-reduction stacks merges them. Handheld multi-shot noise reduction modes, available on several camera brands, capture a short burst and merge it in camera. For landscapes, photographers manually take several identical exposures and average them in an editor using median or mean stacking, which also removes moving people from a busy scene when using the median. The principles are the same as in phones: align first, reject outliers, then average.

Limits and honest description

Burst merging is mostly measurement: it collects more photons over a longer total time. But it also builds an image that represents an interval rather than an instant, and robust merging decides what to keep. A merged image of a moving scene may show each part at a slightly different moment. For most purposes this is harmless and invisible. For forensic and documentary purposes, it can matter: the relative positions of moving objects might not correspond to any single instant. When a photograph's exact timing is evidence, a single-frame capture may be more defensible, and knowing whether your device merged frames is part of honest description, as lesson 11 discusses.