Joint variation, error propagation and sampling distributions

Engineering quantities rarely vary alone. The length of an assembly depends on the lengths of all its parts; a computed density depends on a measured mass and a measured volume; the strength and stiffness of timber tend to rise and fall together. This lesson develops the tools for random quantities that act together. It introduces joint distributions, covariance and correlation for random variables, and the rules for the mean and variance of sums, which lead to statistical tolerancing of assemblies. It then shows how uncertainty in measured inputs propagates into a calculated result. Finally it looks at statistics computed from samples as random variables in their own right, and introduces the chi-square, t and F distributions that underlie the confidence intervals and tests of Lessons 7 and 8.

Joint distributions

Two random variables X and Y are described together by a joint distribution: a table of probabilities \(P(X = x, Y = y)\) for discrete variables, or a joint density \(f(x, y)\) whose double integral over a region gives the probability of falling in it (MATH-209) for continuous ones. The distribution of each variable on its own, its marginal distribution, is found by summing or integrating over the other. X and Y are independent when the joint distribution factors into the product of the marginals, so that knowing one tells you nothing about the other.

Covariance and correlation

The covariance of two random variables measures how they vary together:

It is positive when large values of X tend to go with large values of Y, and negative when they go with small values. Its size depends on the units, so it is usually scaled to the correlation coefficient \(\rho = \text{Cov}(X, Y)/(\sigma_X\sigma_Y)\), which lies between −1 and 1. This is the population version of the sample correlation r of Lesson 1. Independent variables have zero covariance, although zero covariance does not by itself guarantee independence: Y = X² with X symmetric about zero is a standard counterexample.

Sums of random variables and tolerance stack-up

For any constants a and b, the mean of a combination is the combination of the means, \(E[aX + bY] = a\mu_X + b\mu_Y\), and the variance is

For independent variables the covariance term vanishes, and variances (not standard deviations) add. Lesson 4 used this for sums of independent variables; the covariance term matters whenever variables share a common cause, such as parts made from the same bar stock or measurements taken with the same instrument.

The most common engineering application is the tolerance stack-up of an assembly. If n parts with independent dimensions are stacked, the assembly length has standard deviation \(\sqrt{\sigma_1^2 + \cdots + \sigma_n^2}\), which grows only as \(\sqrt{n}\) for equal parts. Worst-case tolerancing, which assumes every part is at its limit in the same direction at once, adds the tolerances directly and so grows as n. For many parts, worst-case design is very conservative and expensive; statistical tolerancing accepts a small, calculated chance of an out-of-tolerance assembly in exchange for much looser part tolerances.

Propagation of error

Most engineering results are calculated from several measured inputs, each with its own uncertainty. If \(Y = f(X_1, \ldots, X_k)\) and the inputs are independent with small standard deviations, a first-order Taylor expansion (MATH-209) gives

with the derivatives evaluated at the mean values. For a product of powers, \(Y = cX_1^{a_1}X_2^{a_2}\cdots\), this simplifies to a rule for relative (percentage) uncertainties:

An input raised to a high power dominates the result, so it deserves the most careful measurement.

When the function is strongly nonlinear or the uncertainties are large, the first-order formula can mislead. Monte Carlo simulation, sampling the inputs and computing the result many times (MATH-300), handles any function and also gives the shape of the output distribution.

Statistics as random variables

A statistic such as the sample mean or the sample standard deviation is computed from random data, so it is itself a random variable with its own sampling distribution: a different sample would give a different value. Lesson 4 showed that the sample mean has mean μ and standard deviation \(\sigma/\sqrt{n}\) (the standard error), and is approximately normal for large n by the central limit theorem. Three further distributions describe other statistics from normal samples.

The chi-square distribution

For a random sample of size n from a normal population, the scaled sample variance follows a chi-square distribution with n − 1 degrees of freedom:

A chi-square variable with ν degrees of freedom is the sum of ν squared independent standard normal variables; it is positive and skewed right, with mean ν and variance 2ν. One degree of freedom is lost because the deviations are measured from the sample mean rather than the true mean, which is also why the sample variance divides by n − 1. This distribution gives confidence intervals and tests for a variance (Lesson 8), important when consistency matters as much as the average, as for the fill weight of packages or the thickness of a coating.

The t distribution

When σ is unknown and estimated by s, the standardized sample mean follows Student's t distribution with n − 1 degrees of freedom:

The t distribution is symmetric like the normal but has heavier tails, reflecting the extra uncertainty of estimating σ from a small sample. As n grows it approaches the standard normal. W. S. Gosset derived it in 1908 while working on small samples at the Guinness brewery in Dublin, publishing under the pen name "Student". For small samples the difference matters, as the simulation below shows.

import numpy as np
from scipy import stats

rng = np.random.default_rng(5)
mu, sigma, n, reps = 50.0, 2.0, 5, 100_000
samples = rng.normal(mu, sigma, size=(reps, n))
xbar = samples.mean(axis=1)
s = samples.std(axis=1, ddof=1)

print(f"SD of sample means: {xbar.std():.3f} (theory sigma/sqrt(n) = {sigma / np.sqrt(n):.3f})")
chi = (n - 1) * s**2 / sigma**2
print(f"(n-1)s^2/sigma^2: mean {chi.mean():.2f}, variance {chi.var():.2f} (chi-square, 4 df: 4 and 8)")

# 95% intervals for the mean using s: with z = 1.96 versus t with n - 1 degrees of freedom
z, t = 1.96, stats.t.ppf(0.975, n - 1)
half_z, half_t = z * s / np.sqrt(n), t * s / np.sqrt(n)
cover_z = np.mean(np.abs(xbar - mu) <= half_z)
cover_t = np.mean(np.abs(xbar - mu) <= half_t)
print(f"t multiplier for 4 df: {t:.3f}")
print(f"Coverage using z = 1.96: {cover_z:.3f};  using t: {cover_t:.3f}")

SD of sample means: 0.893 (theory sigma/sqrt(n) = 0.894)
(n-1)s^2/sigma^2: mean 3.99, variance 7.91 (chi-square, 4 df: 4 and 8)
t multiplier for 4 df: 2.776
Coverage using z = 1.96: 0.877;  using t: 0.950

The simulation confirms the theory: the sample means have standard deviation σ/√n, the scaled variances have the mean and variance of a chi-square with 4 degrees of freedom, and an interval built with 1.96 instead of the t multiplier 2.776 covers the true mean only 88% of the time instead of the promised 95%.

The F distribution

The ratio of two independent sample variances, each divided by its population variance, follows an F distribution with two degrees-of-freedom parameters:

It is used to compare the variability of two processes (Lesson 8) and, most importantly, in the analysis of variance (Lesson 9), which compares several means by comparing variances.