Reliability, redundancy and tiers

Equipment fails. Breakers trip, UPS modules fault, generators refuse to start, pumps seize, chillers trip on high pressure, and a contractor drills through a cable. Equipment also needs maintenance: a transformer cannot be tested and a breaker cannot be replaced while current is flowing through it. Yet many data centers promise their customers that IT equipment will essentially never lose power or cooling.

The only way to keep that promise is redundancy: more equipment than the load strictly needs, and more than one path from source to load, arranged so that a failure or a maintenance shutdown anywhere leaves enough capacity to carry the load. Redundancy is expensive, though. A 2N electrical system roughly doubles the electrical equipment, so choosing how much redundancy to build is one of the biggest decisions on any project.

This chapter shows how reliability is measured and calculated, explains the common redundancy schemes (N, N+1, 2N and their relatives), introduces the tier classifications the industry uses to describe them, and shows how to weigh the cost of redundancy against the cost of downtime.

Reliability and availability

Two related measures describe how dependable a system is.

• Reliability \(R(t)\) is the probability that a system works without failure for a stated time. "The generator has a 98% probability of starting and running for the duration of an outage" is a reliability statement.
• Availability \(A\) is the long-run fraction of time a system is working. It depends both on how often things fail and on how quickly they are repaired.

For a repairable component, availability is expressed with the mean time between failures (MTBF) and the mean time to repair (MTTR):

Availability is often quoted in "nines": 99.9% ("three nines") allows 8.8 hours of downtime a year; 99.99% allows 53 minutes; 99.999% ("five nines") allows about 5 minutes.

Annual averages hide something important for data centers: a single interruption of a fraction of a second can crash servers and take hours to recover from. Operators therefore care about the frequency of interruptions, not just the total minutes.

Series and parallel

Systems are built from components connected in two basic ways (STAT-235 derives these rules):

• In series, every component must work. Power passing through a transformer, then a UPS, then a PDU is a series chain: \(A_\text{series} = A_1 A_2 A_3 \cdots\). Each added component lowers availability.
• In parallel, the system works if any one path works: \(A_\text{parallel} = 1 - (1 - A_1)(1 - A_2)\). Redundant paths raise availability dramatically.

Failures that take out redundant equipment together are called common-mode or common-cause failures: two "independent" feeders in the same trench, two UPS systems controlled by the same faulty software, both cooling loops filled from the same contaminated water, or a technician who switches off the wrong side. Much of the art of data center design is finding and removing these shared weaknesses, and above all finding single points of failure, components whose failure alone stops the load.

Redundancy schemes

Redundancy is described relative to N, the number of units needed to carry the full load.

• N: exactly enough capacity, no spare. Any failure or maintenance reduces capacity below the load.
• N+1: one spare unit. If the load needs four generators, five are installed. N+2 adds two spares.
• 2N: two complete, independent systems, each able to carry the whole load. Each side is often called the A side and the B side, matching dual-corded IT equipment.
• 2(N+1): two independent systems, each with its own spare unit.
• Distributed (block) redundancy: several blocks share the load, with enough spare capacity across the blocks that any one block can fail; for example, four blocks each loaded to 75%, so three can carry the load (sometimes written 4N/3). It achieves much of the benefit of 2N with less equipment.

Notice the diminishing returns, and notice what the calculation leaves out. An N+1 generator plant still shares one switchgear lineup, one fuel system and one control system. 2N protects against failures of those shared parts, and lets an entire side be shut down for maintenance. That is why 2N designs remain common in the electrical system even where N+1 would satisfy the arithmetic for the generators alone.

Concurrent maintenance and fault tolerance

Two design goals summarize what redundancy is for.

• Concurrently maintainable: every component and every distribution path can be taken out of service for planned maintenance or replacement without shutting down IT equipment. This requires redundant components and at least two distribution paths, one of which may be idle.
• Fault tolerant: the facility continues to run through any single unplanned failure of equipment or a distribution path, with the effects contained before they reach the IT equipment. This requires two simultaneously active paths, physically separated (compartmentalized) so that a fire or flood in one cannot reach the other, and automatic response to failures.

Concurrent maintainability is about planned events; fault tolerance is about unplanned ones. A design can be concurrently maintainable but still go down from a single unplanned fault, for example if an operator must manually switch to the alternate path.

Tier classifications

The industry describes these levels with the Uptime Institute's Tier Classification System, defined in its Tier Standard: Topology.

• Tier I, Basic Capacity: dedicated space for IT, a UPS, cooling that runs around the clock and an engine generator. No redundancy; maintenance requires shutdowns.
• Tier II, Redundant Capacity Components: adds redundant power and cooling components (UPS modules, chillers, pumps, generators), but still a single distribution path.
• Tier III, Concurrently Maintainable: adds a redundant distribution path, so that every component and path can be maintained without shutting down IT equipment.
• Tier IV, Fault Tolerant: adds fault tolerance and compartmentalization, so that any single equipment or path failure is contained before it affects IT operations.

Each tier includes all the requirements of the tiers below it. Uptime Institute certifies designs (Tier Certification of Design Documents) and, separately, constructed facilities (Tier Certification of Constructed Facility), which includes on-site testing such as pulling the utility power. The distinction matters: Uptime reports that many designs are not built as designed. Uptime removed expected downtime figures from its tier definitions in 2009, so the percentages sometimes quoted for each tier (such as "99.982% for Tier III") are not part of the standard.

Other systems describe similar levels: ANSI/TIA-942, the telecommunications infrastructure standard for data centers, uses Rated-1 to Rated-4 and also covers cabling and architecture; Europe's EN 50600 uses availability classes 1 to 4. Many owners skip formal certification and specify their own requirements, such as "concurrently maintainable electrical and mechanical systems, 2N UPS, N+1 generators and chillers". Most colocation and cloud facilities are designed to about the Tier III level.

Choosing a level: the cost of downtime

Higher redundancy costs more to build, more to maintain and usually more energy (equipment running at partial load is less efficient). The right level depends on what an outage costs. Uptime Institute's surveys find that outages are becoming less frequent, but about one in ten still causes serious or severe disruption, and large outages commonly cost hundreds of thousands to millions of dollars.

Reliability also depends on people. Industry surveys consistently find that human error, such as failing to follow procedures, plays a part in a large share of outages. Clear methods of procedure, training, labeling, and designs that make the right action easy and the wrong action hard are part of reliability engineering (Lesson 5 and DCE-450).