Text, data formats and exceptions

Engineering data rarely arrives as clean columns of numbers. It comes as text: log files from data loggers and control systems, CSV exports with stray spaces and missing values, configuration files, instrument tags such as "FT-101", timestamps in several formats, and occasional garbage when a sensor fails. Programs that process such data must pull it apart reliably, and they must cope gracefully when something unexpected turns up, rather than crashing or, worse, silently producing a wrong answer. This lesson builds on the strings of Lesson 2 and the files of Lesson 5. It covers string methods for parsing text, formatting numbers for reports, handling errors with exceptions, raising your own exceptions to protect functions from bad inputs, assertions as self-checks, and reading and writing structured data in JSON.

Strings as sequences

A string is a sequence of characters, and the indexing and slicing of lists (Lesson 5) work on strings too: tag[0] is the first character and tag[-3:] the last three. Strings are immutable: methods such as replace or upper return a new string and leave the original unchanged, so you must assign the result to keep it. The most useful methods for engineering text are:

line = "  2026-03-04 14:05 , T-101 , 23.46 ,degC  "
parts = [p.strip() for p in line.split(",")]      # split on commas, trim spaces
print(parts)
stamp, tag, value, unit = parts
date, time = stamp.split()
print(date, time, tag.lower(), float(value), unit.replace("degC", "C"))
print(tag.startswith("T-"), "101" in tag, len(tag))

# Formatting numbers for reports
p, eff = 1234567.891, 0.87654
print(f"Power {p:,.1f} W | {p / 1e6:.3f} MW | {p:.3e} W")
print(f"Efficiency {eff:.1%} | padded [{tag:>8}] [{tag:<8}] [{value:^9}]")

['2026-03-04 14:05', 'T-101', '23.46', 'degC']
2026-03-04 14:05 t-101 23.46 C
True True 5
Power 1,234,567.9 W | 1.235 MW | 1.235e+06 W
Efficiency 87.7% | padded [   T-101] [T-101   ] [  23.46  ]

The first part of the example splits a messy line at its commas and strips the spaces from each piece in a single list comprehension (Lesson 5). Unpacking the four pieces into named variables makes the rest of the code readable. Note that the value is still text until float() converts it: "23.46" + 1 is an error, while float("23.46") + 1 is 24.46.

Formatting output

Reports and logs need numbers formatted consistently. Inside an f-string, a format specification after a colon controls how a value appears:

• .2f gives two decimal places; ,.1f adds thousands separators; .3e uses scientific notation.
• .1% multiplies by 100 and adds a percent sign.
• >8, <8 and ^9 right-align, left-align and center a value in a field of that width, which lines up columns in a printed table.

Choose the number of decimal places to match the precision of the data (ENGG-160): printing a pump efficiency as 87.65432 % implies a precision that does not exist.

Parsing a real data file

Real data files contain surprises: blank fields, error codes written by the instrument, header lines, values with a decimal point dropped. A program that assumes every line is perfect will crash on the first bad one, or, if it is too forgiving, quietly average garbage into the result. The robust approach is to attempt to convert each line, catch the failures, and keep a record of what was skipped and why.

Exceptions: handling errors

When Python encounters an error it cannot handle, such as converting "ERR" to a number, dividing by zero, or opening a file that does not exist, it raises an exception. If nothing handles it, the program stops with a traceback (Lesson 1). A try statement lets a program handle anticipated errors:

• Code that might fail goes in the try block.
• An except block names the kind of exception to handle, such as ValueError, ZeroDivisionError, FileNotFoundError or KeyError, and says what to do instead.
• An optional else block runs only if no exception occurred, and a finally block always runs, which is useful for closing connections or files.

log = [
    "time_s,flow_Lps",
    "0,12.1",
    "10,12.4",
    "20,",                 # missing value
    "30,12.9",
    "40,ERR",              # sensor error code
    "50,13.2",
    "60,131",              # suspicious: decimal point dropped?
    "70,13.0",
]
good, skipped = [], []
for n, row in enumerate(log[1:], start=2):        # skip the header; count file lines from 1
    try:
        t_text, q_text = row.split(",")
        t, q = float(t_text), float(q_text)
    except ValueError as err:
        skipped.append((n, row, str(err)))
        continue
    if not 0 <= q <= 50:                          # physically plausible range
        skipped.append((n, row, "out of range"))
        continue
    good.append((t, q))

print(f"Read {len(good)} good rows, skipped {len(skipped)}:")
for n, row, why in skipped:
    print(f"  line {n}: {row!r} ({why})")
mean_q = sum(q for t, q in good) / len(good)
print(f"Mean flow {mean_q:.2f} L/s")

Read 5 good rows, skipped 3:
  line 4: '20,' (could not convert string to float: '')
  line 6: '40,ERR' (could not convert string to float: 'ERR')
  line 8: '60,131' (out of range)
Mean flow 12.72 L/s

Three lines failed for three different reasons. Two raised ValueError when converted, and the third converted successfully but was physically implausible: 131 L/s from a pump that otherwise delivers about 13 L/s is almost certainly a dropped decimal point. Range checks catch errors that type conversion cannot. The program reports every skipped line with its line number and reason, so a person can check them; silently discarding data would hide a failing sensor.

Two cautions about exceptions. First, catch only the specific exceptions you expect. A bare except: catches everything, including typing mistakes in your own code and the keyboard interrupt that stops a runaway program, and it hides bugs. Second, do not use exceptions to paper over problems you should fix: if half the lines in a file fail to parse, the program should stop and say so rather than report an average of the other half.

Raising exceptions to protect functions

Functions (Lesson 4) should refuse inputs that make no sense rather than return a meaningless number. The raise statement creates an exception with a helpful message; the code that called the function can then handle it or let it stop the program. This is far better than returning a special value such as −1, which a caller might forget to check and then use in later calculations.

import math

def pipe_velocity(Q, D):
    """Mean velocity (m/s) for flow Q (m3/s) in a pipe of inside diameter D (m)."""
    if D <= 0:
        raise ValueError(f"diameter must be positive, got {D}")
    if Q < 0:
        raise ValueError(f"flow cannot be negative, got {Q}")
    return Q / (math.pi * D ** 2 / 4)

for Q, D in [(0.05, 0.2), (0.05, 0.0), (-0.01, 0.15)]:
    try:
        print(f"Q = {Q}, D = {D}: V = {pipe_velocity(Q, D):.2f} m/s")
    except ValueError as err:
        print(f"Q = {Q}, D = {D}: rejected ({err})")

assert math.isclose(pipe_velocity(math.pi / 4, 1.0), 1.0)   # a unit check that must hold
print("Self-check passed")

Q = 0.05, D = 0.2: V = 1.59 m/s
Q = 0.05, D = 0.0: rejected (diameter must be positive, got 0.0)
Q = -0.01, D = 0.15: rejected (flow cannot be negative, got -0.01)
Self-check passed

The assert statement checks a condition that must be true if the program is correct and stops with an AssertionError if it is not. Assertions are for catching programming mistakes, such as checking that a function returns 1 m/s for a flow of π/4 m³/s in a 1 m pipe, not for validating user input, because they can be switched off when Python is run with optimization. Validate inputs with if and raise; check your own logic with assert and tests (Lesson 12).

Structured data: JSON

CSV files suit tables of numbers. Many other kinds of data are structured: a project's settings, a list of equipment each with several properties, nested groups of parameters. JSON (JavaScript Object Notation) is the most common text format for such data. Its structure maps directly onto Python's: JSON objects become dictionaries, arrays become lists, and strings, numbers, true, false and null become str, int or float, True, False and None. The json module reads and writes it.

import json

settings = {
    "project": "Pump station P-3",
    "units": "SI",
    "pumps": [{"tag": "P-3A", "rated_kW": 55, "duty": True},
              {"tag": "P-3B", "rated_kW": 55, "duty": False}],
    "alarm_levels_m": {"low": 1.2, "high": 4.8},
}
with open("settings.json", "w") as f:
    json.dump(settings, f, indent=2)              # human-readable file

with open("settings.json") as f:
    loaded = json.load(f)
duty = [p["tag"] for p in loaded["pumps"] if p["duty"]]
print("Duty pumps:", duty)
print("High alarm at", loaded["alarm_levels_m"]["high"], "m")
print("Same as original:", loaded == settings)

Duty pumps: ['P-3A']
High alarm at 4.8 m
Same as original: True

Keeping settings such as alarm levels, file paths and equipment lists in a JSON file rather than in the code means that changing a setting does not require editing the program, reduces the risk of introducing errors, and leaves a clear record of what values were used for a given analysis. Web services and many instruments also send data as JSON, so the same skills let a program read data from online sources and modern equipment.

Other structured formats you will meet include XML (older instruments and building information exchange), YAML (configuration files), and binary formats such as HDF5 for large scientific datasets. Python has libraries for all of them.

Dates and times

Timestamps are a frequent source of bugs. Date formats differ (is 04/05/2026 April 5 or May 4?), clocks change for daylight saving time, data loggers may record local or UTC time, and durations crossing midnight or month ends are easy to get wrong. Python's datetime module parses and formats dates and computes differences correctly. Good practice for engineering data is to store times in the ISO 8601 format (2026-03-04T14:05:00), preferably in UTC with the time zone stated, and to convert to local time only for display.

Text encodings

Files are stored as bytes, and an encoding maps those bytes to characters. Modern files are almost always UTF-8, which can represent every character, including °, µ, Ω and accented letters. Files exported by older software may use other encodings and show garbled characters (such as "°C" instead of "°C") when read as UTF-8. If that happens, open the file with the correct encoding, for example open(name, encoding="latin-1"), and write your own files as UTF-8.