Securing the development environment and the delivery pipeline

Attackers have learned that it is often easier to compromise software where it is made than where it runs. A malicious change slipped into a build server, a stolen credential for a code repository or a tampered build tool can deliver malicious code to every customer at once, signed and trusted. The SolarWinds compromise discovered in December 2020, in which attackers inserted a backdoor into the build process of a widely used network management product, made this risk impossible to ignore. Lesson 10 examines that case and the wider software supply chain. This lesson looks inward, at the development ecosystem the organization itself controls: developer workstations and tools, source code repositories, build systems, artifact repositories, test environments and the continuous integration and delivery (CI/CD) pipeline that connects them.

The CISSP outline asks you to identify and apply security controls in software development ecosystems, naming programming languages, libraries, tool sets, integrated development environments, runtimes, CI/CD, security orchestration, automation and response, software configuration management, code repositories and application security testing. Testing tools are covered in Lesson 9. Here we concentrate on protecting the ecosystem itself.

The development ecosystem as a high-value target

The development ecosystem deserves the same protection as production, and sometimes more. It holds the organization's intellectual property, credentials for production systems and the ability to change what customers run. Yet it is often less protected: developers have broad privileges, test environments are loosely controlled, and build servers are configured once and forgotten. A useful way to think about it is that every component that can change the final software is part of the trusted computing base of that software. If the build server can be altered, the code review that happened before it is irrelevant.

Developer workstations and tools

Developers need flexibility, but their workstations are a common entry point. Controls include managed and encrypted devices with endpoint protection, prompt patching, separate accounts for everyday work and privileged tasks, and phishing-resistant multifactor authentication for code repositories and cloud consoles. Integrated development environments (IDEs) and their plug-ins, browser extensions, command-line tools and package managers all execute code on the workstation, and malicious extensions have been published in public marketplaces. Organizations should maintain a list of approved tools and extensions, obtain them from trusted sources, and review high-risk categories such as tools that request access to source code or credentials. Remote development environments, in which code is edited on centrally managed cloud workspaces rather than laptops, can reduce exposure by keeping code and secrets off endpoint devices.

Source code repositories and configuration management

Software configuration management (SCM) is the discipline of identifying, controlling and tracking every version of every item that makes up the software: source code, build scripts, infrastructure definitions, configuration and documentation. A version control system, today almost always Git hosted on a platform such as GitHub, GitLab or Bitbucket, provides the technical foundation. For security, the repository is where integrity and accountability are established.

Repositories must also be protected against accidental disclosure. Making a private repository public by mistake, or copying code to a personal account, can expose proprietary logic and embedded secrets. Organizations should monitor public code hosting for their own code and credentials, and should treat repository visibility settings as security-relevant configuration.

Secrets in code

A secret is any value that grants access: passwords, API keys, cloud access keys, database connection strings, signing keys and tokens. Secrets embedded in source code are one of the most common and most damaging development mistakes, because repositories are widely copied and their history keeps every value ever committed. Removing a secret in a later commit does not remove it from history; the only safe response to a committed secret is to revoke and rotate it.

The control is to keep secrets out of code entirely. Applications read secrets at runtime from a secrets manager or a cloud key management service, using an identity granted to the workload rather than a stored password. Pipelines receive short-lived credentials through mechanisms such as workload identity federation instead of long-lived keys stored as pipeline variables. Secrets scanning on every push and periodic scans of history catch the mistakes that still happen.

import re
# Defensive secrets scan over an invented configuration snippet (values are fake)
snippet = '''
db_host = payments-db.internal
db_user = refunds_app
db_password = Ch4ngeMe-Example-Only
api_base = https://api.example.test/v2
partner_api_key = rk_live_EXAMPLE0000000000000000
log_level = info
'''
rules = [
    ('password assignment', re.compile(r'(?i)\b\w*(password|passwd|secret)\w*\s*=\s*\S+')),
    ('live API key pattern', re.compile(r'\brk_live_[A-Za-z0-9]{16,}\b')),
]
for n, line in enumerate(snippet.strip().splitlines(), 1):
    for name, rx in rules:
        if rx.search(line):
            key = line.split('=')[0].strip()
            print(f'line {n}: possible secret ({name}) in "{key}" -> block push, revoke and rotate')
print('Values are never printed: the scanner reports the location, not the secret.')

line 3: possible secret (password assignment) in "db_password" -> block push, revoke and rotate
line 5: possible secret (live API key pattern) in "partner_api_key" -> block push, revoke and rotate
Values are never printed: the scanner reports the location, not the secret.

Two management points follow from the example. First, scanners report locations, not values, because the scan output itself must not become a new place where secrets leak. Second, a detected secret triggers revocation and rotation, not just deletion, and the incident should be logged and reviewed to find out how the secret got into the code.

Build systems and pipeline integrity

The build system turns source code and dependencies into deployable artifacts. Its integrity determines whether what was reviewed is what gets shipped. Good practice is to run builds on isolated, ephemeral workers that are created fresh for each build and destroyed afterwards, so that one compromised build cannot persist into the next; to define the pipeline itself as code, stored in the repository and subject to review; to give each pipeline only the permissions it needs for its stage; and to protect the pipeline's own configuration and secrets as carefully as production credentials. Pipelines that run untrusted code, such as contributions from outside the organization, must not have access to secrets or deployment credentials.

Reproducible builds strengthen integrity further. A build is reproducible if building the same source with the same tools always yields bit-for-bit identical output, which lets independent parties verify that a published binary really came from the published source. Full reproducibility is hard to achieve, but even partial progress, such as pinning dependency versions and tool versions, reduces surprises.

SLSA and build provenance

Supply-chain Levels for Software Artifacts (SLSA, pronounced 'salsa') is an open framework, now hosted by the Open Source Security Foundation, that defines increasing levels of assurance for how software is built. Version 1.0, released in 2023, defines a build track with levels from 0 to 3. The core idea is provenance: a signed, machine-readable statement, generated by the build platform itself, recording what was built, from which source, by which build process and with which inputs. Consumers can verify provenance before deploying an artifact.

# Evaluate pipelines against simplified SLSA build-track criteria (illustrative)
criteria = [
    ('L1', 'provenance generated'),
    ('L2', 'hosted build platform'),
    ('L2', 'provenance signed by platform'),
    ('L3', 'isolated ephemeral builds'),
    ('L3', 'signing keys inaccessible to build steps'),
]
pipelines = {
    'refunds-service':  {'provenance generated', 'hosted build platform', 'provenance signed by platform',
                         'isolated ephemeral builds'},
    'mobile-app':       {'provenance generated', 'hosted build platform'},
    'legacy-batch':     set(),
}
for name, have in pipelines.items():
    level = 0
    for lvl in ('L1', 'L2', 'L3'):
        need = {c for l, c in criteria if l == lvl}
        if need <= have:
            level = int(lvl[1])
        else:
            missing = sorted(need - have)
            break
    else:
        missing = []
    print(f'{name:16s} Build L{level}' + (f'   next: {", ".join(missing)}' if missing else ''))

refunds-service  Build L2   next: signing keys inaccessible to build steps
mobile-app       Build L1   next: provenance signed by platform
legacy-batch     Build L0   next: provenance generated

Code signing complements provenance. Signing an artifact with a private key lets anyone holding the public key verify that the artifact has not changed since it was signed and that it came from the holder of the key. Its weakness is key management: if the signing key is stolen, attackers can sign malicious code. Signing keys should live in hardware security modules or managed key services, be usable only by the release pipeline, and be covered by monitoring. The open source Sigstore project offers keyless signing, in which short-lived certificates are tied to a verified identity and every signature is recorded in a public transparency log, making signing easier and misuse more visible.

Artifact repositories and deployment

Built artifacts, such as container images, packages and binaries, are stored in artifact repositories before deployment. Only the release pipeline should be able to publish to the repositories that production pulls from, and production should deploy only artifacts that are signed and whose provenance has been verified, a rule that admission controllers can enforce automatically in container platforms. Internal mirrors or proxies of public package registries let the organization control which external components developers can use and keep builds working if a public package disappears.

Test environments and test data

Test and staging environments should mirror production closely enough to make testing meaningful, but they are often less protected. Two rules are essential. First, separate environments with distinct accounts, networks and credentials, so that a compromise of a test system does not reach production. Second, do not use real personal or payment data in test environments unless there is no alternative and the environment is protected to production standards. Use synthetic data, or production data that has been masked, tokenized or otherwise de-identified. Privacy law in both Canada and the United States treats personal information in a test database exactly like personal information anywhere else.