Proactive Defense: Hardening Code Pipelines and CI/CD Infrastructure

extract

Proactive Defense: Hardening Code Pipelines and CI/CD Infrastructure Mandiant Introduction The landscape of software supply chain security has undergone a significant shift. Recent campaigns demonstrate that sophisticated threat actors are systematically targeting the engineering lifecycle by compromising trusted security and programming tools. These intrusions reveal three key tactics: - Attackers target trusted security scanners, utility libraries, and AI developer tools to exploit the elevated privileges granted to these systems within build pipelines. - Adversaries target developer workstations and Integrated Development Environments (IDEs) via highly tailored social engineering, malicious extensions, or typosquatted local dependencies to exfiltrate private cryptographic keys, API tokens, and active session credentials directly from local engineering environments. - Rather than relying solely on compromised static credentials, attackers have escalated to advanced pipeline manipulation techniques, including GitHub Actions cache poisoning, OpenID Connect (OIDC) token extraction, and the subversion of mutable action tags to publish compromised packages that still carry legitimate cryptographic provenance. Building upon prior guidance (here, and here), this blog provides an actionable blueprint for software and platform architects designed to safeguard the software supply chain against threat vectors that are actively being exploited, third-party risks, and architectural vulnerabilities throughout the entire Software Development Lifecycle (SDLC). Read on for more on how to establish continuous integration and continuous delivery/deployment (CI/CD) safeguards, strengthen developer workflows, and build robust, end-to-end defense-in-depth. The Multi-Layered Approach Treating each stage of the pipeline as independent security domains is no longer sufficient because these multi-layered attacks target vulnerabilities across the entire build pipeline. Defending against these persistent threats requires a thorough, defense-in-depth approach spanning the five key pillars of the software development lifecycle outlined in Figure 1: Figure 1: The five core pillars for securing the software development lifecycle Endpoint Developer workstations are high-value targets because they hold direct, privileged access to repositories, pipelines, and cloud environments. Threat actors frequently target IDEs, exploiting unmonitored local access to collect personal access tokens (PATs), SSH keys, and proprietary code. Organizations should establish a unified security layer that enforces a consistent security posture across all local host machines and cloud-based development environments. Local Secret Scanning Organizations should deploy pre-commit hooks and IDE-integrated scanning tools to detect and block secrets prior to repository commit. Standardizing local pre-commit templates ensures git trees are fully verified before changes are pushed to central servers. To minimize the impact of a potential leak, organizations should migrate from legacy classic PATs to fine-grained PATs constrained by tight time-to-live (TTL) limits and minimal, environment-specific permissions. Endpoint Security Management Organizations should configure Endpoint Detection and Response (EDR) solutions to monitor developer software integrations and enforce continuous device posture checks. EDR agents should monitor trusted IDE process trees for anomalous file access, unexpected process spawning, and unauthorized outbound network connections. To ensure complete alignment, these EDR compliance signals should be integrated directly with Unified Endpoint Management (UEM) systems to automatically restrict or revoke a user's ability to access Source Code Management (SCM) systems, execute pipeline tasks, or publish code if their device falls out of compliance. Necessary command-line interface (CLI) process exclusions should be strictly restricted to designated, isolated developer environments rather than applied broadly across corporate endpoints. IDE Standardization Organizations should vet and approve specific versions of IDEs, browser integrations, and third-party extensions. IDE and browser marketplaces should be restricted to allow only vetted applications, explicitly blocking unverified extensions. All integrations require a formal third-party risk management review before allowlisting. Organizations should maintain an active software asset inventory paired with strict version-pinning and centralized emergency-block capabilities to stop newly discovered threats. AI-Assisted Security Engineering teams should leverage only approved large language models (LLMs) and AI agents for pre-merge vulnerability analysis and application security testing. This boundary is critical, as threat actors have begun actively inserting malicious code into open-source Model Context Protocol (MCP) packages and tricking AI coding agents (as detailed in our accompanying blog). To mitigate risks like context poisoning and data exfiltration, security teams should deploy context-protection tools to validate inputs before runtime execution. Developers should, wherever possible, exclude local environment (.env) files from the workspace using platform-specific ignore configurations to prevent sensitive credentials from entering the model's context window. Organizations should maintain a human-in-the-loop control model to verify all AI-generated code before it is written to a repository. Isolated Developer Sandboxes To prevent host-level compromises, organizations should, wherever possible, require the use of containerized development environments or dedicated virtual machines (VMs). Sandboxing ensures malicious post-install scripts or dependency-poisoning attacks cannot traverse the local filesystem. Mounting sensitive host paths into workspace containers should be restricted to prevent compromised dependencies from executing with host privileges. Developer guest VMs should be instantiated from centralized, hardened golden images and network isolated from live production environments. All sandbox execution and network activity should integrate into centralized corporate logging. Code Repositories Source code repositories serve as the definitive source of truth for an organization's proprietary software and intellectual property. Hardening this layer requires control over user identities, strict branch governance, and continuous verification of the code history to prevent unauthorized changes from entering the lifecycle. Universal Identity Securing repositories requires strict control over user identities. Implementing a Company Managed User (CMU) model allows organizations to retain full ownership of all accounts, including outside collaborators, and enables the enforcement of phishing-resistant multi-factor authentication (MFA), such as FIDO2 compliant physical security keys or digital passkeys. However, CMU accounts may be inhibited from contributing to external, open-source repositories. Because of this limitation, a standard user model with MFA enforced Single sign-on (SSO) integration remains the recommended approach for teams engaged in public or open-source publishing and private collaboration. Regardless of the chosen account model, identity verification should be continuous. Organizations should deploy conditional access policies to verify device posture before granting access, while monitoring user API activity to quickly detect compromised sessions. Branch Protection Organizations should implement a zero direct-to-main policy, ensuring all changes flow through isolated feature branches that require peer reviews and pass automated CI checks before merging. Administrative bypass policies should be disabled. At the filesystem level, force-push activity should be restricted and monitored. Security teams should continuously analyze audit histories for chronological discrepancies to identify timeline tampering and detect unauthorized dead-drop repositories used for code exfiltration. Credential Lifecycle To prevent long-term persistence, organizations should automate credential rotation, implement just-in-time retrieval mechanisms, and establish a strict token TTL. For developer access, organizations should deprecate PATs which function essentially as static, host-stored passwords vulnerable to local infostealer malware and transition to cryptographically verified SSH-based authentication backed by hardware security keys (such as FIDO2/YubiKey or macOS Secure Enclave). For automated CI/CD pipelines and third-party integrations, organizations should mandate the use of GitHub Apps in place of service account PATs to leverage short-lived, highly scoped access tokens that automatically expire after one hour. Secrets should not be stored in environment variables; local environment files (.env) should be excluded via .gitignore while utilizing native platform secret features for runtime injection. Dependency Security For application manifests utilizing Semantic Versioning (SemVer), organizations should prohibit dynamic version ranges (such as carets ^, tildes ~, or wildcard * operators) that introduce dependency drift during resolution. Instead, configurations should mandate exact SemVer pinning (e.g., 1.4.2) supported by strictly enforced, cryptographically verified lockfiles Unverified execution vectors, such as blind "curl to bash" scripts, should be blocked in favor of direct vendor containers invoked via explicit SHA-256 digests. Organizations should implement Software Composition Analysis (SCA) paired with reachability analysis to prioritize patching vulnerabilities that are actually executed within the application path. Builds should generate a software bill of materials (SBOM) and enforce Supply-chain Levels for Software Artifacts (SLSA) Level 2+ provenance checks. Artifact Management Defending the artifact layer requires controlling what crosses the boundary into the trusted build environment. Point-in-time scanning is no longer sufficient; organizations should continuously inspect and verify upstream components before they propagate downstream. Dependency Cooldowns Organizations should mandate a minimum release-age cooldown of seven days before any newly published public package version becomes installable. Community detection often identifies and removes malicious open-source packages shortly after they are published. Establishing a strict seven-day buffer provides the open-source ecosystem time to detect and pull poisoned releases before they reach internal builds. This delay should be enforced at centralized registries or local configurations; for specific configuration parameters (such as configuring npm's minimumReleaseAge cooldown or secure Python pip indexing), see the technical implementation steps detailed in our accompanying blog. Proxies & Quarantines All external packages and container images should, wherever possible, route through a centralized internal proxy that caches, inspects, and gates each component. Organizations can manage this secure boundary using Google Artifact Registry to host private repositories, configure virtual upstream repositories, and restrict direct build-runner access to public registries. New components arriving through the proxy should be held in a quarantine state and screened, blocking builds automatically on a failed security verdict. Internal repositories should be kept distinct from public registries to prevent dependency confusion attacks, and promotion to the trusted registry should follow a deliberate, policy-driven approval path. Vulnerability Scanning Container images and third-party dependencies should undergo automated scanning at the registry layer and at runtime. Stored artifacts should be continuously re-evaluated as new vulnerabilities emerge. To manage alert volume, results should be prioritized using reachability analysis and real-world exploitation signals, such as the CISA Known Exploited Vulnerabilities (KEV) catalog. Vulnerability Exploitability eXchange (VEX) statements should be used to suppress inapplicable findings and reduce noise. Image Provenance Verifying that an artifact came from a trusted source is as critical as confirming it is free of known vulnerabilities. Provenance establishes this trust by cryptographically signing every internally produced container image and package, then binding each one to the specific build workflow and source commit that created it. Modern signing tooling makes this practical without the burden of managing long-lived signing keys, instead tying each signing event to a build identity and recording it in a public transparency log. A signature is only meaningful when checked, so verification should be enforced at admission, restricted to the exact build identity expected, and performed against an artifact's immutable digest rather than a mutable tag. The same principle extends to the credentials that publish artifacts. Long-lived registry published tokens are a recurring root cause in supply chain incidents, since a stolen token lets an attacker publish poisoned versions under a trusted name. Where possible, these static tokens should be replaced with short-lived, identity-bound publishing tokens issued to a specific build workflow at the moment of release. For first-party builds, adopting a recognized provenance standard provides a consistent benchmark for how and where software was built. SHA Referencing Container image tags and action references are mutable by default, which means an upstream actor can silently replace the content behind a trusted name at any time. Pinning to an immutable cryptographic digest closes this gap, because a digest is a content hash and any change to the underlying artifact produces a different identifier, breaking the reference rather than substituting malicious content under a name the pipeline already trusts. Images should be pinned by digest, and third-party actions should be pinned to a full commit hash rather than a version that can be repointed. This discipline should extend across every image a build touches, not just the primary application image, since base images, sidecars, and init containers are equally viable injection points if left on mutable tags. Teams should also avoid configurations that re-resolve a mutable tag on every restart in production. CI/CD Hardening the automated pipelines within CI/CD infrastructure is a critical requirement for securing the broader software development lifecycle. Because these environments rely on an extensive web of privileged integrations to access source repositories, third-party registries, and cloud infrastructure, they function as high-value targets for adversaries. Securing these build and delivery systems requires the rigorous application of least-privilege principles, the enforcement of strict network boundaries, and the continuous verification of every trusted software component. Runner & Build Servers Hardening CI/CD infrastructure is a critical task because these pipelines require access to code repositories, dependency registries, and cloud environments. To secure these integrations, the primary defensive objective is to eliminate runner persistence. Organizations should use ephemeral, single-use runners, ensuring that every job executes in a fresh, isolated environment that is automatically destroyed upon completion. This clean-slate approach prevents cross-job contamination and denies attackers a permanent foothold. For self-hosted environments, this isolation should extend to the network layer, restricting outbound runner traffic exclusively to pre-approved registries and repository APIs to prevent data exfiltration. Furthermore, to mitigate Poisoned Pipeline Execution (PPE), the execution engine should block unvetted code from pull requests from accessing secrets or triggering deployment-grade runners until an administrator grants manual approval. Additionally, pipelines should protect shared build caches from tampering. Because build caches are frequently shared across branches to speed up builds, a malicious pull request can inject corrupted dependencies directly into the shared cache. If left unrestricted, a subsequent production build will retrieve this poisoned cache and run the malicious code in a trusted environment. Pipeline setups should isolate cache access strictly by branch privilege and reject cache writes from unauthenticated forks. Least Privilege CI/CD - Federated Ephemeral Identities: Prohibit persistent automation secrets within workflows, leveraging OIDC to exchange pipeline identities for short-lived tokens. - Zero-Trust Execution Scopes: Issue read-only or null-permission runner identities by default, requiring components to explicitly request minimum viable permissions. - Shared State Parameterization: Prohibit the automatic inheritance of credentials across downstream templates or nested workflows to isolate sensitive variables. - Runtime Governance: Restrict unsanctioned third-party plugins and marketplace actions. Security teams should also sandbox or disable package installation lifecycle scripts (using configurations like ignore-scripts=true detailed in our accompanying blog) to prevent compromised dependencies from executing arbitrary commands in build environments. - Environment Isolation: Segment network and IAM boundaries so that early-stage validation or linting tasks operate completely decoupled from systems possessing release authority. - Immutable Branch History: Disable history-rewriting functions and force-pushing on canonical branches to maintain an append-only audit trail. - IaC Validation: Scan Infrastructure-as-Code (IaC) prior to deployment to block over-privileged keys, unquoted user-parameter injections, unencrypted webhooks, and runner RBAC misconfigurations. Scanning Gates & Attestation CI/CD scanning gates act as automated quality control within the deployment process, evaluating code against set security standards and automatically halting deployments if the defined criteria are not met. Placing scanning gates as early in the process as possible alerts developers of potential vulnerabilities before they reach production: Secret Scanning (At the Developer Commit / PR Gate): Configure pre-commit hooks and SCM-level scanners to block developer pushes if they contain hardcoded API keys, passwords, or SSH keys. This stops secrets from ever entering your repository's permanent history. SAST - Static Application Security Testing (At the Pull Request / Peer Review Gate): Integrate SAST into your continuous integration (CI) tests to analyze draft code before it is merged into the main branch. This automatically flags structural flaws, logic vulnerabilities, or dangerous functions (like unescaped user inputs) during active development. SCA - Software Composition Analysis (During the Build Phase): Trigger SCA scans when your build environment resolves dependencies. By scanning your package lockfiles (e.g., package-lock.json or requirements.txt) against databases like Google OSV, you can automatically fail builds that attempt to import libraries with active, known CVEs. Container/Image Scanning (At the Registry / Push Gate): Build automated scanners directly into your container registry pipeline. Before a newly built container image is allowlisted for production, the registry scanner should inspect its base OS packages and reject any image containing critical OS-level vulnerabilities or default root access. DAST - Dynamic Application Security Testing (In Staging / Pre-Deployment): Create a temporary, isolated staging instance of your running application as a deployment step. Run automated DAST tests to simulate real-world attacks (like SQL injection or cross-site scripting) against your endpoints, validating that your active runtime defense configurations are working. CSPM - Cloud Security Posture Management (Pre-Deployment IaC Scan & Post-Deploy): Use Policy-as-Code tools to scan your Infrastructure-as-Code (IaC) templates (like Terraform or Kubernetes manifests) before applying changes. This automatically blocks the provisioning of misconfigured cloud environments, such as overprivileged IAM roles or security groups with SSH (port 22) open to the internet. SBOM Generation and Attestation An SBOM is a complete, verifiable inventory of every component that went into a build. Generating and signing the SBOM as part of the build produces this inventory as a tamper-evident attestation rather than an after-the-fact reconstruction. In practice, this means generating the SBOM as a build step in a recognized format such as CycloneDX or SPDX. The resulting SBOM should be signed as an attestation tied to the artifact’s digest, preventing modifications. Signed SBOMs should then be mapped back to affected artifacts without re-scanning every image in the fleet. To keep monitoring useful, VEX statements should be used to flag findings that do not apply to the code, ensuring the inventory remains an actionable triage tool. Deployment Securing the runtime phase ensures that workloads remain protected even if an attacker manages to bypass early pipeline defenses. This operational layer acts as the final quality gate as code transitions from the build pipeline to active production. Workload Protection & Runtime Hardening Workload protection should be enforced directly on running applications and container instances to limit their execution footprint and block active exploits. - Deployment Guardrails: Establish an automated security check at the entrance of your production environment to block any container that lacks a valid cryptographic signature, requests unneeded root privileges, or originates from an untrusted public registry. - Workload Posture: Build workloads from hardened base images and run them on immutable infrastructure with read-only root filesystems and removed SSH capabilities to prevent post-exploit file creation or lateral directory traversal. - Active Application Protection: Deploy Runtime Application Self-Protection (RASP) to block execution-level exploitation attempts like SQL injection. Protect AI workloads from prompt injection and jailbreaks using runtime guardrails such as Google Model Armor. - Just-In-Time Access: Eliminate standing administrative privileges in favor of time-bound, task-scoped access credentials that expire automatically, injecting privileged credentials at runtime only when required. Protecting Live Infrastructure Securing the surrounding network and cloud control plane shields your deployed applications from external threats, blocks lateral movement, and maintains the absolute integrity of your cloud configuration. - Web Application Firewalls (WAF): Deploy edge firewalls to inspect incoming application-layer traffic, filtering out malicious payloads and blocking common web exploits, such as cross-site scripting or OWASP Top 10 vulnerabilities, before they reach your backend services. - API Gateways and Load Balancers: Centralize edge authentication, enforce rate limits, and validate request signatures to prevent direct public exposure of application backends. - Microsegmentation: Enforce granular, identity-aware network policies to isolate workloads and restrict traffic exclusively to pre-authorized service-to-service communication paths, blocking lateral network movement by default. - Configuration Integrity: Deploy Policy-as-Code tooling to continuously validate the live environment against the version-controlled IaC source of truth, automatically reverting out-of-band modifications to prevent unauthorized changes. - Active Posture Scanning: Run Cloud Security Posture Management (CSPM) and Cloud Native Application Protection Platforms (CNAPP) to continuously scan for cloud misconfigurations, overly permissive IAM, and exposed storage. - Continuous Monitoring: Maintain complete visibility across all systems by collecting logs, system metrics, and audit events to quickly detect, trace, and respond to live security events. Conclusion Recent software supply chain campaigns demonstrate that development infrastructure, build pipelines, and developer utilities represent critical threat vectors and key points of compromise. Legacy access controls and point-in-time security scanning are insufficient to defend these environments against sophisticated intrusions. Hardening the development lifecycle requires implementing continuous, automated verification at every stage, unifying security postures across developer endpoints, code repositories, package registries, build runners, and deployment guardrails into a cohesive defensive framework. Ultimately, the objective of pipeline security is to build a resilient architecture capable of isolating and containing an intrusion. By automating cryptographic validation and policy enforcement from the initial code commit to the final production deployment, organizations can significantly reduce their overall attack surface, safeguard downstream consumers, and ensure that any individual compromise is rapidly isolated and resolved before it can spread. Acknowledgements This guidance would not have been possible without the assistance of Arafat Ismail, Bhavesh Dhake, Brentyn Muir, Brian Meyer, Emilio Oropeza, Eyad Mahmoud, Franklin Ramos, Gursev Singh, Omar ElAhdan, Sara Takhim, Stuart Carrera, Stuart Munro, Will Silverstone, and the Mandiant Security Transformation Services team.