5 best practices for securing CI/CD pipelines

The engineer’s mindset is to understand a problem, build a solution, and then figure out how to deploy a robust and secure implementation into production environments.

Unfortunately, it’s often more complicated and expensive to embed security best practices into a solution once it’s implemented, and the pressure to release innovations quickly often leads devops teams to release with security debt. The best devsecops practices are to “shift left” the knowledge, best practices, and security into the development process so that agile development teams are more likely to bake security directly into the microservice, application, or database.

But what about the continuous integration and continuous delivery (CI/CD) pipeline? This automation improves deployment reliability when the manual steps to build, integrate, package, and deliver code to environments are scripted in CI/CD tools. Devops teams with robust CI/CD implementations often take the next step and consider continuous deployment for production environments, which carries more risks but enables more frequent deployments.

Consider these recommendations and best practices to ensure secure, robust CI/CD pipelines.

Establish security development practices well before CI/CD

Kulbir Raina, agile and devops leader at Capgemini, shares a first-things-first principle: “Security and quality must be embedded into the code and should not be left to quality gates when dealing with automation in the CI/CD pipeline.” He continues, “Developers need integrated security tools in their integrated developer environment in order to properly lint the code.”

Linting is a process performed by tools that identify coding style deviations and unsafe practices. More sophisticated Static Application Security Testing (SAST) tools can find buffer overflows, SQL injection flaws, and other issues. Raina recommends integrating SAST into continuous integration.

Steve Jones, devops advocate at Redgate Software, says tools are important, but “like any devops process, ensure you are learning and growing over time.” He says, “It’s critical that you regularly educate your developers on secure coding practices and ensure they are not allowing simple vulnerabilities, such as SQL injection.

Keith Pitt, cofounder and co-CEO of Buildkite, shares several other best practices. He recommends reviewing dependencies from open source and third parties for common vulnerabilities and exposures (CVE). Devops teams should “never put vulnerable software into production.” He suggests “using verifiable signatures for vendor software so if a vendor is compromised, your security supply chain isn’t compromised.”

lkka Turunen, field CTO at Sonatype, agrees. “One of the best practices is to be selective in your search for open source software projects—like in traditional manufacturing, not all parts are created equal.” He recommends, “Looking for projects that are maintained by a group of engaged and responsible developers will not only increase the maintainability of your software supply chain but also decrease the technical debt, rework, and security risk.”

These recommendations are just the tip of the iceberg when it comes to applying best security practices in the software development life cycle, but they are critical prerequisites to developing a secure delivery pipeline.

Build continuous testing into CI/CD pipelines

It’s important to recognize that CI/CD doesn’t just deliver code. It’s also an opportunity to adopt shift-left testing and evolve a continuous testing strategy. Teams that adopt testing as a core principle can then look for opportunities to validate security before triggering CI/CD pipelines to deploy releases to any environment. In addition to integrating SAST security testing, teams should focus on:

• Triggering penetration tests to find back doors and other entry-point vulnerabilities
• Validating security controls and testing authorizations
• Testing for the OWASP Top 10 high-severity issues with Dynamic Application Security Testing (DAST) tools

Testing automation should also factor in steps to remediate common issues, notifications to alert the right teams, and rollback procedures.

Automate data security procedures inside CI/CD

CI/CD pipelines should also be used to automate security procedures that have code and build dependencies. One area to focus on is data security because releases may include new databases, updated data models, or new data sets.

One often-overlooked function is updating development and testing environments with data pulled from production environments. Dev teams should use recently pulled data to validate features and test experiences and employ data masking to obscure personally identifiable information and other data subject to data compliance requirements.

Roman Golod, CTO and cofounder of Accelario, suggests, “Data masking is a critical part of security automation during CI/CD. The development and testing teams need real data to ensure that everything will work smoothly once in production, but the nonproduction systems aren’t usually secure enough.”

Other techniques include using synthetic data and service virtualization. Golod adds, “a synthetic data set to mimic the real thing will further strengthen security as threat actors gain nothing if that database is breached.”

Daniel Riedel, senior vice president of strategic service at Copado, adds a key starting point for devops teams. He says, “Know your data, specifically the security and compliance policies that regulate that data. Once you understand those policies, work carefully to build a phenomenal security automation framework that is well tested and covers you for the rules and controls set out in those policies.”

Apply zero-trust principles to secure the CI/CD pipeline

How should devops teams lock down pipelines so only authorized people can trigger them? Grant Fritchey, devops advocate at Redgate Software, has a recommendation: “The key to automating security within devops pipelines is exactly the same as the key to good security has always been: least-privilege principle,” says Fritchey. He continues, “If you ensure that you only give enough privileges to the pipeline, then automating security in, around, and within it will be simple and deliver the results you want.”

Some basic practices include hiding API keys, defining project- and role-based security credentials in CI/CD tools, and securing access for remote devops team members.

Validate deployments by integrating CI/CD with AIops and security automation

The devops team’s responsibilities don’t end once code is deployed to production. That’s where investments in observability and monitoring become important operational feedback tools. Devops teams should work with the operational teams and tools to respond to incidents and recognize when technical debt is becoming an operational or security concern. Some specifics:

• AIops tools centralize operational data, correlate alerts into incidents, and help automate incident response around performance and reliability issues.
• Security automation protects against threats and attacks while enabling automations that set permissions, patch systems, and respond to security incidents.
• Many CI/CD tools provide two-way integrations with AIops, security automation, and other generalized IT automation tools. Devops teams should trigger notifications to these tools as part of the CI/CD pipeline to inform operations and infosec about code deliveries. They should also allow IT ops and infosec automations to trigger builds or rollbacks to support operational and security needs.

The devops workflow shows a continuous path from planning to monitoring deployments to ensure that teams plan, deliver, release, and run systems reliably and securely. CI/CD is one of the principal devops practices, so embedding security before, inside, and after pipelines is a critical responsibility.