3 security best practices for all DevSecOps teams

It’s been over 10 years since Shannon Lietz introduced the term DevSecOps, aiming to get security a seat at the table with IT developers and operators. The question is, how far has security come since then? Do DevSecOps teams have the culture, practices, and tooling they need to release technology into production faster but also reliably and securely?

The recently published SANS DevSecOps Survey shows significant traction. More organizations are looking to shift-left security to ensure that security is prominent in their development practices. Over 50% of respondents claimed they resolved critical security risks and vulnerabilities in seven days or better. But even though nearly 30% of respondents said they deployed to production weekly, only 20% were assessing or testing for security vulnerabilities at a similar velocity. Additionally, the adoption rate for DevSecOps practices topped out at 61% for automation and 50% for continuous integration (CI). Many organizations are still working toward mature security and continuous deployment.

3 security best practices for DevSecOps

Technology leaders and DevSecOps teams struggle to determine which security practices to prioritize and mature. The SANS survey lists over 25 security practices and techniques that at least 50% of respondents said were useful. The survey also identifies eight code-based methods, but fewer than 30% of respondents said they had applied them to at least 75% of their codebase.

While many DevSecOps practices need consideration, security experts share a consistent message on DevSecOps fundamentals. Frank Schugar, CEO of Aerstone, says, “Remember to “build security in, don’t try to bolt it on,” and that “if you do a good requirements process, you must include security requirements, not just the functional ones.”

“Shifting left needs to be non-intrusive and frictionless in securing the DevOps effort,” adds John Morton, field CTO of Britive. “In practice, every practitioner should be demanding security guardrails versus security roadblocks in policy and tooling.”

Determining which security practices to focus on requires that we account for business goals, risks, development velocity, the technology stack, compliance requirements, and other factors. The following three are my most likely candidates for teams that want to shift left and integrate security into their software development lifecycle and DevSecOps practices:

1. Institute security in API-first strategies
2. Automate code scanning
3. Standardize data observability practices

1. Institute security in API-first strategies

Ever since Jeff Bezos’ famous API mandate, development teams have recognized the importance of API-first strategies. Many dev teams build APIs for internal use, and advanced teams embracing microservices architectures use API gateways to scale and support development and operational API capabilities. APIs are fundamental to building data products and business models, and they enable the next generation of open machine learning and large language models. 

“APIs are now central to devops, from defining API specs and contracts to managing numerous unmanaged and managed APIs,” says Ivan Novikov, CEO at Wallarm.

Wallarm’s API ThreatStats Report Q3’2023 shows 239 API vulnerabilities identified in the third quarter, with 33% linked to authorization, authentication, and access control. “This trend underscores the increasing relevance of API security in devops, making it a critical aspect to address to ensure robust and secure software development processes and achieve the desired business outcomes,” says Novikov.

The report lists top API security risks, including injections, authentication flaws, cross-site issues, API leaks, and broken access controls.

So, while many organizations have adopted the best practice of implementing APIs, some haven’t fully shifted left and applied security practices during API development. The scale and speed large DevSecOps teams are undertaking in developing APIs, and microservices suggests that more should consider upgrading to security-first API strategies.

2. Automate code scanning

Scanning code for vulnerabilities was once a manual process and performed as part of pair programming disciplines or instituted as a late step in the development process. Today, there are many static code analysis tools, also called static application security testing (SAST) tools, for DevSecOps teams to consider. 

Carl Froggett, CIO of Deep Instinct, says today’s applications are more than just code. “As files, data, code, and components are consumed into a devops repository, they should be scanned for malicious content on ingestion and while available in the repository,” he says. “A security scanning service should be readily available and rechecked before any release for testing and release to production or customers and during any element of the CI/CD pipelines. The earlier a threat is caught, the easier it is to fix and the less disruptive it is to the overall pipeline.”

Code scanning tools can find many common developer mistakes that are high-security risks. “The accidental disclosure of secrets in source code has been the cause of many security incidents over the years,” says Kyle Tobener, head of security and information technology at Copado. “By building secret scanning into your devops pipeline, you can detect and prevent the leakage of passwords and API keys in your code.”

Code scanning will become an even more important tool as organizations explore using generative AI in business and where copilots and large language models impact software development. Devsecops teams must also consider extending their continuous testing practices to support generative AI capabilities.

While SAST helps developers identify vulnerabilities before pushing to production, Dan Garcia, CISO at EDB, recommends adding dynamic application security testing (DAST) capabilities. “DAST is a form of testing against the runtime environment that executes automated techniques from threat actors, allowing teams to scale their test coverage as the platform expands,” he says.

If you’re investing in software development, cloud-native architecture, and CI/CD pipelines, there’s no excuse not to include code scanning capabilities to review code and highlight security vulnerabilities.

3. Standardize data observability practices

I recently celebrated publishing my 1,000^th article, after nearly 20 years of writing about technology, data, and digital transformation best practices. I started blogging to share what I learned as a startup CTO, and my very first post was about application logging. At that time, I was the developer, site reliability engineer, and IT ops for my startup, so when there was a production incident, I was the one fixing it, identifying the root cause, and determining whether and how to fix application issues.

Back then, application logging was the easiest way to get observability data, but today, there’s a proliferation of tools and an explosion of data sources to help developers and SREs gain visibility into how applications perform in production.

Therein lies today’s challenge, and Jeremy Burton, CEO of Observe, Inc., says, “Most tools used to troubleshoot problems in modern distributed applications are siloed—originally designed to either analyze logs, monitor metrics or visualize traces— and were never architected to handle the data volumes we see today.”

If application observability, improving reliability, and increasing performance are the goals, DevSecOps teams will find many tools and practices to consider. Observability solutions include application monitoring tools, AIops platforms, and SRE tools for managing service level objectives. 

DevSecOps should expand the scope of observability in two areas. One is security observability to cover the full stack, including application, integration, and cloud infrastructure. “Security observability involves collecting data from various security tools and systems, including network logs, endpoint security solutions, and security information and event management (SIEM) platforms, and then using this data to gain insights into potential threats,” says David Linthicum.

DevSecOps should also extend observability practices into the dataops and machine learning model (MLops) realm, since issues in these domains can also impact reliability, performance, and security.

“Shift-left data observability means proactively addressing data incidents at an early stage and minimizing the potential impact and cost associated with data issues,” says Rohit Choudhary, co-founder and CEO of Acceldata. “This not only ensures the reliability and accuracy of data consumed by users but also safeguards the integrity and trust of downstream processes and decision-making.”

MLops is a delivery pipeline for machine learning models similar to what CI/CD is to applications and infrastructure as code (IaC) is to cloud architectures. Building observability into MLops helps track security issues, such as threat actors triggering pipelines or manipulating data.

Phil Morris, managing director at NetSPI, suggests extending devops to MLops practices and says, “In today’s changing environment, the work, processes, and change controls that have traditionally made up the term devops don’t consider the goals and paradigms of MLOps.”

With so many methodologies, tools, and risks that improving observability can address, the key takeaway for DevSecOps teams is where to create standards. If every application, data pipeline, and ML model uses different observability naming conventions, practices, and tools, it complicates whether SREs and security operations centers (SOCs) can quickly identify and resolve security issues.

Beyond the top 3

I highlighted three security practices likely to impact many DevSecOps teams and where continuous investment and standards can address many security risks.

The SANs report highlights many other application security practices that should already be commonplace in IT organizations, such as third-party penetration testing, security training, and implementing a web application firewall (WAF). Other practices, such as container security scanning and cloud-native application protection platforms, are relevant when DevSecOps is implemented on modernized architectures.

The choice of which security areas to focus on isn’t getting easier, but there are too many risks when IT bolts on security. Instead, teams should devote priority to continuous DevSecOps security practices.