Defining a continuous testing strategy

Continuous testing is both a practice and a mindset. Developers and quality assurance specialists initiate the practice of continuous testing in the devops CI/CD (continuous integration/continuous development) pipeline, triggering a list of automated tests that run with every build and delivery. The mindset comes when developers, engineers, and quality assurance specialists collaborate on testing strategies and implementations.

This collaboration is critically important because many technology organizations do not adequately fund, dedicate resources, or schedule time for adequate testing. That means the development organization must establish a testing strategy that defines an optimal focus, implementation strategy, and ongoing support functions that fit within constraints.

Although development teams should create a holistic testing strategy, they also need a strategy specific to continuous testing for the following reasons:

• Continuous testing is an optimal way to implement a shift-left testing strategy because it provides developers with feedback before code reaches a delivery environment. It is particularly important for running code quality and security analysis so that developers learn and adopt better coding practices.
• It can be a more expensive investment since continuous tests have to be automated first, integrated into the CI/CD pipeline, and configured with alerts so that tools notify the right people of discovered issues.
• Since these tests run during builds and delivery, teams have to be selective of the types of tests to implement and consider their running durations. Long-running tests are not optimal for continuous testing if they slow down developers and build pipelines.

The best way to review the trade-offs and implementation choices and for teams to collaborate on solutions is by aligning on a continuous testing strategy.

Define a persona-based continuous testing strategy

Let’s define a continuous testing strategy using an agile approach. When product owners create agile user stories, a best practice is to write them from the perspective of the end-user who is receiving value and benefiting from the implementation. These stories often start with the phrase “as a certain user type or user persona” to remind the agile development team who the customer is, why the implementation is important to them, and how the customer benefits.

Defining personas should be fundamental to the strategy since continuous testing has different people who benefit from the tests, and we have to prioritize what types of tests to implement. A few of these personas or stakeholders and their risk concerns include:

• Developers who want to ensure code quality and that their code modifications do not break services or other areas of the code that have dependencies.
• Operations teams concerned that code changes don’t introduce performance issues or impact the reliability of the application.
• Information security teams who are interested in static code analysis, penetration tests, and other early indicators of whether new code or other changes create security risks.
• Quality assurance specialists who represent the interests of the application’s end-users and the product owner. Testing APIs, functionalities, and browser and mobile user interfaces are their primary areas to validate that new, changed, and existing functionality all meet business requirements.
• Architects who represent service and API quality and are the best people to define standards on whether new or changed protocols present a quality concern.
• Database governance specialists concerned about whether developers inadvertently introduced new data quality or security issues in the build.

The agile development team must respond and correct issues when a CI/CD build fails, but what testing gets prioritized now has defined personas acting as stakeholders. These stakeholders should define their priorities on what risks and issues should get flagged to developers early and during the build pipeline.

Define the continuous testing implementation strategy

As mentioned earlier, not all automated tests lend themselves well to continuous testing. First, the tools for running the tests must integrate smoothly with Jenkins, CircleCI, Bamboo, or other top CI/CD tools used for continuous integration and continuous delivery. If the team must perform too much work to integrate tests into the CI/CD pipeline, it takes away from other business-critical and technology work. Tools such as SmartBear, BlazeMeter, Tricentis qTest, BrowserStack, SauceLabs, Postman, and many others have integrations and plug-ins for Jenkins.

Second, running continuous tests requires appropriate computing environments to execute automated tests. Many organizations working on manually configured development, test, and production environments struggle to keep configuration and infrastructure changes synchronized across them. Where feasible, it is better to standardize the environments and use infrastructure as code tools such as Puppet, Chef, or Ansible to manage their configurations before investing in continuous testing.

Lastly, continuous tests must be easy to automate, run in a reasonable duration, have defined pass or fail criteria, and have well-defined paths to remediate issues. For example, functional tests that run through thousands of test scenarios may take too long to execute. They might run better overnight as scheduled jobs. Security tests that report many warning alerts should be reviewed by infosec personnel rather than stopping the build pipeline. A data quality test that can’t easily connect to the code, developer, or team that created the issue should not be in the CI/CD pipeline because it halts the build for everyone and may require an extensive team to review the issue.

It’s also important to research best practices. For example, these 14 key points on continuous testing suggest that testing should be a safety net before pushing changes to an environment, provide actionable feedback, and require executing the right set of validations for the appropriate stage of the delivery pipeline. There are also signs that continuous testing is being done poorly, such as overly investing in user interface testing or not automating test data management.

Use agile principles when prioritizing continuous tests

Once the team agrees on a continuous testing strategy, teams must prioritize the work, considering both the benefits and constraints of proposed tests. Stakeholders should prioritize their recommended tests and rank their importance based on the risks they address. The agile team should determine if the test is appropriate for integrating into the CI/CD pipeline and then estimate the implementation. Lastly, it’s important to catalog these tests because having too many may slow down builds or require increasing the required infrastructure.

Investing in continuous testing leads to improved collaborations and higher quality software, but it requires alignment on priorities, scope, and implementation details. Teams that define a strategy and leverage agile principles for defining personas, prioritizing on value, estimating the development, and supporting the implementation are most likely to reduce risks and deliver value from their testing efforts.