It’s time to fix flaky tests in software development

If you’ve ever tended to plants, you’ve likely encountered garden pests. When you start to see signs of these seemingly innocuous, sometimes microscopic critters, you know it’s time to take action. Otherwise, a few discolored or spotted leaves could soon multiply, compromising the entire organism. 

Why am I, a software engineer, telling you about gardening? Because flaky tests are the garden pests of software development. Just as keeping garden pests under control is essential to the health of your plants, eradicating flaky tests is essential to the health of your software development process.

Developers are intimately familiar with flaky tests and the pain and toil they cause. However, technology leaders often miss their significance, or even dismiss them as a small inconvenience. That’s a mistake. If you care about your development team and its goals, it’s absolutely critical to understand the impact of flaky tests on the broader development process.

Let’s take a closer look at this often-overlooked productivity killer.

What are flaky tests?

When all goes according to plan, software tests offer trusted, reliable insights that assure developers that their code works correctly. If a test passes, you know all is well. If it fails, something is amiss and needs to be corrected. Flaky tests throw a wrench into this process. 

Flaky tests are unpredictable, inconsistent tests that sometimes pass and sometimes fail, without any changes in code. This creates confusion, casting doubt on the reliability of the software toolchain altogether and resulting in toil and frustration for developers. 

In time, developers identify which tests are “flaky” and dismiss them. But, this can lead to ignoring real failures, resulting in a lower-quality product plagued by bugs that slipped through the cracks. Ultimately, flaky tests stunt the pace and quality of software delivery, negatively impacting both your company’s product and the software developers that help lay the foundation for its success.

Why you can’t afford to ignore flaky tests

Not only do flaky tests threaten the quality and speed of software delivery, they pose a very real threat to the happiness and satisfaction of software developers. Similar to other bottlenecks in the software development process, flaky tests take developers out of their creative flow and prevent them from doing what they love: creating software. 

Imagine a test passes on one run and fails on the next, with no relevant changes made to the codebase in the interim. This inconsistent behavior can create a fog of confusion, and lead developers down demoralizing rabbit holes to figure out what’s gone wrong. It’s a huge waste of time and energy.

By addressing flaky tests, technology leaders can directly improve the developer experience. Instead of getting tangled up in a web of phantom problems that drain their time and energy, developers are able to spend more time on fulfilling tasks like creating new features or refining existing code. When erratic tests are eliminated, the development process runs much more smoothly, resulting in a more motivated and happier team.

This improves an organization’s ability to attract and retain vital workers, particularly important given today’s shortage of software development talent. Recent research found that less than half (45%) of engineering leaders are confident they will be able to meet their technical hiring targets this year. And this shortage doesn’t seem to be going away any time soon due to the massive amount of new technologies entering the IT world. 

Especially given market turbulence, companies today need every possible advantage to remain competitive, and tackling flaky tests is an important stepping stone on the path to improving developer experience and winning the war for talent. 

How to weed out flaky tests

Prioritizing the correction of flaky tests should be an essential part of any software development strategy. Recognizing that flaky tests will not fix themselves is the first step. From there, set aside proper time to address them. Devote resources to fixing flaky tests in the same way you would for developing new features or fixing bugs. 

Beyond that, it’s important to foster a culture of communication. Encourage developers to offer regular feedback on bottlenecks they encounter, like flaky tests. This will help you to identify and address barriers within the software development process quickly, before they become an even larger issue. 

As developers voice their productivity concerns and feedback, make sure they feel heard. Developers say the most time-consuming thing they do at work besides writing code is waiting on builds and tests, which is a result of the tools they use. Some traditional management practices measure productivity strictly based on the individual rather than taking into account technology challenges. This leaves developers feeling frustrated and distrusting of leadership. Make it clear you don’t consider productivity to be a people problem, but an engineering one, and invest in solutions to overcome process bottlenecks. 

If you’ve considered flaky tests to be an insignificant hiccup, I believe it’s time to make them a priority. By taking flaky tests seriously and consistently taking measures to keep them at bay, software development teams will be positioned to reach full bloom. 

Trisha Gee is lead developer advocate at Gradle.

—

Generative AI Insights provides a venue for technology leaders to explore and discuss the challenges and opportunities of generative artificial intelligence. The selection is wide-ranging, from technology deep dives to case studies to expert opinion, but also subjective, based on our judgment of which topics and treatments will best serve InfoWorld’s technically sophisticated audience. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content.