A brief history of the agile methodology

Every technology organization these days seems to practice some version of agile methodology. Or at least they believe they do. Whether you are new to software development or you started decades ago, your work today is at least influenced by agile methods.

But what is agile, and how do developers and organizations incorporate agile methodologies? This article is a brief history of agile development and how it differs from the classic waterfall methodology. I’ll discuss the differences between agile and waterfall methods in practice, and explain why agile is so much better suited to how developers and teams actually work, especially in today’s development environments.

Before agile: The waterfall methodology

Old hands like me remember when the waterfall methodology was the gold standard for software development. Using the waterfall method required a ton of documentation up front, before any coding started. Usually, the process started with a business analyst writing a business requirements document that captured what the business needed from the application. These documents were long and detailed, containing everything from overall strategy to comprehensive functional specifications and visual user interface designs.

Technologists used the business requirements document to develop a technical requirements document. This document defined the application’s architecture, data structures, object-oriented functional designs, user interfaces, and other nonfunctional requirements.

Once the business and technical requirements documents were complete, developers would kick off coding, then integration, and finally testing. All of this had to be done before an application was deemed production-ready. The whole process could easily take a couple of years.

The waterfall methodology in practice

The documentation used for the waterfall methodology was called “the spec,” and developers were expected to know it just as well as its authors did. You could be chastised for failing to properly implement a key detail, say, outlined on page 77 of a 200-page document.

Software development tools also required specialized training, and there weren’t anywhere near as many tools to choose from. We developed all the low-level stuff ourselves, such as opening database connections and multithreading our data processing.

For even basic applications, teams were large and communication tools were limited. Our technical specifications aligned us, and we leveraged them like the Bible. If a requirement changed, we’d put the business leaders through a long review-and-sign-off process. Communicating changes across the team and fixing code were expensive procedures.

Because software was developed based on the technical architecture, lower-level artifacts were developed first and dependent artifacts came next. Tasks were assigned by skill, and it was common for database engineers to construct tables and other database artifacts first. Application developers coded the functionality and business logic next, and the user interface was overlaid last. It took months before anyone saw the application working. By then, stakeholders were usually getting antsy—and often smarter about what they really wanted. No wonder implementing changes was so expensive!

In the end, not everything you put in front of users worked as expected. Sometimes, they wouldn’t use a feature at all. Other times, a capability was widely successful but required re-engineering to support scalability and performance. In the waterfall world, you only learned these things after the software was deployed, following a long development cycle.

Pros and cons of the waterfall methodology

Invented in 1970, the waterfall methodology was revolutionary because it brought discipline to software development and ensured there was a clear spec to follow. It was based on the waterfall manufacturing method derived from Henry Ford’s 1913 assembly line innovations, which provided certainty about each step in the production process. The waterfall method was intended to ensure that the final product matched what was specified in the first place.

When software teams started adopting the waterfall methodology, computing systems and their applications were typically complex and monolithic, requiring discipline and clear outcomes to deliver. Requirements also changed slowly compared to today, so large-scale efforts were less problematic. In fact, systems were built assuming they would not change but would be perpetual battleships. Multiyear timeframes were common not only in software development but also in manufacturing and other enterprise activities. But waterfall’s rigidity became its downfall as we entered the internet era, and speed and flexibility were more prized.

The pivot to agile methods

Software development started to change when developers began working on internet applications. A lot of the early work was done at startups where teams were smaller, were colocated, and often did not have traditional computer science backgrounds. There were financial and competitive pressures to bring websites, applications, and new capabilities to market faster. Development tools and platforms changed rapidly in response.

This led many of us working in startups to question the waterfall methodology and look for ways to be more efficient. We couldn’t afford to do all the detailed documentation up front, and we needed a more iterative and collaborative process. We still debated changes to the requirements, but we were more open to experimentation and adapting our software based on user feedback. Our organizations were less structured, and our applications were less complex than enterprise legacy systems, so we were more open to building versus buying applications. More importantly, we were trying to grow businesses, so when users told us something wasn’t working, we usually listened to them.

Having the skills and abilities to innovate became strategically important. You could raise all the money you wanted, but you couldn’t attract talented software developers, able to work with rapidly changing internet technologies, and then force them to follow “the spec.” Developers rejected project managers who led with end-to-end schedules describing what we should develop, when applications should ship, and sometimes even how to structure the code. We were terrible at hitting the three-month and six-month schedules that our project managers drafted and unceasingly updated.

Instead, we started telling them how internet applications needed to be engineered, and we delivered results on a schedule that we drew up iteratively. It turns out we weren’t that bad at delivering what we said we would when we committed to it in one-week to four-week intervals.

In 2001, a group of experienced software developers realized that they were collectively practicing software development differently from the classic waterfall methodology. Not all of them were in startups, either. This group—which included technology luminaries Kent Beck, Martin Fowler, Ron Jeffries, Ken Schwaber, and Jeff Sutherland—came up with the Agile Manifesto that documented their shared beliefs about how a modern software development process should operate. They stressed collaboration over documentation, self-organization rather than rigid management practices, and the ability to manage constant change rather than being locked into a rigid waterfall development process.

From those principles the agile methodology for software development was born.

Why agile development delivers better software

When you take the aggregate of agile principles, implement them in an agile framework, leverage collaboration tools, and adopt agile development practices, you usually get applications that are better quality and faster to develop. You also get better technical methods, aka hygiene.

The main reason is that agile is designed for flexibility and adaptability. You don’t need to define all the answers up front, as you do in the waterfall method. Instead, you break the problem into digestible components that you then develop and test with users. If something isn’t working well or as expected, or if the effort reveals something that you hadn’t considered, you can adapt the effort and get back on track quickly—or even change tracks if that’s what’s needed. Agile lets each team member contribute to the solution, and it requires that each member assumes personal responsibility for their work.

Agile principles, frameworks, and practices are designed for today’s operating conditions. Agile typically prioritizes iterative development and leveraging feedback to improve the application and the development process. Both iteration and feedback are well suited to today’s world of operating smarter and faster.

Agile development also encourages ongoing improvement. Imagine if Microsoft ended Windows development after version 3.1, or Google stopped improving its search algorithms in 2002. Software is in constant need of being updated, supported, and enhanced; agile methodology establishes both a mindset and process for that continuous improvement.

Finally, agile development leads to better software because people on agile teams are typically more productive and happier. Engineers have a say in how much work they take on, and they are proud to show their results. Product owners like seeing their vision expressed in software sooner and being able to change priorities based on the latest insights. Users like getting software that does what they actually need it to do.

Today, enterprises need a high level of software competency to deliver exceptional digital experiences in a hypercompetitive world. And they need to attract and keep great talent to build great software. Agile development helps enterprises do both.