A Brief History of Agile

I’ve written about The Agile Manifesto here (values) and here (principles), with this post I will dig even more in how Agile came to be.

The first notable thing to mention is that Agile didn’t start with… well Agile. Many of the frameworks that sit under the Agile umbrella are more than two decades old, and the term “agile” to define a new way of developing software emerged from existing frameworks (some as experiments, some long-lived already) that needed a new definition, or a new category for them to fit together.

Overview

I created a visual timeframe of all facts, frameworks, practices that preceded and influenced the Agile creation and evolution to what it is today:

 

Before The Agile Manifesto

1930 – 1990

In the ‘30’s, Walter Shewhart (physicist and statistician with Bell Labs), looked into process and products improvement with the PDSA cycles (Plan – Do – Study – Act).

After the World War II, Professor Edward W. Deming, famous quality guru, heavily promoted the PDSA cycle, and used it intensely in Japan. Toyota brought Deming to train and work with their leadership and engineers, and this set the basis for the birth of the Toyota Production System around 1948, known presently as “lean thinking“.

The 1950s, the X-15 hypersonic jet was built using iterative and incremental development, and the practice was considered a key for the success of the jet.

In 1986, Hirotaka Takeuchi  & Ikujiro Nonaka wrote an article in HBR – The New New Product Development Game –  which presented a “rugby approach” to product development: “where a team tries to go the whole distance as a unit, passing the ball back and forth.”, with an emphasis on speed and flexibility. 

1990 – 2001

In 1992, Allistair Cockburn created the Crystal framework. Cockburn probably set the stage for the Agile movement, with this set of frameworks that are characterised by simplicity, teamwork, communication and continuous improvement.

In 1993, Jeff Sutherland used rapid application development, object-oriented design, PDSA cycles, and skunkwork to create a new product for Easel Corporation, and dived into learning and research on how to make a team reach maximum productivity. Scrum was born, and presented to the public in 1995, by Sutherland and Ken Schwaber.

Also in 1993, William F. Opdyke presented the concept of refactoring to the ACM conference on Computer science, in Indianapolis Indiana. “Refactoring is the process of changing a software system in such a way that it does not alter the external behaviour of the code yet improves its internal structure” – Martin Fowler, in “Refactoring. Improving the Design of Existing Code”.

In 1994 Dynamic System Development Method (DSDM) was created by a Consortium of software engineers, most notable and influential in the development of the framework being Jennifer Stapleton and Arie van Bennekum (the latter being one of the signatories of the Manifesto). DSDM is considered the predecessor of agile project management; it emphasises backlog prioritisation (the MoSCoW technique), short iterations, constant user feedback, flexible scope (while time and budget are fixed), all essential elements for present product management.

In 1995, Scrum was made public and software development in pairs was written about independently – but at the same time – by two engineers: Jim Coplien wrote “Pattern Languages in Program Design” (1995), one of the organisational patterns being “developing in pairs“; in “Constantine on Peopleware“, Larry Constantine reports about seeing “Dynamic duos” writing code faster in one of his projects. Both authors state that programmers working together to write the same piece of code is the sure path to more efficiency, faster learning, and higher code quality.

In 1997, Feature Driven Development was created by Jeff De Luca. FDD is a lightweight software development method, it has short cycles (2 weeks) during which the design, coding, testing happens (entire software life cycle); in the early phase a “model” of the product is created, with an accompanying feature list, and the planning is done by feature; reporting to management is made for feature delivered or in progress. If you want to learn more about FDD, check out Jeff De Luca’s book, The Story Behind Featured-Driven Development (with Kevin Aguanno).

1999 was a very productive year for the software development industry, as quite a few notable and influential events took place:

1. Extreme Programming was brought to public for the first time. XP is a framework created by Kent Beck, one of the Agile Manifesto signatories, while working at Chrysler; he published Extreme Programming Explained: Embrace Change (The XP Series) in 1999 bringing XP to the general public. XP has probably some of the simplest rules among the current popular agile frameworks: short – 1 week – iterations, no specific roles within the team (anyone in the team can take whatever role is needed for that sprint), the Customer sits and works with the team daily, delivery of  running software each iteration, but also frequent releases to the end-user.

The most important aspect of XP is the practices it reinforces and makes part of the framework: Whole Team, Planning Game, Small Releases, Customer Tests, Simple Design, Pair Programming, Test-Driven Development, Design Improvement, Continuous Integration, Collective Code Ownership, Coding Standard, Metaphor, Sustainable Pace. These practices are an enormous contribution to software development  and are being used across all other Agile frameworks.

“An additional, and equally important, contribution of XP is the focus on practice excellence. The method prescribes a small number of absolutely essential practices and encourages teams to perform those practices as good as they possibly can, almost to the extreme. This is where the name comes from.” excerpt from Agile Alliance’s article on XP.

Notable supporters and promoters of XP are Ward Cunningham (creator of wiki), Ron Jeffries (worked with Kent Beck on the Chrysler project and is a considered a co-founder of XP), and Martin Fowler (author of Refactoring: Improving the Design of Existing Code), all signatories of the Agile Manifesto.

2. Adaptive development was developed by Jim Highsmith, one of the Agile Manifesto signatories, in the book Adaptive Software Development: A Collaborative Approach to Managing Complex Systems (released in December, 1999). Adaptive development focuses on building software rapidly and releasing often; the framework doesn’t include specific steps to take, but focuses on collaboration – development team, customer, testers, self-managed teams, learning, using a dynamic life-cycle. The AD life-cycle is speculate (initiation and planning), collaborate (feature development by team through collaboration), and learn (inspect, adapt, feedback), replacing the traditional plan – design – build – test method.

3. Pragmatic Programmer: the concept of the Pragmatic Programmer was introduced to the wold by Andy Hunt and Dave Thomas, in the book The Pragmatic Programmer: From Journeyman to Master, released also in 1999. The book is a guide for software developers ever since, and it’s still as valid as ever. It gives practical advice on soft skills (from communication to collaboration), and also hard skills (technical skills, refactoring, etc). To this day, it’s considered a must-read for software developers of all levels, and became also a textbook for universities.

2001: The Agile Manifesto

“On February 11-13, 2001, at The Lodge at Snowbird ski resort in the Wasatch mountains of Utah, seventeen people met to talk, ski, relax, and try to find common ground—and of course, to eat. What emerged was the Agile ‘Software Development’ Manifesto. Representatives from Extreme Programming, SCRUM, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming, and others sympathetic to the need for an alternative to documentation driven, heavyweight software development processes convened.“, from the history of the Agile Manifesto.

All the above practices, frameworks, and learning set the stage for the creation of the Agile Manifesto. The manifesto signatories came up with the word “Agile” to encompass the traits exhibited by the above frameworks and practices after much discussion, but in full agreement.

“At the core, I believe Agile Methodologists are really about “mushy” stuff—about delivering good products to customers by operating in an environment that does more than talk about “people as our most important asset” but actually “acts” as if people were the most important, and lose the word “asset”. So in the final analysis, the meteoric rise of interest in—and sometimes tremendous criticism of—Agile Methodologies is about the mushy stuff of values and culture.”, from the history of the Agile Manifesto.

I wrote detailed posts on the Agile Manifesto, its signatories, the Agile values and principles, with original images from the singing of the Manifesto (courtesy of Allistair Cockburn), so if you want to dig deeper into the manifesto check it out.

 After The Agile Manifesto 2001+

The frameworks described above are all part of the Agile movement. The Agile movement didn’t stop here, the following years brought new concepts, popularised or solidified old ones, and won over more and more enthusiasts.

In 2002, Test-Driven Development (TDD, introduced previously as a practice in XP) became more popular with the release of Test Drive Development: By Example, by Kent Beck, creator of XP. TDD is a programming style that has three activities: coding, testing (unit tests), and design (refactoring); the simplest explanation is this: you first write a test (before you even begin to work on the feature), then you run the tests (which will fail, as you didn’t build anything beforehand), then you write the minimum amount of code to make the test pass, then refactor the code (according to the simplicity criteria from XP); repeat.

Why is TDD essential for the Agile movement? TDD brings the idea of testing inside the work of the developer, which is a big step towards agility (previously testing was done by a separate team, or separate roles within the team). So testing becomes a regular activity for a developer and currently it is part of the job of software engineers.

Find below a brief history of the role of testing in software development (the original is here):

• 1976: publication of “Software Reliability“ by Glenford Myers, which states as an “axiom” that “a developer should never test their own code” (Dark Ages of Developer Testing)
• 1990: testing discipline dominated by “black box” techniques, in particular in the form of “capture and replay” testing tools
• 1991: independent creation of a testing framework at Taligent with striking similarities to SUnit (source)
• 1994: Kent Beck writes the SUnit testing framework for Smalltalk (source)
• 1998: article on Extreme Programming mentions that “we usually write the test first” (source)
• 1998 to 2002: “Test First” is elaborated into “Test Driven“, in particular on the C2.com Wiki
• 2000: Mock Objects are among the novel techniques developed during that period (source)
• 2003: publication of “Test Driven Development: By Example” by Kent Beck.

2003 marks the publication of another important step in the Agile movement – Lean Software Development: An Agile Toolkit, by Mary and Tom Poppendieck. Remember that lean manufacturing was around for over fifty years and there is where the Poppendiecks got their inspiration from (they used to work with lean at 3M). Lean Software Development (LSD) applies principles of lean manufacturing to software development; the 7 Principles of LSD are (1) Eliminate waste; (2) Amplify learning; (3) Decide as late as possible; (4) Deliver as fast as possible; (5) Empower the team; (6) Build integrity in; (7) See the whole.

In 2005 Large Scale Scrum (LeSS) was made public by Bas Vodde and Craig Larman through the publication of Large Scale Scrum. More With LeSS. LeSS builds on Scrum (one of their principles after all is that LeSs is Scrum), adding on top engineering practices (CI, CD, clean code, test automation, TDD), some borrowed from XP, and lean concepts (go see, continuous improvement, queueing theory, lean thinking etc.). I am a big fan of lean and I also like the structure of Scrum, so I consider LeSS one of the Agile scaling frameworks that really covers the gaps that Scrum explicitly leaves open (e.g. collaboration between teams, feature teams, etc.).

In 2007, David J. Anderson (professor and author) experimented with a kanban pull system (practice from lean manufacturing) at Corbis, after successful results at Microsoft. From his work and experiments, he developed the Kanban Method, which is defined by workflow visualisation, work item types, cadence, classes of services, specific management reporting, and operations reviews. He published the accompanying book in 2010: Kanban: Successful Evolutionary Change for Your Technology Busines.

In 2009 DevOps was popularised by the work of Patrick Dubois and Andrew “Clay” Shafer. DevOps is around for about a decade. Patrick Dubois met Andrew Schafer in 2008, at a discussion about Agile infrastructures; same year they formed a group of Agile System Administration on Google, while the first DevOps conference happened in USA in 2010. There is an ongoing controversy around DevOps, but the intention of the concept is to connect the operations part of software building with the development part, with final objective to increase product delivery speed. The predominant opinion in the software industry is that the operations should be part of the development’s team work, just as other tasks needed to deliver the product.

Woody Zuill made one step beyond pair programming and brought to the world mob programming in 2012. Mob programming is a programming method where the entire team works on the same code, at the same time, in the same space, at the same computer. It’s a technique that involves the whole team into collaborating to write code. The team works together to deliver everything the team needs to build the product (building requirements, writing user stories, etc.). It is an essential step forward for the Agile movement, as there’s face-to-face communication, continuous code review, whole team involvement and collaboration and it is proven to be extremely efficient and motivating for the teams.

In the same year – 2012 – Disciplined Agile Delivery (DAD) was created by Scott Ambler. It is a self-proclaimed “light-weight guidance to help organizations streamline their processes in a context-sensitive manner, providing a solid foundation for business agility”, according to the official website. DAD is a hybrid that takes methods from other frameworks, like Scrum, XP, Kanban, etc. with the intention to customise your own agile approach.

What I didn’t include in here:

• Enterprise Scrum: the framework was developed by Mike Beedle and made public around 2016. The purpose of the framework is to allow companies of all sizes to be managed like startups and targets business agility (Beedle compares it with models such as holacracy, flat organisations or Management 3.0). I didn’t find much information on it (nor experiments made for a longer period of time) to include it in.
• Scaled Agile Framework (SAFe): I didn’t include SAFe in this post, as there’s a lot of controversy around it not being Agile (I tend to agree). If you’d like to read some of the criticism, check articles from Ken Schwaber, Ron Jeffries, and also Martin Cagan (thought leader for technology product management).
• “The Spotify model“: I didn’t add the very popular Spotify model for a different reasoning altogether: the model works great for Spotify, but it’s not a framework per se. It’s a great example of using different frameworks to reach agile transformation, so I would definitely recommend you look into it.
• There are a few more concepts out there I didn’t tackle, an overview of the Agile landscape was made by Deloitte, but such connections and relations were severely criticised by the Agile community.
• You can also check the Agile Alliance’s map of agile practices here.

Conclusion

To conclude, I would like to point towards a post written by the late Mike Beedle on the Agile movement:

“The Agile movement …is NOT very Agile!!”

[…] Is there a way to change this and improve?

Yes, it’s an obvious answer …let’s make the Agile Movement more AGILE!!!

[…] We, the Agilistas, …. are NOT that Agile!

 

IF you understand the problems stated above, and since you made it reading this far, that it is your moral obligation now that you understand these problems to help in whatever shape or form, and in whatever role you are involved with, to IMPROVE the Agile Movement as much as you can!!

I’ll take that as a “YES”. Thank you in advance. I plead to do the same!

-Mike Beedle

Other resources:

• The Secret History of Agile Innovation, by Darrell K. Rigby, Jeff Sutherland, Hirotaka Takeuchi
• Walter A Shewhart, 1924, and the Hawthorne factory, by M Best and D Neuhauser
• Iterative and Incremental Development: A Brief History, by Craig Larman and Victor R. Vasili
• The New Product Development Game, by Hirotaka Takeuchi  & Ikujiro Nonaka
• a very good post on the history of Agile – which was a lot of help, especially for the 1992 – 1998 period, can be found here
• Kanban: Successful Evolutionary Change for Your Technology Business by David J. Anderson