Development methodologies

Content:

  1. Introduction
  2. Waterfall
  3. Iterative
    1. Unified Process (UP)
    2. Agile
      1. True Agile
      2. Boxed Agile
      3. Principles
      4. eXtreme Programming (XP)
      5. Scrum
      6. Kanban
    3. Hybrids of UP and Agile
      1. Scaled Agile
      2. Disciplined Agile
      3. Large Scale Scrum
  4. Comparison

Introduction:

Development methodologies are obviously an important part of software development. It is also an area where many developers have strong feelings but not necessarily much evidence about what methodoly is best.

This area is not an exact science, so even though I will try and describe many alternatives with pro's and con's, then this text will also have some subjective evaluations by me. And be aware that I am not an expert in development methodolgies or project management, so my subjective evaluations should not be taken as the one and only truth. But hopefully they will make you think about the problems, realize that this is a very complex topic and make your own conclusions.

The following text will show 4 phases of development:

Requirements
Finding out what the solution needs to do. Many possible artifacts are possible including traditional requirements documenets, use cases, user stories etc..
Design
Deciding on how the the requirements can be met. This goes from high level architecture to low level software design. Possible artifacts include UML diagrams, design documents, pseudo code and in case of TDD also unit test code.
Implementation
Producing the solution: write the code, install infrastructure etc.. Writing the code include writing unit tests if that was not done as part of design. The artifacts are completed software and infrastructure.
Test
Test whether the solution meet requirements. Many different tests must be done: functiocnal test, performance test etc.. The artifacts are bug reports and test reports.

Waterfall:

The waterfall model simply do each of the 4 phases of development sequentially and each phase is completed before moving on to the next phase.

This model is very old. It was described by Herbert Benington in 1956 and Winston Royce in 1970. But it is still widely used.

Requirements are usually documented in a traditional and often very detailed requirements document.

It sort of has a bad reputation. Many developers especially those with little or no experience in larger technical complex projects consider it a major reason for project failures.

An obvious question is what happens if a new requirement show up. That question has two answers.

When someone trying to sell a training course or a book about iterative/agile methodologies describe it, then it is handled by starting all over:

Waterfall model

In the real world it is handled by Change Requests (CR). A new or changed requirement show up. It get documented in a CR. It gets discussed by those managing CR's (usually some sort of project steering commitee). If it get approved, then it get done in a separate stream also following a waterfall model.

Waterfall with CR model

It is common with many CR's in larger projects. In that case it is normal to group CR's together in a few CR batches and treat each CR batch as separate waterfall instead of treating each CR individually.

So it is actually possible to handle some changes in requirements using waterfall model.

But the CR mechanism does have its limitations. If the number of CR's become too big and especially if the CR's start overlapping each other in functionality, then the project becomes a mess.

Evaluation:

Criteria Score Explanation
Suitability small-medium sized projects B Detailed documentation is not necessary for smaller projects
Suitability large-very large projects A Detailed documentation scale well to large teams
Suitability inhouse projects A The stakeholders can get a functional description + schedule for delivery + cost
Suitability customer projects A The customer can get a functional description + schedule for delivery + cost
Suitability fixed requirements A When there a few CR's things go smoothly
Suitability changing requirements C CR mechanism can handle some changes to requirements, but there are limits (see above)
Suitability technical simple systems B Upfront design is not necessary for techncially simple projects
Suitability technical complex system B Upfront design is required for technically complex projects, but problems in the design may be found very late
Suitability for good teams B The process does usually not create much enthusiasm
Suitability for mediocre teams A The process is easy to understand for the team

Iterative:

Iterative models are a group of very different models just sharing the characteristic that the work is done in iterations.

Unified Process (UP):

Unified Process (UP) was defined by Ivar Jacobson, Grady Booch and James Rumbaugh in 1999.

It split up the development in multiple iterations. Each iteration contain all 4 phases of development.

A special characteristica of UP is that iterations are not necessarily complete. It is possible for a given feature to do requirements in iteration N, design in iteration N+1, implementation in iteration N+2 and test in iteration N+3. As a consequence early iterations contain more requiremenst and design while late iterations contain more implementation and test.

Usage of UP usually involve lots of architecture work and UML disgrams.

The distribution of functionality on iterations is not done based on customer priority but based on technical risk. High risk functionality get done early to allow for sufficient time for redesign in case of problems with design.

Requirements are documented in the form of use cases.

Iterations are typically 4-10 weeks.

The usage of UP sort of got tied to the use of a commercial product to manage the process: Rational Unified Process (RUP). The company Rational got acquired by IBM.

UP/RUP was rather popular back in the early 00's, but has since almost disappeared - replaced by agile and hybrid methodologies.

Iterations are grouped in 4 projects phases:

UP model

Evaluation:

Criteria Score Explanation
Suitability small-medium sized projects B Detailed process is not necessary for smaller projects
Suitability large-very large projects A Detailed process scale well to large teams
Suitability inhouse projects A Functional description + schedule for delivery + cost usually stabilize early enough to make stakeholders happy
Suitability customer projects B Functional description + schedule for delivery + cost not fully reliable upfront
Suitability fixed requirements A Works fine
Suitability changing requirements B Easy to change requirements for later iterations, but too large changes in scope can create chaos
Suitability technical simple systems B The amount of architecture and design are not necessary for techncially simple projects
Suitability technical complex system A The amount of architecture and design plus the early test of high risk functionality is perfect for techncial complex projects
Suitability for good teams A The process makes sense for the team
Suitability for mediocre teams C The process is too heavy and too complex for the team

Agile:

Agile methodologies are a relative broad family of methodologies that share some basic characteristics:

Many agile principles are old, but the it was defined using the term in 2001 by a group of 17 experienced software people including Alistair Cockburn, Martin Fowler, Kent Beck, Robert Martin and Jim Highsmith.

Common characteristics are:

Agile methods can actually be used in two different ways. Which I will label with the terms:

The two concepts are widely known, but the terms I use are not widely used.

True Agile

What I call "true agile" is where requirements is done for each iteration and there is no defined end goal. Stakeholder provide requirements for iteration, team implements requirements for iteration and that continue until stakeholders consider the solution complete.

True Agile model

Evaluation:

Criteria Score Explanation
Suitability small-medium sized projects A The lightweight process is perfect for these projects
Suitability large-very large projects C The informal processes does not scale well
Suitability inhouse projects B Stakeholders like to have functional description + schedule for delivery + cost
Suitability customer projects C Customers insist on having a functional description + schedule for delivery + cost
Suitability fixed requirements A Works fine
Suitability changing requirements A Changing requirements have practically no impact as requirements are done as late as possible
Suitability technical simple systems A Works fine
Suitability technical complex system C Not knowing all requirements making it diffucult to create a detailed architecture can result in disaster
Suitability for good teams A Thrive with the delegation of responsibility
Suitability for mediocre teams B Things will be forgotten creating unnecessary problems

Boxed Agile

What I call "boxed agile" is where requirements are done up front and the end goal is defined. Each iteration implements a part of the requirements.

It is important to note that even though requiremenst are done upfront similar to waterfall, then the requirmenets are typical much less detailed for boxed agile than for waterfall.

Many believers in agile consider this boxed agile model to be a form for waterfall made to look like agile. But non the less it is widely used.

Boxed Agile model

Evaluation:

Criteria Score Explanation
Suitability small-medium sized projects A The lightweight process is perfect for these projects
Suitability large-very large projects C The informal processes does not scale well
Suitability inhouse projects A The stakeholders can get a functional description + schedule for delivery + cost
Suitability customer projects A The customer can get a functional description + schedule for delivery + cost
Suitability fixed requirements A Works fine
Suitability changing requirements B Requirements are done upfront, but changing them is not so bad as the requirements are high level
Suitability technical simple systems A Works fine
Suitability technical complex system B Architecture is usually rather high level so problems can arise
Suitability for good teams A Thrive with the delegation of responsibility
Suitability for mediocre teams B Things will be forgotten creating unnecessary problems

Manifesto & Principles

The original manifesto:

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

That is, while there is value in the items on the right, we value the items on the left more.

The original 12 agile principles:

  1. Customer satisfaction by early and continuous delivery of valuable software
  2. Welcome changing requirements, even in late development
  3. Working software is delivered frequently (weeks rather than months)
  4. Close, daily cooperation between business people and developers
  5. Projects are built around motivated individuals, who should be trusted
  6. Face-to-face conversation is the best form of communication (co-location)
  7. Working software is the primary measure of progress
  8. Sustainable development, able to maintain a constant pace
  9. Continuous attention to technical excellence and good design
  10. Simplicity—the art of maximizing the amount of work not done—is essential
  11. Best architectures, requirements, and designs emerge from self-organizing teams
  12. Regularly, the team reflects on how to become more effective, and adjusts accordingly

eXtreme Programming (XP)

eXtreme Programming (XP) was defined by Ken Beck in 1999.

Requirements are done in the form of user stories.

XP does not use any upfront design. In case of doubt a POC called a Spike can be done.

Key characteristics of development include:

Iterations are typical 2 weeks.

XP was very popular around 2000-2002, but is very rare today.

Scrum

The history of Scrum goes back to 1986 where Hirotaka Takeuchi and Ikujiro Nonaka described the process for physical product development. It was defined around 1995 for software by Ken Schwaber and Jeff Sutherland.

Scrum defines 3 primary roles:

Product owner
Stakeholder/customer voice
Development team
Team of 3-9 developers implementing solution
Scrum master
Process facilitator

Iterations called sprints are typical 2 weeks.

Requiremenst get into the requirement backlog and each iteration/sprint take a number of those.

A key element is the daily 15 minutes standing meeting called scrum where status is presented.

Scrum is very widely used today. In some circles agile and Scrum has almost become synonyms.

Kanban

Kanban history goes back to mid 1980's where Taichi Ohno developed it for manufacturing as part of lean/JIT manufacturing. In the late 2000's it became extended to software development. Kanban in some ways build on Scrum.

Kanban visualize workflow using a Kanban Board. For software development the columns can be the traditional:

Kanban key principles are:

Kanban had become used a lot in recent years.

Hybrids of UP and Agile

Experience with agile for small projects has been very good.

But agile does not scale.

Many agile proponents may deny that, but some of the items in the agile manifest and principles simply work better for small teams than for large teams.

Specifically:

It is true that developer A talking to developer B is more effective that written documentation.

But in the real world it works like:

Team size Location Direct communication
10 same room great
100 multiple rooms same building bad
1000 multiple buildings impossible

So as a result several methodologies that are hybrids of agile and other methodologies especially UP has been created to try and handle this problem as good as possible.

They all try to keep as much of the good things from agile while adding more structure to work for larger teams.

The hybrid methodologies are usually closer to "boxed agile" than "true agile".

Evaluation (assuming boxed agile style):

Criteria Score Explanation
Suitability small-medium sized projects B Too much process for these projects
Suitability large-very large projects A This is the problem they solve
Suitability inhouse projects A The stakeholders can get a functional description + schedule for delivery + cost
Suitability customer projects A The customer can get a functional description + schedule for delivery + cost
Suitability fixed requirements A Works fine
Suitability changing requirements B Requirements are done upfront, but changing them is not so bad as the requirements are high level
Suitability technical simple systems A Works fine
Suitability technical complex system B Architecture is usually rather high level so problems can arise
Suitability for good teams B Too much democracy and too little meritocracy
Suitability for mediocre teams B Things will be forgotten creating unnecessary problems

Scaled Agile

SAFe (Scaled Agile Framework) was described by Dean Leffingwill in 2011.

SAFe comes in two flavors:

3 level SAFe consist of:

  1. Team of 5-10 people working as an XP team doing 2 week sprints
  2. Program of 5-10 teams coordinates teams and own backlog doing program increments of 3-5 development sprints + 1 planning sprint
  3. Portfolio of multiple programs managing budgets and using Kanban techniques for cross-program initaitives

4 level SAFe consist of:

  1. Team of 5-10 people working as an XP team doing 2 week sprints
  2. Program of 5-10 teams coordinates teams and own backlog doing program increments of 3-5 development sprints + 1 planning sprint
  3. Stream of multiple programs providing coordination focusing on delivering value fast
  4. Portfolio of multiple streams managing budgets and using Kanban techniques for cross-program initaitives

Critics consider it more UP than agile.

Disciplined Agile

DAD (Disciplined Agile Delivery) is created by Scott Ambler while working for Rational/IBM. It started as AUP (Agile UP) in 2006 and ended up in its current form as DAD in 2012.

It is a more discplined approach to agile suitable for larger projects.

DAD split project in 3 phases (UP has 4 phases as described above):

Inception is focused on planning and architecture.

Construction is focused on development including test.

Transition is focused on final test and deployment.

DAD also like UP prioritize doing critical complex development early to reduce risk.

Again critics consider it more UP than agile.

Large Scale Scrum

LeSS (Large Scale Scrum) was defined by Craig Larman and Bas Vodde in 2013.

LeSS has two levels:

The teams in LeSS are simply Scrum teams. And the structure on top tries to follow Scrum principles. Each teams sprints are synchronized.

Smaller LeSS:

Huge LeSS:

LeSS is less UP than the previous two.

Comparison:

Comparison of scores:

Criteria Waterfall UP True Agile Boxed Agile Hybrid
Suitability small-medium sized projects B B A A B
Suitability large-very large projects A A C C A
Suitability inhouse projects A A B A A
Suitability customer projects A B C A A
Suitability fixed requirements A A A A A
Suitability changing requirements C B A B B
Suitability technical simple systems B B A A A
Suitability technical complex system B A C B B
Suitability for good teams B A A A B
Suitability for mediocre teams A C B B B

Article history:

Version Date Description
1.0 October 1st 2017 Initial version

Other articles:

See list of all articles here

Comments:

Please send comments to Arne Vajhøj