Following on from the recent blog about some of the key success and failure factors of Agile implementations, I wanted to discuss some of the differences in approach when attempting to tackle new projects in an Agile manner, as opposed to Business as Usual (BAU) work.
When conducting BAU work, the value in following an Agile methodology is really about three things:
- Delivering value to the customer regularly
- Delivering the current highest priority requirement (enhancement or defect fix)
- Delivering changes in requirements or priorities of requirements to the customer without a significant delay
The basic idea being that at the beginning of every sprint (generally a 2 – 4 week cycle) a selection of enhancements and possibly defects in the form of user stories are identified and committed to by the team for delivery at the end of the sprint. Within that sprint, should the priorities change such that waiting for the following sprint isn’t an option, the scope can be changed. This comes at a cost – e.g. the likely non-delivery of other some other committed-to items – but it can be done and allows the customer to see the value of a changing requirement fairly quickly.
The Product Owner will ultimately decide if they want to actually release the items that are ‘Done’ in a sprint to a Production environment but the goal is that they should be able to if they want to.
New Projects are Different
By new projects, I refer to the creation of a product that doesn’t already exist or the complete overhaul of an existing one – architecturally or from a look and feel perspective. I have seen a number of new projects fail by organisations taking the same (usually Scrum) approach they have been using successfully whilst working in BAU mode and applying it directly.
Some of the aspects of Waterfall approaches worked fairly well for new projects. They generally specified distinct phases that allow the activities that are required for a new project. Take a traditional Waterfall phased approach like below:
The Waterfall model suggest that these phases should be distinct such that one only starts when the previous one has finished. These phases still apply regardless of whether you are conducting BAU work or working on a new project – the main difference being that the phases will be much shorter for BAU work as you will generally only be delivering a few items in some form of maintenance release. Whichever mode you are working in, the distinct characteristic of Waterfall is that the analysis for EVERYTHING you are intending to release is conducted before the requirement specification phase begins. Then the requirement specification phase for EVERYTHING you are intending to release is conducted before the design phase begins and so on and so forth.
Now, if you are working in BAU mode with regular maintenance releases, the negative impact of taking this kind of approach is not that significant but let’s consider a new project that is estimated to take about a year to complete. By the time you get to the testing and integration phase a significant amount of time has already passed and in that time, the requirements will almost certainly have changed (how many businesses / customers know exactly what they want a year before they get it?) and only at the point of testing where you actually start to see the software that has been developed do any issues relating to any of the previous phases become truly apparent. By this time, the cost of fixing these issues is likely to be huge and the only way to cope with a change in requirement is to go right back to the beginning of the cycle again.
This is Where Scrum Comes In
In a nutshell, Scrum suggests breaking requirements down in to the smallest deliverable and valuable chunks and only working through these phases for the current highest priority items (or functional areas). Also, it suggests that these things can happen in parallel for a number of different user stories at the same time (not too many!) with the goal of delivering a small chunk of something to a tester at the earliest possible opportunity and thus being able to show any relevant stakeholders what that small chunk actually looks like at the earliest opportunity rather than a scenario where all requirements are developed before delivering to a test environment and all requirements are tested before delivering to some kind of user acceptance testing environment. Only conducting analysis, requirements specification and design on the next priority items means that you don’t waste time conducting these activities for requirements that may well have changed by the time the team is in a position to develop them.
So, I think people understand the value of using an Agile methodology to deliver value to a customer but here’s why the approach needs to be slightly different when working in a BAU mode and when working on a new project.
Let’s talk about BAU first. You have an existing product. You have an existing product backlog containing high level user stories. Each sprint, the Product Owner will select the highest priority items from the backlog, the team will commit to a set of user stories they believes they can deliver by the end of the sprint. The team will work on them and get them to a ‘Done’ state, a short period of regression testing will likely occur (preferably automated but manual if necessary), the Product Owner will review what has been ‘Done’ and decide whether they want to release these ‘Done’ items to Production – which they likely will when working in BAU mode. The analysis and design phases will generally be very short in this kind of scenario because the likelihood when working in BAU mode is that nothing hugely significant will be changing behind the scenes in order to deliver a set of user stories (if it was – you’d probably be calling it a new project!).
Organisations often get comfortable with this approach and when it comes to working on a new project, a Scrum Team is formed and the team attempts to start delivering value to the customer from Sprint 1 without taking in to account the following considerations – which are the typically some of the characteristics of a new project:
- You have no existing product – therefore
- You have no existing architecture for the product
- You have no existing product backlog
- You have no existing regression scripts (automated or manual)
- You have no existing unit tests
- You have no existing documentation
Whilst some of the analysis, requirements specification and design (fleshing out of user stories) can still be done on a sprint by sprint basis – eliminating the Waterfall issue of conducting these activities for all requirements for the project before any development and testing can begin – there is still a need on a new project to conduct these distinct phases before any development can begin. During these phases, you would be doing the following types of activities:
- Gathering high level requirements and forming a product backlog
- Considering the architectural requirements for the project as a whole
- Considering what needs to be in place in order for the project to start
- Creating a / some proof of concept(s) to determine whether the project is doable
Likewise, whilst testing user stories can be conducted on a sprint by sprint basis and reviews of what have been delivered can be conducted with the Product Owner / users on a sprint by sprint basis, there is still a distinct phase of integration testing, user acceptance testing and preparation for a production release that simply needs to be done when you are actually intending to release a ‘Ready’ product to Production. Depending on the nature of what you are building, this may only be at the end of the project or there may be distinct phases throughout the project where there is value to the customer in delivering what has been ‘Done’ so far.
But Scrum says I should be able to release at the end of every Sprint?
Yes, it does but that doesn’t mean that you should. In a new project scenario where you are creating a product from scratch, whilst you will hopefully have some stable software in a working state at the end of each sprint, until enough user stories have been completed whereby delivering what has been built will actually deliver any value to the customer, it would make no sense to do so. You may consider deploying to some kind of integration or staging environment but again, it only makes sense to do so if there is a purpose of doing so.
There are overheads involved in releasing to Production such as integration testing, full regression testing, user acceptance testing, release preparation and documentation, knowledge transfer to operations or support teams etc. These activities obviously impact the amount of development and testing that can be conducted on user stories so until there is value in releasing to production for a new project, there is little value in doing so.
If we are not releasing to Production every Sprint, why use Scrum at all?
Well, the value in using Scrum for a new project is less about quick delivery of a finished product to Production and more about mitigating risk and coping with changing requirements. Using Scrum for new projects when it comes to fleshing out, developing and testing user stories and reviewing these with the Product Owner allows early visibility of the team going off track, early visibility of what the product is actually going to look like and allows the user to change their mind about the requirements without significant impact to the project.
In summary, whilst pure Scrum works well for BAU work, my experiences have led me to the conclusion that using Scrum plus some of the more traditional techniques suggested in the Waterfall model for new projects has been the most successful approach and the one that has most often led to the outcome that all customers want where the required scope is delivered on time and on budget (or even under!).