I was first introduced to Agile methodologies about 8 years ago, specifically the Scrum framework. Like many people, I was sceptical at first. I had worked in environments using waterfall methods, some very successfully for many years and the idea of throwing away heavily detailed requirements documents in favour of lighter weight user stories that were ‘fleshed out’ by the team via discussions seemed bizarre to me at first. It felt like trying to fix something that wasn’t really broken.
After a few months, however, as the team and the organisation that first introduced me to Agile became more mature and confident in its implementation, I really got to see the benefits that Agile can bring when done well:
- How much quicker you can respond to a customer’s needs
- How much better you can cope with changing scope
- How much easier it can be to mitigate risk
- How much earlier you can gain visibility on progress
Since my initial introduction to Agile, I have had many opportunities to experience Agile done well and Agile done not so well and to understand the factors that influence its success and its failures. This blog post aims to share some of my observations of when Agile works well, when it doesn’t and to discuss some of the different approaches that I have seen work best in different circumstances.
My first experience of Agile has turned out to be the most successful. There were a number of things about the organisation I was working for at the time that made it an ideal candidate for using Scrum:
- The organisation had a number of existing products that required fairly regular, small enhancements
- Each product had a clearly defined group of people on the business side that were ultimately responsible for the product
- IT Teams were co-located and in almost all cases, they were also co-located with the business
- The organisation at a Senior Management and board level was open to change
- Existing products were largely in Business as Usual (BAU) mode
It certainly isn’t the case that Scrum or other Agile approaches can’t be successful in an environment that doesn’t have all the characteristics listed above but it does mean that there are extra and different challenges that may require more effort to overcome.
The characteristics of this particular organisation weren’t the only factors in the successful implementation of Agile to the organisation. The key driver for success was the approach to implementation.
Once identified as a good candidate environment for using the Scrum framework, the head of the IT department – the Agile champion – understood that in order for the implementation of Agile to be successful, it had to be a joint effort – i.e. a joint effort between the business and IT. He understood that being Agile wasn’t just a way to for IT folk to develop software, it had to be a more of a culture – a culture for the organisation as a whole – and he spent significant time and effort ensuring that not only were the Senior Management from both the IT and the business side engaged but also pursued full engagement at board-level. Implementing Agile was going to require some changes to be made to the organisation – some fairly dramatic ones and this wouldn’t be possible without backing at the highest level. It was understood that Agile wasn’t a silver bullet and wouldn’t solve the organisation’s problems overnight and it was understood that in order for it to be successful, a significant amount of time and money would need to be invested initially before the benefits started to be seen.
Once the organisation as a whole was on board and ready for this fairly significant change in delivering value to the customer, the company invested a lot of time and money in to the approach by taking the following steps:
- Product Owners were identified from the business. Strictly, there was one Product Owner per product. That Product Owner may well decide to designate certain activities to other team members and they may well represent the sometimes conflicting views of a number of different stakeholders from the business side but ultimately, and this point is key, that one person was the decision maker from the business perspective.
- Product Owners were trained and their roles and responsibilities were adjusted to make them available to carry out Product Owner-related activities.
- Multi-disciplined IT teams were formed (Scrum Teams) and aligned to a product or a suite of products (i.e. instead of having a team of developers; a separate team of testers; a separate team of business analysts etc., a single team was formed that consisted of the key disciplines needed for that team to deliver value to the customer).
- All IT team members were trained.
- All Scrum Teams were empowered in a true sense. Again, I believe this was key to the success of Agile at this organisation. Each Scrum Team could decide:
- The ideal length of sprint
- The method for estimation (hours, points, t-shirt sizes, it didn’t matter, it was clear that the estimation was only significant for that Scrum Team and not something that could be measured holistically)
- Their own standards for methods of communication
- Their own methods for documentation, knowledge sharing
- Their own processes, essentially. All Scrum Teams were given guidance in the basics of Scrum but the specifics of exactly when meetings took place, how support issues were handled etc. were decisions that were made by the team.
- Scrum Teams were responsible for all aspects of product development, including support. How they managed support was up to the team.
- Scrum Teams were encouraged to continually ‘inspect and adapt’ – identify processes, activities and methods that were working, identify those that were not and come up with their own ideas of how things could be done better.
- Scrum Teams were urged to share their successes, failures and lessons learned with other teams so that the organisation as a whole could benefit from the experience.
The implementation of Agile was not perfect, of course. No significant change in the way a company does something is going come without pitfalls and mistakes but the key to success is being open to identifying those things that are not working and being in a position to change them. For example, one of the things that happened when this particular organisation first started using Agile and requirements became user stories in a Product or Sprint backlog was that general product documentation was not being produced or maintained. The focus became delivering as much value as possible to the business at the end of each sprint and some of the artifacts or activities that didn’t directly contribute to that goal were being sacrificed to achieve this goal. Some examples of the kind of thing I am referring to here are technical documentation, business process documentation, user documentation and code refactoring. The problem with keeping the focus always on delivering value to the customer quickly in lieu of some of these activities that provided more longer term value is that it comes back to haunt you later in the form of technical debt, overhead in training new team members and many other ways. After some time, this was identified as a problem and each team had to deal with it but, again, how each team dealt with it, was up to them. Some teams chose to factor in a certain amount of time for each sprint to conduct some of these activities. Some teams chose to have a ‘synchronisation’ sprint every few sprints – the frequency of which being determined by the team – where all team members would focus on activities that would reduce technical debt, assist with knowledge transfer etc.
I think what also heavily contributed to the success of Agile at this organisation was the fact that the IT Senior Management continued to engage the business and board members in the process long after the initial implementation of Agile and held regular open discussions with them about how it was working, not working, what they thought could be done better. This continuous process of inspecting and adapting at an organisation as well as Scrum Team level never stopped.
So, I’ve talked a lot about how Scrum can work well. To give some balance to this topic, I want to take some time to share my observations about when Scrum doesn’t work so well.
The Type or Organisation
As I mentioned earlier in this post, one of the key factors in using Scrum successfully in my experience has been about empowering Scrum Teams and allowing them to make their own decisions where appropriate.
In an ideal world, all organisations wishing to implement Agile would have an environment where teams truly could make their own decisions about how they do things and when they deliver to the customer but in reality, for many organisations, this just isn’t realistic. I have found that organisations that are heavily regulated and audited, such as those in the Financial Services industry, particularly struggle with the adoption of Agile. In these kinds of organisations, there is often a need to be able to provide evidence that supports some kind of decision that has been made by the organisation quickly and in a standard format. The nature of this means that these companies tend to be more heavily focused on fairly fixed standard processes for all, standard tool use for all, fixed delivery windows for all etc. etc. By definition, Scrum Teams simply can’t be self-organising and truly empowered in this kind of environment. That isn’t to say it isn’t possible to implement Agile methodologies in these kind of organisations, but it is to say that it is a far greater challenge and that these organisations will have to work harder to find a ‘flavour’ of Agile that works for them and be prepared to make far greater changes to the current organisational set up.
Over-Standardising the Process
I am not suggesting that having standard processes is a bad thing and should be avoided by any organisation wishing to become more Agile. Having standard processes for certain activities is good and necessary. Having too many, too heavy, too inflexible standardised processes is likely to make being Agile impossible. For example, having a standard tool that everyone uses to capture requirements (user stories) makes sense. Standardising the exact format all user stories should take and the specific method that should be followed to gather and groom the backlog in which they reside, doesn’t.
The problem I am getting at here is when organisations implement Agile for the sake of implementing Agile rather than for the sake of improving the speed at which the organisation can respond to change, mitigate risk and deliver value to the customer which, in my mind, is really what it is all about. Implementing Agile holistically is important in terms of creating a culture and environment in which the benefits of Agile can be realised but implementing fixed interpretations of Agile methodologies as standard processes holistically – something I have seen done in a number of organisations – takes away the fundamental ideology of being Agile that makes it work, such as the self-organising and empowering aspect of it. The key to success is having the common sense to identify what to standardise and what not to. For example, standardise the information that each Scrum Team has to provide to stakeholders in order to allow for a quick response to an auditing or regulatory request, standardise where that information is held so that it can be accessed with ease but don’t standardise the method the teams must take to collect that information. As another example, standardise a set of documentation that each Scrum Team should provide and be responsible for the updating and maintenance thereof but don’t standardise the exact format of the documentation – allow teams to document their product in the manner that best suits them.
The Business Not Being Fully Engaged
This is something I have seen happen a lot and one which is a key factor in Agile being unsuccessful. When an organisation, whose primary business is not IT – i.e. one where IT is a supporting function to the business – tries to implement Agile from an IT perspective without fully engaging the business, it leads to all sorts of problems which are, in my opinion, impossible to solve from an IT perspective alone. The issue of the business not being fully engaged / on-board can present itself in a number of common ways:
- No Product Owner. Without a key decision maker from the business, as an IT department / team, you are simply guessing what the business wants and hoping that what you deliver will provide value
- A Product Owner that doesn’t understand their role or have the time available to conduct Product Owner activities. This is a fairly common issue in implementations of Agile, particularly Scrum, that I have seen fail. A Product Owner who can make decisions but isn’t available to respond to the Scrum Team, prioritise backlog items and be available to review ‘ready’ software fairly promptly is likely to create a scenario where teams are continually carrying items of work over to the next sprint or where what is delivered isn’t what the business actually wanted.
- Multiple Product Owners. Again, a fairly common issue I have come across. This is where a Scrum team have more than one person from the business designated as the decision maker. Often these people have conflicting viewpoints and this leads to time being wasted discussing and negotiating different viewpoints and the likely delivery of software than none of the multiple Product Owners are happy with.
Not Adapting the Process to Fit The Circumstance
Another common mistake I have seen organisations make is in selecting a framework to follow and not adapting it for changing circumstances. For example, an organisation may decide that the Scrum framework will work best for them and invest a lot of time and effort in implementing it but fail to adapt the approach for different circumstances.
From my experience, there are three main scenarios in which software is developed and delivered:
- New Projects: Building a new product from scratch or a complete overhaul and re-work of an existing product.
- Business as Usual (BAU): Delivering software for an existing product consisting of enhancements and non-emergency defect fixes on a frequent (generally scheduled) but not immediate basis
- Support: Dealing with customer issues and if necessary, delivering software containing emergency defect fixes without the need to wait for the next scheduled release
The purest form of Scrum and the one that most people understand and feel comfortable with works pretty well in a BAU scenario but when it comes to new projects and dealing with support issues, trying to fit the same framework to these scenarios is likely to cause issues. I will be writing a separate blog post that goes in to more detail around why taking different approaches for BAU and new project work is important but, to give an example, when you are building a new product or completely overhauling the architecture behind of the look and feel of an existing product, it is neither realistic or necessarily advisable to aim to deliver software to the customer at the end of every sprint so the value of using Scrum is more around mitigating risk and coping with changing requirements than delivering value to customers quickly and regularly. Equally, when dealing with support, the goal is to always work on the highest priority item and deliver it to Production as quickly as possible so selecting items for a sprint, prioritising them and estimating them all serves little purpose and wastes time unnecessarily when the highest priority item may change more frequently than a standard sprint can cope with. In this scenario, something like Kanban is more effective.
Finally, a quick note on Waterfall methodologies – a term that has sadly almost become a dirty word! I am a fan of Agile. When implemented well, I believe it can add a huge amount of value to an organisation but I also believe there is still a place for Waterfall methodologies in certain circumstances and organisations shouldn’t be discouraged from following them where those circumstances apply. In environments where, for example, scope is generally fixed and unlikely to change (which is not as unusual as you may think – changes necessary to meet regulatory requirements for example), teams are not co-located, the business are not willing to actively engage in IT delivery etc., following a Waterfall methodology is likely to be far more successful than any implementation of Agile could be and if that is the case, then organisations should do that.