Software delivery in many organisations is still far too waterfall, inefficient, and often unhealthy to be a part of. The myriad of methodologies which can be applied in different combinations to achieve something more efficient can be overwhelming. Normal organisational structures deny people the authority to make the efficiency savings which seem logical to them. Trust is hard to come by. The different roles are constantly blocking each other. It’s time to change… again.
Agile isn’t dead, it’s just misunderstood, mangled, and sold for profit. The teams benefitting from high agility are generally pulling from much deeper ideas than just the Agile Manifesto. Not to say the manifesto is wrong (it remains one of the most important collection of guidelines in modern software engineering), but the ideas from which it takes inspiration are (in my opinion) a stronger foundation.
Agility is also not the sole end goal. Many situations call for more focus on speed, rather than keeping options open. Agility is a sliding scale and there are often multiple factors to weigh against it. Focusing on ‘being agile’ will at best only result in agility.
My personal experiences of different approaches suggests that the concepts around continuous delivery provide early ROI, faster completion times, higher quality, a better working environment, and also high agility. I believe combining modern software delivery techniques with a fresh views on how to enable teams, rather than block them, will give the business the delivery focus they want, and give the delivery teams a healthy and inspiring place to work.
To this end, I recommend a collection of delivery focused, lean concepts, coupled with a few ideas based on personal observation. At a high level, these are the concepts I think matter most:
- There are 2 primary stakeholders in software delivery: The Business, and Software Developers.
- Non developer jobs (QA, Architecture, BA, and others) are not departments, they are roles within a delivery team.
- Within a Software Delivery Team there are developers and there are people helping the developers build the right thing.
- Delivery artefacts, such as stories and architecture diagrams, are there to help the developers build the right thing. They are created because they are needed to communicate something within the team. If there is no need to communicate something, the artefact is not created.
- Reporting artefacts are there to help The Business understand what is happening. They give confidence that the right stuff is being delivered in an efficient manner. Delivery artefacts are not reporting artefacts.
- Supporting the Developers is completely detached from supporting the Business. Any effort focused outside of the team is time taken away from delivery, which needs to be acknowledged by all stakeholders.
- CI/CD and lean processes are a must – ‘dark agile’ is to be avoided.
- Products not projects
When I think about these concepts working together, I see how they remain true and drive the same delivery focused efforts no matter what the scale of the organisation. Let’s take a look at each in turn and expand on them a little:
1. The Business, and Software Delivery Teams
Take any organisation which depends on software delivery, and you can remove everyone who is neither running the business or writing software without completely destroying the business. Many organisations will have grown from humble beginnings, where there were only a handful of Developers and no Architects, no BA’s, no QA’s, etc. These other roles can be incredibly important, but they are supporting roles there to enable software delivery. They are introduced to make sure the right things are getting built, they should be given that focus.
2. Roles Within a Team
Traditional waterfall methods give ownership of ‘project stages’ to each discipline. BA’s define functionality and requirements, then Architects define larger system units and how they integrate, then Developers build what’s been defined, then QA’s check what was built matches what was specified. One of the many problems with this is that it breaks up the team and introduces arbitrary divisions where none need exist. These divisions cause conflict, they can’t possibly not do – everyone is standing like Gandalf in the storm “You shall not pass!” – this isn’t team work.
To remove friction, these divisions must be removed. The roles must become a team, and they must all recognise that their number one objective is to deliver software.
3. Supporting the Development Effort
Within the team, anything an Architect or a BA or a QA does should be with the singular purpose of enabling the delivery of great software. There should be no forces external to the team driving their own agendas. For example, if a piece of work is of a highly technical nature, then the story reflecting that work will likely contain some very technical details. That should be fine – there shouldn’t be anyone from the business looking so closely into each individual story that they need everything dumbing down to their level. The stories are delivery artefacts, they are there to communicate to the people delivering. The same goes for architectural diagrams – they are a communication tool which should be used to help the Developers build the right thing. If a team decide they don’t need a particular diagram, then it isn’t created – don’t waste effort. QA can support development by helping to make sure that the quality bars are understood before the development work is carried out. Linking AC’s with automatable tests which Developers can write to prove the story before it ever gets to into a testing phase.
4. Delivery Artefacts
Architecture diagrams. Stories. Epics. Test approach documents. Decision registers.
There are undoubtedly more than this. Anything produced to help with delivery is produced with the sole intention of helping developers build the right thing. It’s a ‘pull’ system; the developers need something, it’s created.
5. Reporting Artefacts
The Business can’t be blind to what their Development Teams are doing. They require artefacts which communicate the correctness of what is being built, and a rough velocity. They need to be able to plan future work and be given confidence in their team, but the generation of these artefacts should in no way impact the teams, and no delivery artefact should ever be created with the intention of dual use. The idea that a single artefact can be both useful to a technical audience and also understandable to a non-technical audience is deeply flawed.
6. Support the Team or Support the Business
At any one time, an individual is either working within the team to support the delivery of great software, or they are working outside of the team to help the business understand what’s going on. The difference in these undertakings is called out explicitly because working outside of the team is done outside of team time. That’s not to encourage a big effort to calculate what percentage of a person’s time should be dedicated to the team and how much should be dedicated to reporting artefacts. Instead it should just be recognised that if a team member is busy with reporting artefacts, they can’t be busy supporting the development effort. There is a trade off. Ignoring this trade off pulls individuals in two directions, forcing decisions which seem helpful, but can ultimately slow everyone down. For example: trying to use the team’s stories to inform the business of what’s being built.
7. Agile, Lean, CI/CD
Truly delivery focused agility doesn’t come from any ‘dark agile’ methodologies, so forget SAFe. SCRUM is better, but comes with a lot of baggage around estimation. In my experience, most teams will function better if they focus on Kanban with lean practices – identify and eliminate waste, control how work flows into the team, learn and improve constantly with the focus on your worst bottle neck, keep work flowing in one direction. You can learn much more about Lean Principles by reading The Phoenix Project. It’s important not to be too detailed with your Kanban flow – start with things as simple as you can, and then add new steps only when they are needed. Your goal is to continue improving so you find the right working practices which fit your team in your organisation at a given time – expect things to keep changing (and hopefully improving) that’s the point.
Implementing CI/CD is critical, but be careful to do it right – just adding automation is not CI/CD; read this for more on the subject. Well implemented automation will enforce quality while enabling fast delivery – if you aren’t getting both of these things, you’re doing it wrong.
8. Products not Projects
I’m amazed that this isn’t the doctrine everywhere. If your team is a product team, then it can deliver functionality quickly, in the knowledge that they will iterate based on statistics taken from actual users. They can make technical decisions which work for their product, but which would be surprising to anyone outside of the team. They hold domain knowledge about the system so there are no surprise features getting blown away when they deploy something new.
On the flip side, a project team has to first gain as much knowledge as they think is relevant to the systems they will be working with, as they think will allow them to safely make the change. They need to work out how each system is meant to be delivered – how is quality maintained? When they finally deliver, they probably need to hand over to a BAU team, so they have to document every last detail of what they’ve done. They need their delivery processes to be fool proof, because no-one will be looking at them until something goes wrong. When something does go wrong, the BAU team will find processes to deal with the problem, whereas a product team would have handled the bug and fixed the issue.
There are no benefits to projects where there could be product teams.
Working within the team
There’s a lot more to be said about each of the above, but next I’d like to draw a picture of how the work of different roles is focused either within the team, or external to the team. Recognising the difference and calling it out for different tasks is important because it’s a trade off. Moving quickly requires dedicated focus on the development effort. Taking someone away from that effort will slow the delivery, even if it enables later delivery. This isn’t a bad thing, it just needs to be recognised.
Let’s consider a very simple team with only developers in it. Within a the team, developers write code. They can also write tests to prove their code, they can write DevOps code, and they can manage the CI/CD pipeline. It might be that this is enough – all you need are developers. This will work where the communication between the business and the team is good, and where the developers have enough experience to be able to test and design with autonomy, as well as write code.
As the team expands, it doesn’t mean the introduction of new roles. These roles were already present. It merely means introducing individuals who are specifically focused on one or more of the supporting roles. The way to introduce these individuals is to work backwards from the sharp end and see where more focus is needed. If you are keeping a frank relationship with your developers, then they should be able to tell you what help they need.
The BA role
When I talk about the BA role, I’m talking about how work gets defined and broken down in time to be built. I see this as synonymous with Product Owner in most cases.
If things are very simple, stories can be derived by talking to a representative from the business, on the fly. Getting together every couple of weeks to make sure there are enough fully defined stories available for the next couple of weeks, is enough. When the business priorities keep changing, or the solution being built reaches a level of complexity that multiple teams are involved, having someone prioritise and outline stories in advance of the team seeing them can make those sessions move faster. As the way forward becomes harder to see, it’s often easier for an individual with experience to help drive the right direction.
At this point, the BA role is fulfilling two needs: they’re helping control the flow of work into the team, making sure that priorities and details are in place just in time for the team to produce stories; and they’re also working outside of the team, to help interpret the needs of the business. Their outward facing role can also be extended to include communicating progress and performance. This is a real skill as many KPI’s of a lean software delivery team are incomprehensible to the business – it’s like the driver of a car trying to understand how the engine management system relates valve timing with engine revs and throttle position, and why it’s important for efficient forward motion.
The important concept when reporting back to the business, is that no requirement for information should ever impact how the team works.
The Architecture role
Initially, it’s probably fine for developers to build the things which seem sensible, but at some point, things might start getting messy. It might also start to make sense if some technologies and approaches are shared between teams, creating a common solution.
Within the team, an architect should be leading the technical thinking, making sure the choices being made are taking the enterprise in the right direction. They’ll be heavily interested in the DevOps side of things, utilising automation to drive the use of preferred infrastructure and services. They help map the journey from current state to future state, making sure any change in direction is implemented iteratively so as not to impact delivery. They are there to drive the development effort in specific technical directions, so the individuals undertaking the role should be respected technically by the developers.
The architecture role also acts outside of the team. They need to align the technical roadmap with the product roadmap, which means working with the BA role and business. It’s preferable for the technical roadmap to be heavily influenced by the preferences of the delivery team. It isn’t enough for an architect to design something which will work, they have to design something which lends itself to being delivered by the team they have. A good architect in a delivery focused, agile environment, might have more than one workable solution and may choose (with the team) to deliver what they know is not the final cut – iteration, refactoring, improving, is what happens when you’re product focused.
It’s very easy to fall into the trap of having too many architects. Architecture as a process should be ‘on demand’, as required, pull not push.
The QA role
The QA mindset is the glue that sticks built software to requirements. It’s the QA’s critical thinking which gets sufficient detail into the behaviours described in stories that a high level of confidence can be derived simply from the unit test coverage. By driving good BDD practices, such as Specification by Example and Executable Specifications, the QA role ensures quality is enforced from the top down, instead of as an after thought.
This can happen without a single dedicated QA, if there is enough experience in the team, and the complexity/scale of the work allows. As things scale, it’s critical to push as much testing as possible to as early as possible in the CI/CD pipeline – including the unit tests written by developers in the test plan. They will be involved in reviewing tests written to satisfy themselves that test cases are covered, and regression issues are mitigated.
The QA role is mostly focused inwards, on the team. They should be involved in reviewing architecture for testability concerns, but normally with the team.
The software development role
Finally, let’s talk about the role of the Software Developer.
The developers are at the sharp end of everything, they are right against the rock face. They need to be supported and enabled by the other roles so they can focus on moving as fast as possible with small increments. They are continuously integrating, so they are responsible for knowing their changes will probably not break the build with each push, so they work in tight loops, delivering small changes. They are invested in seeing each small change get into production because they don’t want to be pulled away from their next task to fix something they wrote earlier. They like certainty – if a story has a truth table in it, or some other example of inputs and outputs, then they will turn those samples into automated tests. They will likely write other tests for scenarios which they see in the code which weren’t known beforehand. They aren’t looking for some arbitrary code coverage statistic, they are simply making sure their logic has tests around it which prove it’s right.
Continuous refactoring and improving is normal. There is no such thing as ‘finished code’, even changes which have made it into production are not ‘finished’. Everything is open to change.
A developer should feel that everyone around them is there to help. The only gate to delivering is quality, and the entire team should be working to bake that in right from the start and constantly reaffirmed, without resorting to large, manual efforts, carried out too near the end of a project.
You can’t aim a hose before you turn on the water
As the team scales, it becomes harder for Developers to cover everything alone. They need help dealing with the business, they need help defining work, and ensuring quality. The important word here is ‘help‘. It’s one team, and if that team is going to deliver value quickly, the development effort mustn’t be blocked. Mistakes are guaranteed to happen, but that is always true, it’s better to acknowledge that, respond to problems and improve, than it is to create a gridlock of fear.
Developers do have a role outside of the team. They’re experience of the current codebase can be essential when deciding what direction to take the product in. Given two equally profitable ideas, but one happens to involve making changes in a minefield of code which everyone has been avoiding for years, the business might agree to be guided toward the other option.
What it Means
Being delivery focused has always been associated with high pressure and tight deadlines, but this isn’t the way. Using modern techniques to help developers write high quality product which is proven before it even hits a dedicated testing phase, is fast, very fast. Then call out where team member’s attention is being pulled away from the team, and you have an open, fast paced delivery team, who aren’t stressing or wanting to quit. Agility comes almost as an after thought.