Avoiding Delivery Hell

Some enterprises have grown their technical infrastructure to the point where dev ops and continuous deployment are second nature. The vast majority of enterprises are still on their journey, or don’t even realise there is a journey for them to take. Businesses aren’t generally built around great software development practices – many businesses are set up without much thought to how technical work even gets done, as this work is seen as only a supporting function, not able to directly increase profitability. This view of technical functions works fine for some time, but eventually stresses begin to form.

Failing at software delivery.

Each area of a young business can be easily supported by one or two primary pieces of software. They’re probably off the shelf solutions which get customised by the teams who use them. They probably aren’t highly integrated; information flows from department to department in spreadsheets. You can think of the integration points between systems as being manual processes. 

While the flow of work is funnelled through a manual process such as sales staff on phones or shop staff, this structure is sufficient. The moment the bottleneck of sales staff is removed (in other words, once an online presence is built where customers can be serviced automatically) things need to run a bit quicker. Customers online expect to get instant feedback and delivery estimates. They expect to be able to complete their business in one visit and only expect to receive a follow up communication when there is a problem. Person to person interaction can be very flexible; a sales person can explain why someone has to wait in a way which sounds perfectly fine to a customer. The self-service interaction on a website is less flexible – a customer either gets what they want there and then, or they go somewhere else.

And so businesses start to prioritise new features by how many sales they will bring in, either by reducing the number of customers jumping out of the sales flow or by drawing additional customers into the sales flow.

Problems arise as these features require more and more integration with each of the off the shelf solutions in place throughout the various areas of the business. Tight coupling starts to cause unexpected (often unexplained) outages. Building and deploying new features becomes harder. Testing takes longer – running a full regression becomes so difficult and covers so many different systems that if there isn’t a full day dedicated to it, it won’t happen. The online presence gets new features, slowly, but different instabilities make it difficult for customers to use. The improvements added are off-set by bugs.

Developers find it increasingly difficult to build quality software. The business puts pressure on delivery teams to build new features in less time. The few movements among the more senior developers to try to improve the business’ ability to deliver are short lived because championing purely technical changes to the business is a much more complicated undertaking than getting approval from technical piers. It’s not long before enough attempts to make things better have either been shot down or simply ignored, that developers realise there are other companies who are more willing to listen. The business loses a number of very talented technologists over a few months. They take with them the critical knowledge of how things hang together which was propping up the development team. So the rate of new features to market plummets, as the number of unknown bugs deployed goes through the roof. 

It’s usually around this point that the management team starts talking about off-shoring the development effort. The same individuals also become prime targets for sales people peddling their own off the shelf, monolithic, honey trap of a system – promising stability, fast feature development, low prices, and high scalability. The business invests in some such platform without properly understanding why they got into the mess they are in. Not even able to define the issues they need to overcome, never mind able to determine if the platform delivers on them.

By the time the new system is plumbed in and feature parity has been achieved with existing infrastructure, the online presence is a long way behind the competition. Customers are leaving faster than ever, and a decision is made to patch things up for the short term so the business can be sold.

Not an inevitability.

Ok, so yes this is a highly pessimistic, cynical, and gloomy prognosis. But it’s one that I’ve seen more than a few times, and I’m guessing that if you’re still reading then it had a familiar ring for you too. I’m troubled by how easy it is for a business to fall into this downwards spiral. Preventing this is not just about being aware that it’s happening, it’s about knowing what to do about it.

Successfully scaling development effort requires expertise in different areas: architecture, development, operations, whatever it is the business does to make money, people management, leadership, change management, and others I’m sure. It’s a mix of technical skills and soft skills which can be hard to come by. To make things even harder, ‘good’ doesn’t look the same everywhere. Different teams need different stimuli in order to mature, and there are so many different tools and approaches which are largely approved of that one shoe likely won’t fit all. What’s needed is strong, experienced, and inspirational leadership, with buy in from the highest level. That and a development team of individuals who want to improve.

Such leaders will have experience of growing other development teams. They will probably have an online presence where they make their opinions known. They will be early adopters of technologies which they know make sense. They will understand how dev ops works. They will understand about CI and CD, and they will be excited by the idea that things can get better. They’ll want to grow the team that’s there, and be willing to listen to opinion.

Such leaders will make waves. Initially, an amount of effort will be directed away from directly delivering new functionality. Development effort will no longer be solely about code going out the door. They will engage with technology teams at all levels, as comfy discussing the use of BDD as they are talking Enterprise Architecture. Strong opinions will emerge. Different technologies will be trialled. To an outsider (or senior management) it might appear like a revolt – a switch in power where the people who have been failing to deliver are now being allowed to make decisions, and they’re not getting them all right. This appearance is not completely deceptive; ownership and empowerment are huge drivers for team growth. Quality will come with this, as the team learn how to build it into their work from architecture to implementation, and learn how to justify what they’re doing to the business.

Software delivery becomes as much a part of the enterprise as, for example, Human Resources, or Accounts. The CEO may not understand all aspects of HR, but it is understood that the HR department will do what needs to be done. The same level of respect and trust has to be given to software delivery, as it is very unlikely the CEO or any other non-technical senior managers will understand fully the implications of the directions taken. But that’s fine – that’s the way it’s meant to be.

In time to make a difference.

Perhaps the idea of strong technical leadership being critical to technical success is no surprise, it seems sensible enough. So why doesn’t this happen? 

There are probably multiple reasons, but I think it’s very common for senior managers to fear strong technical leadership. There seems to be a belief that devolving responsibility for driving change among senior technicians can bring about similar results as a single strong leader while avoiding the re-balancing of power. I see this scenario as jumping out of the same plane with a slightly larger parachute – it’ll be a gentler ride down, but you can only pretend you’re flying. By the time the business makes it’s mind up to try to hire someone, there’s often too much of a car crash happening to make an enticing offering. 

If we accept that lifting the manual sales bottleneck and moving to web based sales is the catalyst for the explosion of scale and complexity (which I’m not saying is always the case) then this would be the sweet spot in time to start looking for strong technology leadership. Expect to pay a lot more for someone capable of digging you out of a hole than for someone who has the experience to avoid falling in it to begin with. And other benefits include keeping your customers, naturally.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s