We all know the stories about agile failures. About things like Flaccid Scrum, Cargo Cult Agile, and all the others. We all hear the tales of micromanaging PMP-certificate wielding “scrum-masters”, absentee-landlord product owners, and sadistic pitiless stakeholders. What’s gone wrong? A few things. But mainly, organisations are signing up to Agile for the wrong reasons, and they’re not making the right changes. They’re trying to get away with the least possible disruption: just changing the way software development work is structured. Agile is about software development. But Agile is also about much more than that. It requires changes to the entire organisation.
The traditional view of IT is that it is a weird specialist engineering function, separate from the rest of the organisation, and very separate indeed from anyone in the “business”. Those people have jobs that involve understanding customers, and business, and money and making more money. And what do engineers know about that? Nothing! They’re just a bunch of nerds, right? We should just use our business smarts to figure out what to build, and then tell them to build it, and wait until it’s finished, right?
If that is the company’s attitude, they may as well just go all the way. Sack all the IT people, and hire some overpriced vendor or agency. Then we can all pay through the nose, get something nobody wanted, and bring in the lawyers and argue over contracts like the good old days in the 90s.
This view is, of course, a profoundly stupid way of looking at technology. We are now living in the 21st century. IT has become a fundamental part of how all firms operate and interact with customers (and suppliers, partners, and so on). Business and technology need to work together. They can’t be at an arms’ length from each other anymore. In fact, the distinction between business and IT needs to be dissolved altogether. And we need to start applying Agile principles to not just the way we build software, but the way we run businesses.
We all know times have changed. We live in a completely different world than the one we lived in 30 years ago, let alone the one we lived in 90 years ago. But we still structure organisations, manage projects, choose strategies, hire and reward people, draw up budgets and fund work the same way we did 90 years ago. The days of scientific management need to be resigned to the dustbin of history.
Everyone needs to recognise these (whether they like them or not):
If you don’t get with the times, you will be disrupted and obliterated. It’s that simple.
And the values and principles of the Agile manifesto are a great starting point for doing that. Why? Because they focus on time to market and delighting customers. Not stakeholders: customers. The job of a traditional product lead or project manager is to “make the stakeholders happy”. Problem is, stakeholders don’t make or break the company, customers do. And stakeholders rarely know what customers want. Product owners often don’t either, which is why you need to get software to the customers quick as you can and watch what they do and find out what they want.
Contrary to popular belief, the US military is not a shining example of a big hierarchical organisation with strict chains of command. They’re smart and they know that doesn’t work in the chaos of war. They follow the principles of Auftragstaktik or “mission command“: form many small units of smart empowered cross-functional teams, give them some high-level objectives, and give them the freedom to adapt to their situation and find the best way to get the job done. Putting layers of governance and bureaucracy around people means that decisions won’t get made and work won’t get done in time. Which spells death. Usually a slow one, but these days, more of a fast one.
No more treating technology people as strange nerds to be mistrusted and micro-managed. We need to hire really good people and reward them appropriately. Not by dangling fat bonuses in front of them, tied to stupid KPIs, but by giving them freedom, autonomy, and buy-in to the project and the value they are delivering. Trust people, give them all the tools they need to get their job done, pay them what they’re worth, and get the hell out of their way. That’s the “secret” to management: I just saved you three years and $50,000 so you don’t have to get an MBA.
We need to let developers do their jobs properly, build things properly, and pay down technical debt properly. Otherwise, we get swamped with it and our products go to hell. It’s called “technical debt” for a reason: it grows by itself if you don’t do something about it. And if you don’t, it gets out of control and becomes technical bankruptcy. I’ve seen it happen and it’s not pretty.
Agile means iterative development, not incremental. Building things breadth-first, to detect and capture customer value in as broad a design space as possible, then simple set-based concurrent engineering to hone down on the delighters that will bowl customers over and maximise throughput. We need to stop being mindless feature factories, pulling random tickets off boards that some stakeholder thinks might look good on a CV or pick up some press on Hacker News. We need to stop doing depth-first incremental development. This type of work is a form of a hill-climbing algorithm that maximises output, locally, instead of optimising globally for outcomes and throughput.
There is no point getting people to do “scrums” and “sprints” if you’re giving them a twelve-month project from a pre-canned business plan with fixed features and benefits. That’s not putting a square peg in a round hole, that’s setting the peg on fire and throwing it in a dumpster. The whole point of Agile is frequent discovery and delivery of value. If some budget committee has decided 18 months ago what “value” means for this product, without talking to any customers, or even building a prototype, the exercise is a colossal waste of time and money.
We need to move to independently funded value streams with small-batch discretionary funding, responsible for their own P&Ls. Does that sound scary? Probably not as scary as throwing away tens or hundreds of millions of dollars every year on pointless projects (which a lot of companies do).
The simple truth is that traditional cost-based accounting methods were fairly outdated and inappropriate in the 1980s, and are now laughably bad. Amortising all operational expenditure across all business outputs leads to sub-optimal decisions that never maximise throughput. Focusing on cost-reduction instead of revenue maximisation suffers from the same malaise. Big successful companies are:
The traditional project mindset worked fine for civil engineering projects in the 1950s, which is where it came from. I’ll leave it to the civil engineers to figure out if it still works for their domain today. It didn’t work well for software in the 1950s and it is despicably bad for software today. The truth is, most valuable software work today is done building products, not delivering projects, and products:
It’s not just building it, it’s running it too. No more throwing big piles of buggy code over the wall to poor overworked “Operations teams”. If you’re lucky, they’re in a bunker in the suburbs somewhere – if you’re unlucky, they’re 7000 miles away in a different timezone. Good luck “collaborating”.
We need to switch from the horrid half-baked ITIL crap people are doing now to DevOps. Everybody knows it, not enough people are doing it. You build it, you own it, you run it. Developers won’t hate doing this if you let them do their jobs properly, with automated test pipelines running from Dev through Test, Staging and Production. If you give them responsibility for a sensible domain-based fine-grained set of services, instead of some ugly legacy monolith.
When it comes to risk, most organisations treat product or software teams like children. They assume they don’t understand risk, governance or compliance. They force them to run around to risk forums and governance committees and compliance audits. And they force them to “raise risks”, fill out forms, explain themselves over and over to people who frequently don’t understand the product or service they are building. This is condescending and a waste of time and money.
The truth is, using Agile approaches de-risk work from the beginning, instead of frantically throwing mitigants around at the end. They do this by reducing technical risks through spikes, reducing value risk through early prototypes and releases, reducing quality risk through automated testing, and reducing operational risk through small batch size and continuous delivery.
Most governance processes and problems exist because firms have a poor technical and organisational architecture. This means managers have to spend more time playing traffic controller, making people put their toys away, and stopping people crashing into one another, instead of leading and building teams. When people work in an environment based on small business domain focused cross-functional teams, operating on solid microservice architectures, with security and compliance testing as part of their automated test pipelines most governance problems go away (or don’t turn up in the first place).
Traditional marketing was like most business functions: big, dumb and slow. Fortunately, marketing people have embraced the new digital age with a furious passion and enthusiasm. More power to them! They even have an Agile Marketing Manifesto! Great stuff. Digital marketing, like Agile software development, is all about small and fast: targeting and understanding individual customers, responding quickly to changing environments, maximising returns through late decisions and reduced waste. Fortunately, marketing departments as a whole understand this really well and are leaving the rest of their companies behind.
It’s quite simple: Agile won’t work if everybody in a company isn’t behind it. It’s not just changing the way software developers do their work. And it’s not even about that plus changing how the testers and business analysts and project managers do their work. It’s about changing the way the whole organisation works: from big slow batches, full of risks and uninformed decisions, to small nimble autonomous and empowered teams. From big clumsy budget cycles and business cases to iterative discretionary funding and continuous delivery.
Some firms understand this and are generating supernormal returns. Others don’t, won’t or can’t, and I don’t need to tell you what will happen to them in the next five years.