Skip to content
There’s a tempting belief in the data world: if your AI model isn’t working as well as you’d hoped, you just need a smarter algorithm. Train longer. Add more layers. Try the latest architecture from the research papers.
In our experience, that’s almost never the real problem.
We’ve worked with a lot of organisations on data and AI projects, and the pattern we see again and again is this: the algorithm is fine. What’s failing is everything around it. The plumbing, the structure, the way the system is put together. In short, the architecture.
The Uncomfortable Truth About AI Failure Rates
A 2023 survey by Rexer Analytics found that only 32% of machine learning projects actually make it to production. That’s a sobering number when you consider the time, budget, and enthusiasm poured into these initiatives.
The reasons? They’re rarely about the model itself. They’re about data pipelines that break silently, deployment environments that look nothing like the training environment, monitoring that doesn’t exist, and teams that built a great notebook but couldn’t turn it into a reliable product.
Google’s AI research team famously published a diagram showing how ML code typically represents a tiny sliver of a real production system. The vast majority is infrastructure: data validation, serving systems, monitoring, configuration, and process management. Clever algorithms without solid architecture around them don’t survive contact with reality.
What “Clean Architecture” Actually Means in Practice
We’re not talking about abstract software theory here. Clean architecture, in practical terms, means building AI systems that are modular, maintainable, and observable.
It means separating your business logic from your data pipelines from your model serving. It means having monitoring in place from day one, not bolted on afterwards. It means your model can be retrained, swapped, or updated without the whole system collapsing. And it means the people who eventually own and maintain the system can actually understand what’s going on.
When these things are in place, AI projects tend to work. When they’re not, even the most sophisticated model will underperform or fail entirely.
The Silent Failure Problem
One of the trickiest aspects of poorly architected AI systems is what practitioners call “silent failure.” Unlike a traditional application that crashes and throws an error, a poorly monitored model just starts giving worse predictions over time. Nobody notices until the damage is done.
This happens because the world changes. Customer behaviour shifts. Data sources evolve. A fraud detection model trained two years ago is looking at a very different landscape today. Without proper monitoring baked into the architecture, drift goes undetected until someone complains.
We see this with clients who’ve built technically impressive models but have no systematic way of knowing when those models stop being accurate. The fix isn’t a better algorithm. It’s building the observability infrastructure that should have been there from the start.
Starting Simple, Building Solid
A principle we return to often: build end-to-end early, and start simpler than you think you need to.
We’ve seen projects spend months perfecting a model in isolation, only to discover that integrating it into the real system surfaces a dozen new problems. Getting a basic version into production quickly, even if it’s not yet optimised, forces you to confront architectural challenges before they become expensive.
Once you have a working end-to-end system, you can iterate and improve. The architecture gives you a stable foundation to build on. Without it, you’re optimising in a vacuum.
Where This Leaves AI Strategy
For business leaders investing in AI, this has a practical implication: don’t evaluate your data science team by how sophisticated their algorithms are. Evaluate them by how reliably their systems perform in production, how quickly they can iterate, and how well the solution holds up six months after launch.
A well-architected system with a straightforward model will almost always outperform a brilliantly complex model sitting on shaky foundations.
This is something we think about a lot when we’re helping clients build out their AI capabilities. The goal isn’t to build the cleverest thing in the room. It’s to build something that works, keeps working, and can grow with the business.
If your AI projects aren’t delivering the results you expected, there’s a good chance the answer lies in the architecture rather than the algorithm. We’d love to chat about what that might look like for your organisation.
