The clever version kept losing

I built an AI workspace for my colleagues: a chat interface wired up to our own documents, so you can ask a question and get an answer grounded in the source material. Useful, and fairly boring on paper. I was the only developer on it, with colleagues waiting to use it and stakeholders asking how it was going.

Going in, I assumed the hard parts would be technical: the model, the retrieval, keeping the code from buckling under its own weight. That’s not where the pain came from. Almost all of it came from moments where I had a boring option and a clever one, and picked the clever one. It cost me every time. What follows is where that happened, and what I’d tell myself if I could start again.

I lived on Streamlit for seven months

I started on Streamlit, like a lot of people do. You write Python, add a few decorators, and you’ve got a web app by the afternoon. No frontend, no API design, nothing to deploy. For proving an idea it’s genuinely hard to beat. I could change a line, refresh, and show a stakeholder something working the same morning they asked for it.

Then I stayed. Six, seven months in, still on it, still patching. What began as a simple data tool had quietly become a workflow app with file uploads, sessions, background jobs, and multi-page navigation. Each addition felt reasonable on its own. Together they pushed Streamlit miles past what it was built for.

The cracks were specific. Streamlit re-runs your whole script on most interactions, which is fine until each run is firing LLM calls and shuffling state around. I was writing custom CSS to hide that it was Streamlit, hand-rolling session management before the framework supported it properly, and watching the thing wobble the moment more than a couple of people used it at once. Users would click, see nothing happen, and refresh, which kicked off the expensive work all over again. Each of those workarounds was me choosing another clever patch over the boring rewrite.

I knew the signs months before I acted. I was spending more time fighting the framework than building features, and people kept calling the app “rough.” Sunk cost was part of it: I had months of work tangled up in it, and a rewrite felt like going backwards. But the bigger problem was that the call was mine alone, and I kept not making it. I kept trying out one more feature on the prototype, waiting for traction to tell me it was time to build properly. Inside a business that signal arrives slowly. Adoption is gradual, and the formal go-ahead is slower still. The right move was to stop experimenting the moment the concept felt concrete, build the real thing, and ask for forgiveness later.

The rewrite to a proper web application took a while and paid off the moment it landed. Proper loading states, real error handling, an interface that didn’t feel like a demo. The lesson is dull and I’ll repeat it anyway: use a prototyping tool to prove the idea, and once the concept is concrete, treat what you’ve built as scaffolding and move on.

My RAG pipeline lost to a single call

The retrieval engine behind the chat went through the same cycle, and it’s the one that still stings. I’d read the papers on multi-step reasoning and watched the demos where an agent “thinks” its way through a hard question. So instead of plain retrieval and generation, I built a pipeline: retrieve, analyse the gaps, retrieve again to fill them, synthesise, format. Five steps, each its own LLM call, each with its own prompt and its own way of going wrong.

It looked great in a diagram. Stakeholders loved watching it work through a query step by step. Then a user asked a straightforward question during testing, and my clever pipeline took over a minute to return an answer that was worse than the obvious alternative. The analyse step had invented a detail. The re-retrieve step had dropped the context that actually mattered. I ran the same question through plain retrieval plus one well-prompted call and got a better answer in about fifteen seconds.

I’d been optimising for how sophisticated the system looked, not for whether it helped anyone. Users didn’t want to watch the machine reason. They wanted an accurate answer quickly, and every extra step I’d added was one more place for it to go wrong.

I still reach for agentic frameworks (LangGraph, AutoGen) for demos and quick exploration. They’re good at that. In production they came straight back out.

Getting the documents into shape

For an app whose whole job was answering questions from our documents, I spent remarkably little time thinking about the documents. I assumed that work was done because we had them: years of company documents, templates, compliance guidelines, technical specs, all sitting in repositories someone described as “well-organised and up to date.” The small samples looked clean.

The full corpus was a different animal. Some PDFs had real text, others were scans that needed OCR. Word files were full of embedded objects and version artefacts that wrecked retrieval. Documents marked “current” turned out to be superseded by newer copies stored somewhere else. Compliance lines that should have been standard had a dozen near-identical variants whose small wording differences changed their legal meaning. We spent weeks just working out which documents were actually authoritative.

Then came the part that really hurt: most answers needed context spread across different repositories. The technical requirements lived in one place, the compliance detail in another, the historical outcomes in a CRM in a different format again. We tried to fix it by indexing everything, and the more we indexed, the worse retrieval got. Any change to any document triggered re-indexing that could break search for hours.

What worked was the opposite. Define the smallest set of documents that gives a useful answer, clean that, and get retrieval solid on it before adding anything else. When we finally cut back to the core documents people actually referenced, answer quality jumped. The instinct to cover everything was the thing holding us back.

It worked until five people used it at once

The backend underneath all of this was FastAPI, and it was fine in testing, because I tested like a polite single user: one request, wait for it, then the next. Real users don’t do that. They fire off several questions at once, switch document sets, and refresh when something feels slow.

The architecture couldn’t take it. Each request ran the full retrieval and generation pipeline synchronously in the main thread: vector searches, ranking dozens of chunks, embedding calls, then generation. One patient user was fine. Three people running real analyses at the same time made the server unresponsive. Polling made it worse, because slow jobs meant the frontend was constantly asking “done yet?”, piling cheap status checks on top of the expensive work. And background document ingestion was running on the same box, fighting live requests for the same resources.

The fix was a different shape, found mostly through desperation. Server-sent events let the server push progress to the client instead of the client asking on a loop.

Celery moved the heavy retrieval and generation onto background workers, so a request could return a task ID straight away and report progress as the work ran. Redis cached the repeated queries and held session state. None of this made the underlying work faster. It made the app feel responsive while the work happened, which turned out to be the part users cared about. Moving the work onto queues had a side benefit I hadn’t planned for: I could finally see it. Queue lengths and worker times showed exactly where things backed up, which the synchronous version had kept invisible. The real lesson: you design for concurrency up front. You can’t bolt it on once people are already hammering the thing.

The same mistake, away from the code

The pattern wasn’t limited to architecture. Before we’d even shipped a first version, we’d set up Linear with story points, grooming sessions, and burndown charts. We spent something like a quarter of our time writing tickets and updating statuses for features that didn’t exist yet. We once spent forty-five minutes debating the priority of a request from a single user who hadn’t opened the product in weeks, while real bug reports sat in the backlog because they weren’t “groomed.” The tickets gave us a comforting sense of progress while we built things nobody needed. Heavy process belongs to a product that has found its shape. Early on, a short list of the next few things to do was plenty.

Feedback was its own trap. Users couldn’t tell me what they wanted from a workflow they’d never used. They’d ask for “better search” and then, the moment they saw a working system, discover they actually needed to trace answers back to source documents and check the original compliance wording. Fair enough. Seeing it working cut the other way too: every demo turned into a brainstorming session about contract analysis or compliance checking, which felt like enthusiasm and behaved like scope creep. And the feedback disagreed with itself. One user loved the chat interface, another wanted structured forms, a third wanted everything automated while a fourth wanted manual control over every step. You can’t average those preferences into a product. Someone has to pick.

The harder problem was getting time with them at all: demos cancelled, sessions slipping by weeks. What eventually worked was treating feedback like a proper meeting instead of a favour: a recurring slot, a specific agenda (“Tuesday we decide the filtering options”), and saying plainly that feedback delays push delivery dates by the same amount. I learned to build that waiting time into the plan instead of crunching for it at the end.

Deployment humbled me the usual way. Everything ran on my machine, where I had broad permissions, localhost everywhere, and services set up exactly how I liked them. The cloud had none of that. SSL requirements that didn’t exist locally, network restrictions, permission errors, environment variables that weren’t set. The fix is cheap and I skipped it anyway: stand up a production-like environment early and keep deploying to it as you go, so the surprises arrive one at a time instead of all at once on launch day.

If I could start again

Looking back, nearly all my pain came from overcomplicating the architecture and underrating the boring parts. The good stretches were the simple ones: a single API call, a small clean dataset, a scrappy version in front of a real user. The bad ones were every time I fell for a framework or a process before I understood the problem it was meant to solve.

The technology choices mattered less than I expected. React or Vue, Postgres or Mongo: I agonised over those calls and they barely changed the outcome. The human parts (managing expectations, chasing feedback, working out what people actually needed) were harder than any of the code.

I’d build it again. I’d just assume from day one that the clever version is going to lose, and reach for the boring one first.