There is a specific failure pattern in AI projects that is common enough to be predictable. A prototype gets built quickly, it demos well, budget is approved on the strength of that demo, and then the thing quietly stops being used within a couple of months. Nobody declares it failed. It just stops appearing in anyone's week.
The reason is almost never the model. It is that a demo and a production system are different artifacts, and the distance between them is mostly unglamorous engineering that nobody costed.
A demo is allowed to be wrong
The single largest difference is tolerance for error. In a demo, a wrong answer is a talking point. In production, a wrong answer has a consequence: a customer gets misinformed, a document gets misfiled, a payment gets misrouted. The moment there is a consequence, you need to know how often the system is wrong, in what way, and what happens next.
That requirement alone generates most of the work that separates the two:
- A confidence signal, so the system knows what it does not know.
- An escalation path to a human, which means a human who has actually agreed to be on it.
- Logging good enough to reconstruct why a specific decision was made three weeks ago.
- An evaluation set, so a prompt or model change can be checked rather than hoped about.
Demos run on clean data
Prototypes are almost always built against a curated sample, because that is the sensible way to build a prototype. Production runs against everything: the scanned document that is slightly rotated, the customer record with the name in the wrong field, the message in three languages, the PDF that is actually a photograph of a screen.
The demo-to-production gap is largely the long tail of inputs nobody put in the sample. This is measurable in advance, and skipping the measurement is the most common estimating mistake in this kind of work. Run the prototype against a genuinely random slice of real data before anyone commits to a number, and the surprise arrives during estimation rather than after go-live.
Nobody owns it
A system that runs continuously needs an owner. Not a stakeholder, an owner: the person who is told when it breaks and who is expected to do something. Pilots frequently have no such person, because during the pilot the builder was implicitly the owner and everyone assumed that would continue.
When the builder is an agency, and the engagement ends at delivery, the system enters production with nobody watching it. It then degrades in the ordinary way software degrades, an upstream API changes, a credential expires, a source format shifts, and because nobody is watching, the degradation is discovered by a customer.
The integration was the project all along
In most business automation, the model is a small component surrounded by a large amount of plumbing: authentication into systems that were not designed to be integrated, rate limits, retries, idempotency so a retry does not double-charge somebody, and reconciliation when two systems disagree.
This is ordinary software engineering, it is where the bulk of the effort goes, and it is invisible in a demo. A prototype can fake every one of those things with a hardcoded token and an optimistic assumption. Production cannot fake any of them.
What to ask for instead of a demo
If you are evaluating a proposal, the questions that separate a production plan from a demo plan are unglamorous:
- What happens when the system is not confident, and who receives it?
- How will we know if quality degrades after launch, without a customer telling us?
- What does this cost per month to run, at our actual volume, not at demo volume?
- Who owns it after handover, and what does that person do on the day it breaks?
- Show me it running against a random sample of our real data, not a chosen one.
A supplier who has built production systems will have ready answers, and some of those answers will be caveats. A supplier who has only built demos will find the questions surprising. That difference is more informative than any demo.