The gap between a good demo and a shipped feature
Every company we talk to in 2026 has already seen the demo. Someone on the team wired an LLM to a spreadsheet, or built a chatbot on internal docs over a weekend, and it looked great in the all-hands. Then it sat in a Slack channel for three months, unused in production, because nobody could answer the question: what happens when it's wrong in front of a customer, or worse, wrong in front of an auditor?
That gap — between "the model produced a good answer once" and "this runs unattended against real data, for real users, every day" — is where almost all of our AI/ML engagement time actually goes. Building the prompt is an afternoon. Making it safe to leave running is the project.
Here's what actually separates a demo from a production system, and what it costs to close that gap.
Demos skip the parts that make things reliable
A demo is one input, run once, by someone who knows how to phrase the question. Production is thousands of unpredictable inputs, run continuously, by people who don't know or care how the model works. Three things break first when you make that jump:
1. There's no way to know if a prompt change made things better or worse. Without a fixed set of test cases and expected outputs — an eval set — every prompt tweak is a guess. We build an eval set from real user questions before we touch a production prompt, and we re-run it on every model or prompt change. If a client can't tell us the pass rate moved from one number to a better one, they can't safely ship a change at all, they're just hoping.
2. The model is only as good as what it can retrieve. For anything grounded in your own data — support docs, contracts, internal wikis — retrieval quality matters more than which model you're calling. Naive chunking and a single vector search get you a demo that works on the three questions you tried. Production RAG needs hybrid search (keyword plus vector), reranking, and a hard requirement that every answer cites its source document. If the system can't point to where it got an answer, don't ship it to anything customer-facing.
3. Tool-using agents need a permission model, not just a prompt. The moment an agent can create a refund, update a CRM record, or send an email on someone's behalf, "the model decided to" stops being an acceptable explanation. Every action needs a defined blast radius: what the agent can do autonomously, what needs a human approval step, and what it's never allowed to touch. We maintain an explicit no-go list for every agent we build — usually anything irreversible or financial — before we write the first tool definition.
The failure mode nobody plans for: confident wrong answers
An API outage is obvious. A model that answers fluently and incorrectly is not. This is the specific risk that kills trust in an AI feature faster than anything else — one wrong answer delivered with total confidence, and the team that championed the project spends the next quarter defending it.
The fix isn't a better model. It's structural: require citations on anything grounded in your data, add a confidence threshold that routes uncertain answers to a human instead of guessing, and log every prompt, retrieval, and completion so you can reconstruct exactly what happened when something goes wrong. We treat this logging layer as non-negotiable, the same way we'd treat error logging on a payments flow. Nobody notices it until the day they desperately need it.
Internal automation is the fastest AI ROI right now
Customer-facing agents get the attention, but the highest-certainty AI work we do in 2026 is internal: triaging inbound support tickets before a human sees them, summarizing sales calls into CRM notes, extracting line items from invoices and contracts, and letting staff ask questions against internal docs instead of pinging a Slack channel. The stakes are lower — an internal tool that's occasionally wrong gets corrected by a colleague, not a customer — so the guardrails can be lighter and the payback is faster.
A scoped RAG assistant over a company's own documentation is realistically a $7,000+ engagement with a working prototype in 2 to 4 weeks, and a production-hardened version — auth, logging, an eval set, source citations — in 8 to 12 weeks. Tool-using agents wired into real systems (a CRM, a ticketing system, an ERP) start around $20,000+, because the permissioning and testing work scales with what the agent is allowed to touch. Pure workflow automation — no model in the loop, just reliable data movement between tools — starts lower, from $3,000 for a single integration.
What we require before writing the first prompt
On every AI engagement, four things get decided before any code ships, not after:
- An owner for the eval set. Someone whose job it is to notice when quality drops, not "the team" in the abstract.
- A documented no-go list. The specific actions the system is never allowed to take without a human confirming first.
- A cost ceiling. Per-user or per-day token budgets, with alerts before the bill surprises anyone. Cheap-model triage plus frontier-model escalation, not frontier model on everything.
- A rollback plan. Model providers change behavior under a stable API name more often than teams expect. If accuracy drops after a silent vendor update, you need a previous prompt version and a previous eval score to compare against.
Skip any of these and the project doesn't fail loudly — it fails quietly, weeks after launch, when someone notices the system has been confidently wrong for a while and nobody was watching.
The bottom line
The model is rarely the bottleneck in 2026. Frontier APIs from OpenAI and Anthropic, paired with retrieval frameworks like LlamaIndex and vector stores like pgvector, are good enough for the overwhelming majority of business use cases. What determines whether an AI feature survives contact with real users is the boring engineering around it: evals, grounding, permissions, logging, and a cost model that doesn't surprise finance.
That's the work our AI & machine learning team does — turning a working demo into a system a business can actually run unattended. If the internal automation side of this is your bigger itch — connecting tools and erasing repetitive work rather than building a new AI feature — our automation & integrations team scopes those the same way: prototype fast, harden before it touches anything that matters.
