A chatbot that answers from a PDF in a sales meeting is a demo. A feature that answers from the right documents, for this user, without inventing policy, at a cost you can live with, is a product.
The gap is not "better prompting." It is software: retrieval, auth, evaluation, logging, and a fallback when the model is wrong. Myrafy's AI work is that gap - not a research lab, not a wrapper with a logo.
If you need the retrieval pattern itself, read RAG architecture for production. This article is the release bar: what must be true before you put an LLM in front of customers or staff.
The Demo Lies in Predictable Ways
Demos are curated: clean PDFs, one happy user, no latency budget, the founder asking questions the deck already answered. Production has:
- Stale, conflicting, or permissioned documents
- Users who ask out of scope ("write me a legal opinion")
- Prompt injection in uploaded files and tickets
- Token bills that only show up after marketing turns the feature on
- A support team that will be blamed for fluent nonsense
Treat the demo as a spike, then pay for discovery that names sources, roles, and failure modes. Do not skip from the spike to a six-month "AI transformation."
Production Means a Source of Truth
Write down:
- What the model is allowed to know. Which systems, which collections, which dates.
- What it must not do. Legal advice, medical advice, competitor dump, PII echo - domain-specific.
- Citations. If the user cannot see where an answer came from, you do not have a grounded feature; you have a confident intern.
No source of truth โ no RAG, no "fine-tune it on our Slack." Fine-tunes do not replace permissions.
For customer-facing bots, the same rule as OpenAI chatbots for business: the wrapper is easy; policy-safe answers are the work.
Evals Before Rollout, Evals After Every Change
You cannot regression-test an LLM like a calculator, but you can stop flying blind.
A minimum eval set:
| Slice | Example |
|---|---|
| Should answer | Top 30 real questions with a gold snippet or citation |
| Should refuse | Out-of-policy, medical/legal, "ignore your instructions" |
| Permission | User A must not retrieve User B's file |
| Freshness | Question whose answer changed last month |
| Adversarial | Prompt injection in a document title or PDF footer |
Score citation faithfulness and refusal quality, not just "sounds good." Run the set in CI on prompt, chunking, or model changes - the same discipline as any other release, even if the assertion is a threshold, not a string match.
If nobody owns the eval set, you do not have an AI feature. You have a demo on a schedule.
Permissions Are Not a System Prompt
"You are a helpful assistant. Only use the user's documents" is not access control.
Enforcement belongs in retrieval and tools:
- Filter chunks by ACL before they enter the prompt
- Tool APIs use the same auth as the rest of the product (JWT / session rules still apply)
- Audit: who asked, what was retrieved, what was returned
Healthcare and finance buyers should assume a prompt can be extracted. Design as if it will be.
Latency, Cost, and a Boring Fallback
Users will not wait eight seconds for a paragraph that might be wrong.
- Stream tokens when the UI is a chat (SSE is often enough)
- Cap context; rerank; do not stuff the PDF - see LLM cost optimization
- Fallback: keyword search, a canned "I don't know - here's the doc," or a ticket. Silence and hallucination are both failures; hallucination is worse.
Rate-limit expensive routes. An unbounded chat box is a billing incident (API rate limiting).
Observability for Prompts
You need traces for retrieval + model + tools (OpenTelemetry on Node if that is your stack), plus:
- Token counts per feature (not one shared OpenAI key for the whole company)
- Sampled prompt/response storage with retention and PII policy
- User feedback (thumbs, "this cited the wrong policy") that feeds the eval set
Do not log full customer documents to a vendor debugger by accident. Redact.
What to Put in the SOW
When you hire a company to ship this, the SOW should include:
- Named corpora and owners
- Eval set size and pass bar
- Auth model for retrieval
- Cost cap / model routing
- Human fallback or review path
- What "done" means for v1 (often: one workflow, not "an AI platform")
If the proposal is only "integrate GPT" with a screenshot, it is a demo estimate.
Rollout: One Workflow, Then Expand
v1 should be a single job: "answer from the HR policy set," "draft a support reply from the knowledge base," "summarize this ticket for a human." Measure that job. Then add a second.
A "platform" with six personas and no evals is how demos die in production. The same sequencing we use on product engineering: smallest useful increment, weekly demo, written acceptance.
Staff who will live with the feature should sit in discovery. If only leadership saw the demo, you will rediscover requirements after launch.
When Not to Use an LLM
Skip generation when a form, a classifier, or a search box will do. LLMs are expensive, non-deterministic, and politically visible when they fail. Use them where language is the interface and you can bound the knowledge.
That is not anti-AI. It is how features stay on after month two.
The Bottom Line
Shipping Production AI is product engineering: source of truth, evals, ACL-aware retrieval, fallbacks, and unit economics. The model is a dependency. The product is everything around it.
If you cannot name those pieces, you are not ready for customers. You are ready for a capped discovery.
Adding a grounded assistant or automation to an existing product? See Myrafy AI services or contact us for a free estimate.