Production engineering / NOTE.005

From AI prototype to production: the work the demo skipped

A prototype proves that a narrow path can work under controlled conditions. Production is the system of controls that keeps the path useful when inputs, users, vendors, and business rules change.

By Kurt MoralesAugust 28, 20269 min read

Confirm the product decision

Before hardening anything, confirm that the prototype changed the relevant evidence. Did it reduce handling time, improve consistency, unlock a previously impractical task, or reveal that the workflow was not suitable? Avoid treating technical feasibility as automatic permission to scale.

Name an owner for the business outcome and an owner for the running system. They may be different people, but neither role can be left implicit. The business owner defines acceptable behavior; the system owner manages reliability, access, changes, and incidents.

Rebuild the data boundary

Prototype data is often manually selected, cleaned, and moved. Production needs documented sources, permissions, retention, deletion, regional constraints, and rules for sensitive fields. Minimize what is sent to each service and preserve a clear reason for processing it.

Verify vendor terms and data controls for the exact service and account configuration being used. A model name alone does not answer whether inputs are retained, used for improvement, logged, or transferred across regions.

Make evaluation a release gate

Convert the prototype evaluation into a repeatable suite. Include representative cases, known edge conditions, prohibited outcomes, and operational checks. Store expected results or review rubrics with version history so changes can be compared rather than judged from memory.

Run evaluation when prompts, retrieval data, models, parsing rules, or surrounding code change. Define which regressions block release and who can accept a tradeoff. Production quality is a managed threshold, not a one-time score.

Task quality on representative and difficult examples
Unsupported or unsafe output checks
Latency and failure behavior under expected load
Cost per completed workflow item
Reviewer effort and escalation rate

Engineer failure and fallback paths

External models time out, return malformed output, reject requests, and change behavior. Use bounded retries, timeouts, schema validation, idempotent operations, and a queue where appropriate. Do not let a retry send the same message or update the same record twice.

Define degraded behavior. The workflow might pause, route to manual processing, use a simpler deterministic method, or display a clear unavailable state. A fallback should preserve source data and accountability rather than silently substituting lower-quality output.

Add observability that explains outcomes

Log request identifiers, configuration versions, model and retrieval versions, timing, validation results, and final workflow status without exposing unnecessary sensitive content. Metrics should connect technical events to completed business items.

Sample outputs for quality review and monitor changes in input mix, escalation, correction, cost, and latency. An uptime dashboard can stay green while the system becomes less useful. Quality monitoring requires operational reviewers, not only infrastructure alerts.

Release gradually and preserve rollback

Start with internal users, a limited workflow segment, or draft-only behavior. Compare results with the existing process and expand only when the evidence supports it. Feature flags and versioned configuration make exposure easier to control.

Keep a tested way to disable the automated path and resume manual operation. Document incident ownership, communication, and recovery steps. Production readiness is demonstrated when the team can operate the system safely on an ordinary bad day, not only when the demo succeeds.

The distance from prototype to production is mostly operational engineering: ownership, controls, evaluation, failure handling, monitoring, and reversible release.