A financial services company deployed an AI document review system in the first quarter of last year. The system processed loan applications: it reviewed documents, flagged issues, and made recommendations that underwriters acted on. By most operational measures, it was working. Processing times were down. Volume was up. The executive summary looked strong.
Four months in, a vendor model update was pushed silently. The provider's release notes described it as an "improvement." For most input types, it was. For one specific category of loan documents -- a minority of the total volume but a significant one -- it degraded performance by roughly 30%. The underwriters who knew their domain well had quietly started ignoring those particular recommendations. The ones who did not know the domain as well were following them.
The company had no monitoring that would have caught this. No evaluation infrastructure that ran against a benchmark. No alerting on output quality changes. The signal that something had gone wrong was one alert underwriter who mentioned it in a team meeting eight months after deployment -- four months after the degradation began. The company had been making consequential financial decisions on degraded AI outputs for four months.
Three Questions Most Teams Cannot Answer
Before discussing the failure modes in detail, it is worth establishing a baseline diagnostic. Most teams running AI systems in production cannot answer three basic questions about those systems.
Teams without measurement infrastructure answer these questions with some version of: "We would hear from users." That is the most expensive quality assurance mechanism available -- it means users are finding the bugs, absorbing the consequences, and deciding whether to keep using the system before the team knows anything is wrong. By the time the signal reaches the team through user complaints, the problem has already produced downstream effects that are much harder to measure and remediate than the output quality issue that caused them.
The Three Failure Modes
Model drift
LLM providers update their models continuously. These updates are not always fully documented. Providers typically describe them as "improvements" -- which is often accurate in aggregate while being inaccurate for specific use cases. A model optimized for a slightly different distribution of training data may be better on most tasks and demonstrably worse on some. The provider has no way to know which tasks are "some" for your specific application without knowing your specific application.
The degradation from a model update is rarely catastrophic. That is what makes it dangerous. A 30% performance decline on 15% of input types -- as in the financial services case -- does not produce obvious system failures. It produces subtly wrong outputs on a minority of cases, handled by users in ways that generate no immediate error signal. The system is running. The metrics are green. The outputs are wrong.
Teams without evaluation infrastructure cannot detect this. Teams with evaluation infrastructure detect it within hours of the update -- because the evaluation suite runs automatically on every deployment event, including implicit ones triggered by upstream model changes. The difference is not the degradation. The degradation happens to both teams. The difference is four months of operating on degraded outputs versus a same-day alert and rollback.
The mechanism for detecting model drift is straightforward: a versioned evaluation suite, a baseline score from the last known-good deployment, and an alerting rule that fires when the current score falls below threshold. Building this takes 2-4 weeks. Discovering the degradation through user reports takes longer and produces worse outcomes. The teams that build evaluation infrastructure before launch are not being cautious -- they are making the iteration loop safe, which is what allows them to move fast sustainably.
Distribution shift
Production traffic does not look like your test set. It never does, and the divergence grows over time as users find patterns you did not anticipate, as the user base expands to include people who interact with the system differently than your initial users, and as seasonal or contextual patterns emerge that were not represented in the original data.
A system that handles 95% of test inputs correctly and 80% of production inputs correctly is not failing obviously. The 15-point gap is not a disaster -- it is a normal property of AI systems deployed into real-world conditions. The problem is not the gap. The problem is not knowing the gap exists. Teams without production monitoring are often confident in their system's performance based on test set results that represent a distribution the system will never see in production at the same frequency.
Distribution shift is harder to detect than model drift because there is no discrete event to trigger an evaluation run. The shift is gradual. The solution is continuous monitoring: a sample of production outputs evaluated against quality criteria on a rolling basis, with alerting on sustained quality decline rather than single-event triggers. The evaluation sample does not have to be comprehensive -- a 5% sample of production traffic, evaluated automatically, is sufficient to detect population-level quality changes within days rather than months.
The practical challenge with distribution shift monitoring is that it requires ground truth: some mechanism for knowing whether a sampled output was actually correct. For some applications this is available directly -- loan decisions that proved correct or incorrect, recommendations that were accepted or rejected, classifications that were confirmed by downstream events. For others it requires a human review sample. The important thing is having some systematic signal, however partial, rather than no signal at all.
Prompt erosion
System prompts accumulate changes over the lifetime of a production system. Each change is locally rational: a patch for an edge case that was causing problems, a clarification for a term that the model was interpreting inconsistently, an instruction added after a user complaint, a constraint added after a compliance review. After twelve months of this, the prompt has instructions that conflict with each other, edge case handling that overrides base behavior in ways nobody intended, and redundant instructions that create ambiguity about which one takes precedence.
Nobody who touches the system has a clear mental model of what the current prompt instructs. The prompt has grown beyond anyone's ability to hold in working memory. Engineers making changes are doing local reasoning -- "this change fixes the problem I can see" -- without awareness of how it interacts with the accumulated prior changes. The prompt works, mostly, until it encounters a combination of conditions that triggers one of the conflicts, at which point the output is wrong in a way that is extremely difficult to trace back to a specific change.
Prompt erosion is the silent failure mode that develops in almost every production AI system that receives ongoing attention without disciplined version control and evaluation. It is not caused by bad engineering. It is caused by the absence of the infrastructure that would make it visible -- specifically, the absence of a versioned evaluation suite that runs against the same benchmark on every prompt change.
The fix for prompt erosion is not to write better prompts initially. It is to treat prompt changes like code changes: version controlled, reviewed, and evaluated against a consistent benchmark before deployment. When every prompt change runs the evaluation suite and produces a quality score, erosion becomes visible. A change that drops the score triggers a review. A change that holds or improves the score is safe to deploy. Without this gate, every prompt change is a bet made without information.
The Competitive Consequence
Teams that cannot measure cannot safely iterate. Every prompt change is a bet they cannot vet. Every model update is an event they cannot evaluate before it reaches users. The feedback loop for discovering a bad bet is measured in weeks rather than hours. The natural response -- the rational one -- is to make fewer changes. Systems calcify. Teams stop pushing improvements because every change feels dangerous.
This calcification is the real competitive problem. The financial cost of degraded outputs -- incorrect loan recommendations, missed document flags, suboptimal user interactions -- is measurable and bad. But it is bounded by what the degraded system does. The cost of not iterating on a system because iteration feels dangerous is unbounded and compounding. Competitors who built measurement infrastructure first can run tight iteration loops: test, measure, ship, repeat. Competitors who did not are afraid to touch production. The capability gap grows every sprint.
McKinsey's research on AI operations identifies "inability to measure performance" as the top operational blocker for teams trying to iterate on production AI systems. This is not a surprising finding -- the constraint is obvious to anyone who has tried to improve a system they cannot measure. The surprising finding is how few teams have addressed it: the same research finds that fewer than 30% of companies running production AI systems have systematic quality monitoring in place. The other 70% are doing the financial services equivalent of discovering degradation through one alert employee eight months in.
What Good Looks Like: Four Steps
Step 1: Define correct before you build
Not "the summary should capture the main points" -- that is not a testable definition. Not "the recommendation should be good" -- that is not a definition at all. Define "correct" precisely enough that a benchmark can evaluate it without human judgment on each individual case. This forces answers to questions like: what are the required components of a correct output? What are the disqualifying components of an incorrect one? What is the minimum acceptable quality on a scale that can be computed automatically?
This step is harder than it sounds. Most teams discover they do not have a crisp definition of correct until they try to write one. The process of writing the definition surfaces implicit assumptions, unresolved disagreements about what the system should optimize for, and edge cases that require explicit policy decisions before they can be encoded in a benchmark. This is uncomfortable. It is also the work that makes everything that follows valid. A benchmark built on a vague definition of correct is not a measurement -- it is a ritual.
Step 2: Build a human-labeled evaluation set from real production inputs
200-500 examples is sufficient to detect meaningful quality changes reliably. Not synthetic test cases designed to cover scenarios you thought of in advance -- actual inputs that represent the distribution you expect in production, labeled by humans who know what "correct" looks like for your specific application. The labeling investment is 2-4 weeks of work, primarily human time for annotation. It is not glamorous. It is also the foundation that every subsequent quality measurement rests on.
The evaluation set should be treated as a product asset: versioned, maintained, expanded as production distribution evolves. New input categories that emerge in production should be added to the evaluation set as they appear. The evaluation set that is not maintained drifts from the production distribution and eventually stops measuring what it claims to measure.
Step 3: Automate evaluation as a deployment gate
Run the evaluation suite on every deployment that touches the model, system prompt, tool schema, or retrieval configuration. Automate the gate: if the evaluation score drops below a defined threshold, block the deployment and alert. This turns evaluation from a periodic audit into a continuous safety mechanism. It also makes the development loop faster, not slower -- engineers get feedback within minutes on whether a change improved or degraded quality, rather than discovering the answer through user reports weeks later.
The tooling landscape for this has matured substantially. Langfuse, Braintrust, and Arize all support automated evaluation pipelines with CI/CD integration. The integration work is 1-2 weeks for most stacks. The value of that integration accrues on every subsequent deployment, indefinitely.
Step 4: Version everything and define rollback precisely
System prompt, tool schema, model version, retrieval configuration -- all of it versioned, all of it included in a named deployment artifact that has a known evaluation score. Rollback should mean returning to a specific named set of components with a documented quality score, not "the version from last Tuesday" or "the way it was before that last change." Vague rollback definitions produce vague rollbacks that may or may not restore the quality that was lost. Precise rollback definitions produce precise restorations.
| Failure Mode | Detection Without Monitoring | Detection With Monitoring | What Monitoring Requires |
|---|---|---|---|
| Model drift | User complaints; weeks to months | Automated evaluation on update event; hours to days | Versioned eval suite + deployment event hook |
| Distribution shift | User complaints; inconsistent signal | Rolling production sample + quality monitoring; days | 5% production sample + ground truth signal |
| Prompt erosion | Tracing failure to specific change; often impossible | Score delta on every prompt change; immediate | Eval suite run on every prompt commit |
The Build-Before-Ship Principle
The question "how do I know my AI is working correctly in production?" has a concrete answer. The answer is measurement infrastructure: an evaluation suite, a production sample pipeline, versioned deployments, and automated quality gates. None of these components require novel technology. All of them require upfront investment before they deliver visible returns. None of them can be bolted on effectively after a production incident has already revealed their absence.
The principle that follows: if you cannot answer "how will I know if this degrades?" before launch, you are not ready to launch. You are ready to guess -- to deploy a system and hope that your initial quality holds, that the model provider does not push an update that hurts your use case, that the production distribution does not shift in ways that expose gaps in your test set. Guessing is not a launch posture. It is an unacknowledged risk that accrues until the moment it becomes visible, at which point it is substantially more expensive to address than if it had been addressed before the system went live.
The financial services company whose story opened this essay eventually built the measurement infrastructure. It took four months after the incident. They now know within 24 hours if a model update affects their output quality. They have rollback capability that takes hours rather than days. Their deployment cycle is faster, not slower, because engineers can validate changes before they ship rather than discovering problems after. The measurement infrastructure they built reactively, under pressure, after a costly incident, is the same infrastructure they could have built proactively, before launch, at lower cost and without the incident in between. Both paths arrive at the same destination. Only one of them passes through four months of degraded outputs.