The Probabilistic Contract

In April 2025, a provider-side model update silently reached 180 million users and began systematically endorsing bad decisions — affirming plans to stop psychiatric medication, praising demonstrably poor ideas with unearned enthusiasm. The provider's own alerting didn't catch it. Power users on social media did. The rollback took three days.^[8] Every latency metric was green. Every error rate was nominal. Every on-call engineer had nothing to page on. The model was simply, quietly, confidently wrong — and the organization had built no infrastructure capable of knowing the difference.

That story is not an edge case. It is the template for how LLM failures actually unfold in production, and it exposes a fault line running through every engineering team that has shipped an AI feature in the last two years: the entire operational model — how you alert, how you define incidents, how you write SLAs, how you structure on-call — was engineered for a world where the same input always produces the same output. That world no longer exists for your most user-facing features. And most teams haven't noticed yet, because the failure is invisible to the tools they trust.

This paper makes a specific argument: enterprise engineering teams are systematically misclassifying LLM failures not because they lack good models or smart engineers, but because their operational contract — the shared agreement between product, engineering, and the business about what "working" means — was written for deterministic systems and has never been rewritten. The teams winning in production aren't the ones with better foundation models. They're the ones that rebuilt their operational assumptions first.

The Deterministic Contract and Why It Fails

The deterministic contract is so deeply embedded in engineering culture that most teams don't even name it. It's the invisible assumption underneath everything: a function, given the same arguments, returns the same value. A service, given the same request, returns the same response. An incident, therefore, is when that stops being true — when the function throws, when the service returns a 5xx, when the latency spikes above a defined threshold. SLAs are written around these events. Runbooks are written to diagnose and reverse them. On-call rotations are staffed to respond to them.

Deterministic systems fail loudly. That's the hidden gift that engineers have taken for granted for decades. When your Postgres query suddenly takes 40 seconds instead of 40 milliseconds, every monitoring tool you have screams. When your API starts returning 503s, PagerDuty lights up. The failure is legible — it announces itself in a format that your existing infrastructure is built to receive and route.

Probabilistic systems fail quietly. An LLM feature can degrade gradually over six hours while all your SLO metrics stay green.^[8] The failure may be behavioral — the model became subtly more agreeable, or began hallucinating product names, or started producing outputs that are technically coherent but factually wrong in ways that matter enormously to your users — rather than functional. No 5xx. No latency spike. No error log entry. Just a slow, invisible erosion of quality that your users feel before your dashboards do.^[1]

The distinction matters because business infrastructure is built on the deterministic assumption just as much as technical infrastructure is.^[5] Payroll runs on fixed rules. Regulatory reporting follows explicit formulas. SLA routing operates on defined thresholds. When a company layers an LLM onto workflows built on this logic, they inherit a collision: the system beneath expects repeatability, and the model above is structurally incapable of providing it. Most teams resolve this tension by pretending it doesn't exist — and shipping the probabilistic component under a deterministic SLA.

How the Misclassification Happens

The failure mode is specific and repeatable. A team ships an LLM feature. They instrument it with the same observability stack they use for everything else: latency p50/p95/p99, error rate, throughput, token count. These metrics all look fine. They define an SLA around uptime and response time — the only dimensions their existing SLA framework supports. They write a runbook for "LLM feature down" that covers API timeouts, rate limiting, and prompt injection attacks. They put it in the on-call rotation.

What they have not done: defined what "quality" means for this feature in measurable terms, established a baseline for that quality, built alerting that triggers when quality drops below the baseline, or created an incident classification for "the model is producing subtly wrong outputs." The runbook has no entry for "the model was correct yesterday." The on-call engineer has no playbook for a degradation that doesn't manifest as an error.

This is not negligence. It is a rational response to the tools available and the operational frameworks inherited. The problem is that LLM failures don't fit the taxonomy those frameworks were built around. When teams rely on ad-hoc scores, gut-check metrics, or spreadsheets to assess output quality, they are left with "quality blind spots" — wasting hours hunting down context and leaving every result in doubt.^[1] The misclassification is structural: quality degradation gets filed as "no incident" because it triggers no existing incident type.

The LLM Supply Chain Problem Nobody Budgeted For

There is a second layer to this problem that compounds the first, and it lives outside your codebase entirely. Hosted LLM services evolve continuously through provider-side updates — without explicit version changes, without changelog notifications, and without any mechanism that triggers your existing deployment governance.^[3] You did not deploy a new version. You did not change a line of code. But the model your application is calling today is behaviorally different from the model it was calling last Tuesday.

Research from NTNU documents this precisely: silent provider-side updates can introduce behavioral drift, causing regressions in functionality, formatting, safety constraints, or other application-specific requirements.^[3] Prior work shows these silent updates cause measurable regressions — but the operational frameworks most deployer organizations have built do not include any mechanism for detecting or governing them, because they were designed around a model where you control what gets deployed and when.

The practical consequence: your LLM system's behavior is now partially determined by a third party operating on a timeline you cannot observe, following an update policy you have no visibility into. This is a supply chain problem, not a prompt engineering problem. And it requires supply chain governance tooling — production contracts that define how the model is allowed to behave, risk-category-based testing suites, and release checkpoints that block behavioral regressions from reaching production.^[3] Almost no Series B–D company has built this. Almost all of them are exposed to it.

What a Probabilistic Operational Contract Actually Looks Like

The teams that are winning in production have done something that sounds obvious in retrospect and is genuinely difficult in practice: they have rewritten the operational contract between product, engineering, and the business to match the nature of the system they are actually running. Here is what that rewrite looks like across four dimensions.

1. Quality as a First-Class SLA Dimension

Traditional SLAs measure availability, latency, and error rate. These are necessary but not sufficient for LLM features. A probabilistic SLA adds quality dimensions: faithfulness (does the output accurately reflect the source material?), relevance (does the output address what the user asked?), safety (does the output avoid harmful content?), and task-specific correctness metrics defined by the product team for each feature.^[1]

The critical operational change is that these quality dimensions must have defined thresholds and automated alerting, just like latency does. "Quality dropped below 0.78 faithfulness score on the document summarization feature" needs to generate a page with the same urgency as "p99 latency exceeded 2 seconds." Most teams have not built this alerting. The tools to build it now exist and are mature enough for production use.

Unlike traditional SLAs, AI-specific agreements must address model accuracy, explainability, and the probabilistic nature of outputs — not just uptime and response time.^[6] The framing shift is significant: you are no longer guaranteeing that the system is up. You are guaranteeing that the system is performing above a defined quality floor, measured continuously, with automatic escalation when it drops below that floor.

2. Behavioral Incident Classification

The standard incident severity taxonomy — P0 (service down), P1 (major degradation), P2 (partial degradation), P3 (minor issue) — was built around functional failures: the thing stops working. LLM features require a parallel taxonomy for behavioral failures: the thing keeps working, but starts working badly.

A behavioral incident classification looks like this: B1 (systematic harmful output, immediate rollback required), B2 (measurable quality regression above threshold, engineering response within 2 hours), B3 (quality drift trending negative, investigation required within 24 hours), B4 (quality metric anomaly, monitoring ticket created). The B-tier incidents require different detection (continuous evaluation, not error monitoring), different diagnosis (four-layer root cause analysis across retrieval, prompt, model, and output layers^[8]), and different remediation (prompt revision, retrieval tuning, or model version pinning, not service restart).

3. Continuous Evaluation as Production Infrastructure

The operational model for deterministic systems treats testing as a pre-deployment gate. You test before you ship; once it's shipped, monitoring takes over, and monitoring watches for functional failures. For probabilistic systems, this model breaks because the system's behavior continues to change after deployment — through provider updates, through distribution shift in user inputs, through changes in the retrieval corpus, through prompt drift as the system is modified incrementally over time.

Effective LLM monitoring requires a continuous feedback loop: evaluations running in production, not just in CI. This means instrumenting every production trace, running automated quality evaluations against each trace using a defined metric suite, tracking quality trends over time, and auto-curating evaluation datasets from production traffic to feed back into regression testing cycles.^[2] The observability framework for this is built on OpenTelemetry standards, with quality metrics layered on top of standard distributed tracing.^[1]

The key architectural principle: traditional observability tells you whether a request succeeded. LLM observability tells you whether the response was good enough.^[2] Both are necessary. Most teams have only built the first.

4. Deployer-Side Governance for Provider Updates

The fourth dimension requires the most organizational lift and delivers the most protection against silent regressions. Deployer-side governance means: before any provider-side model update affects your production traffic, you test it against a defined behavioral contract and block it from reaching users if it fails.

This requires three components working together: production contracts that define the behavioral envelope the model must operate within (format, safety, task-specific correctness), a risk-category-based testing suite that validates the model against those contracts, and release checkpoints that gate production traffic on passing the test suite.^[3] For teams using hosted APIs, this translates to: pin model versions where the provider supports it, run a behavioral test suite on every new version before migrating traffic, and treat a model version migration as a deployment event with the same governance your code deployments receive.

The Organizational Architecture Problem

Technical changes are necessary but not sufficient. The deeper problem is that the probabilistic contract requires renegotiating responsibilities that span organizational boundaries — and most companies haven't had that conversation.

In a deterministic world, the split is clean: engineering owns uptime and latency, product owns feature requirements, the business owns SLA commitments to customers. In a probabilistic world, quality is a dimension that none of these groups owns cleanly. Engineering can build the infrastructure to measure it, but they cannot define what "good enough" means for a customer-facing summarization feature — that's product. Product can define quality criteria, but they cannot write the evaluation metrics or set the alerting thresholds — that's engineering. The business has made SLA commitments to enterprise customers that were written without any concept of output quality as a measurable dimension.

The companies that have solved this have done so by creating an explicit cross-functional process for quality governance: product defines quality criteria per feature in plain language; engineering translates those criteria into automated evaluation metrics; both groups review quality dashboards in a regular operational review (weekly at minimum, not quarterly); and the business-facing SLA is updated to include quality floor commitments that are actually tied to the metrics engineering can measure. This is not technically sophisticated. It is organizationally disciplined, and discipline is harder to ship than code.

Dimension	Deterministic Model	Probabilistic Model
SLA definition	Uptime %, latency p99, error rate	Uptime % + quality floor score per feature, behavioral envelope definition
Incident trigger	Error rate spike, latency breach, service unavailability	Above + quality score below threshold, behavioral drift detected, safety violation
Incident taxonomy	P0–P3 by severity of functional failure	P0–P3 for functional + B1–B4 for behavioral failures
On-call runbook	Diagnose error, identify root cause, rollback or fix	Four-layer diagnosis (retrieval → prompt → model → output), behavioral rollback, provider version audit
Testing cadence	Pre-deployment gate in CI/CD	Pre-deployment gate + continuous production evaluation + provider update testing
Quality ownership	Engineering (functional correctness only)	Cross-functional: product defines criteria, engineering measures, both review
Deployment governance	Controls your code deploys	Controls your code deploys + provider model version migrations

The Role of the AI SRE

One organizational response to this challenge is the emergence of the AI SRE — a site reliability engineering function specifically equipped to operate probabilistic systems. The traditional SRE role was built around the assumption that failures are reproducible: you can reproduce the failure in a test environment, identify the root cause, and prevent recurrence by fixing the specific defect. LLM failures often cannot be reproduced deterministically, because the model's output is a sample from a probability distribution, not a deterministic function.^[7]

The AI SRE function extends traditional SRE responsibilities in three specific ways. First, it treats logs, diffs, runbooks, and postmortems as semantic inputs — using LLMs themselves to interpret incident histories and generate hypotheses about behavioral regressions.^[7] Second, it maintains the behavioral test suite that governs provider updates, updating it continuously as the application evolves. Third, it owns the quality review cadence — the regular operational review where quality metrics are examined across all LLM features, trends are identified, and investigations are opened before users notice problems.

Most Series B–D companies do not have a dedicated AI SRE function. Most do not need a full headcount; they need someone in the existing SRE or platform engineering team to own these responsibilities explicitly, with the organizational mandate and tooling to execute them. The gap is rarely capability — it is authority and assignment.

The Diagnostic: Are You Running a Probabilistic System on a Deterministic Contract?

If you answered "no" to four or more of these questions, you are running a probabilistic system on a deterministic operational contract. You are not detecting the quality regressions you are shipping. The question is not whether you have had a silent degradation event — it is whether you were positioned to know about it when it happened.

What To Do This Quarter

The temptation when diagnosing an organizational problem is to recommend a multi-year transformation program. That is the wrong prescription here. The probabilistic contract problem is urgent — silent quality regressions are happening now, not in the future — and the most important interventions can be sequenced into a 90-day program that delivers meaningful protection without requiring new headcount or new tooling budgets.

Week 1–2: Establish a quality baseline. Pick your highest-traffic LLM feature. Define three to five quality dimensions for it in plain language with product. Choose evaluation metrics that map to those dimensions — faithfulness, relevance, and task-specific correctness are a sensible starting point.^[1] Run those evaluations against the last 30 days of production traces. This baseline tells you what "normal" looks like. You cannot detect drift without it.

Week 3–4: Instrument continuous evaluation. Deploy an observability layer that runs your quality metrics against production traces continuously — not just in CI. The tooling to do this is mature: OpenTelemetry-based LLM observability platforms that instrument your inference calls with one line of code and run quality evaluations against every trace are widely available.^[1] Set alerting thresholds based on your baseline. Wire the alerts into your existing incident management tooling.

Month 2: Rewrite the runbook. Add a behavioral incident classification (B1–B4) to your severity taxonomy. Write the on-call runbook for each tier, using the four-layer diagnosis framework: start with retrieval, then prompt, then model version, then output post-processing.^[8] Run a tabletop exercise simulating a B2 incident — quality score dropped 15% overnight, no errors, all functional metrics green — and identify the gaps in your diagnosis and remediation process before a real event surfaces them.

Month 3: Implement provider update governance. Audit which of your LLM providers support model version pinning and pin every production endpoint. For providers that do not support pinning, build a canary traffic mechanism that lets you test new model versions against your behavioral contract before migrating all production traffic. Draft the production contract document: the formal definition of how the model is permitted to behave, by feature, in terms your engineering team can automate tests around.^[3]

Ongoing: Weekly quality review. Stand up a 30-minute weekly quality review — engineering lead plus product lead per LLM feature. Review quality metric trends, open investigations on negative trends, close investigations from prior weeks. This meeting is the organizational mechanism that closes the loop between continuous evaluation and human judgment. It does not require new tooling. It requires discipline and a standing calendar invite.

The probabilistic contract is not a technology problem awaiting a technology solution. The observability tools are ready. The evaluation frameworks are mature. The governance patterns have been documented. What's missing, in almost every Series B–D company we've worked with, is the organizational decision to treat LLM quality as a first-class operational concern — worthy of the same infrastructure investment, the same incident discipline, and the same cross-functional ownership as the uptime and latency SLAs that have governed engineering operations for the last two decades. Make that decision explicitly. Write it down. Then instrument it.

The teams that figure this out first will not just ship better AI features. They will have built the operational muscle to iterate on probabilistic systems without the fear that every change might introduce a quality regression they'll only discover when a user tweets about it. That confidence is a compounding competitive advantage. The teams that don't build it will keep shipping silent regressions into production — architecturally incapable of knowing they're doing it.