The conventional reliability engineering toolkit was built for systems that behaved the same way each time given the same input. AI agents do not behave the same way each time given the same input. The classic tools - unit tests, integration tests, deterministic replay, traditional monitoring - all assume a property that the systems being operated do not have. This mismatch is not a small operational annoyance; it is the central challenge of running AI agents in production, and the patterns for handling it are still being worked out.

This is a working note on what those patterns look like in 2026.

TL;DR

  • Agent non-determinism is structural, not a bug. Same input produces different outputs because the underlying model is sampling, not because something is broken.
  • Traditional testing approaches partially apply but need extension. Unit tests at the tool-call level work; pass-or-fail end-to-end tests do not.
  • Evaluation, not testing, is the right frame. Continuous measurement of behaviour across many trials, with statistical assessment rather than binary pass-fail.
  • Replay and trace-based debugging have become essential tooling. When something fails, you need the full conversation transcript to understand why.
  • The hardest failure mode is the silent regression - the agent gets gradually worse without any single trial failing in an obvious way.

What makes this different

A few specific properties make agent reliability fundamentally different from classic system reliability:

The same input produces different outputs. Sampling from the model is stochastic by default. Even at temperature zero, small differences in prompt assembly, model version, or other context can produce divergent traces.

Failure is often a quality degradation, not an outage. The agent returns an answer, just not a good one. Uptime monitoring shows green. The user sees a worse experience.

The error surface is unbounded. Classic systems have well-defined inputs and outputs. Agent systems take natural language input that can be anything and produce natural language output that can be anything. The space of possible failures is open-ended.

Tools are an additional failure surface. Agents that use tools have all the failure modes of the model plus all the failure modes of the tools, plus the new category of “the model called the wrong tool with the wrong arguments.”

Reproducing failures is hard. Even with the same input, the bug may not reproduce. Even with the full trace, understanding why the model made a specific decision is often not possible.

What works in practice

A few patterns that have emerged for actually running agents reliably in 2026:

Multi-trial evaluation. Run each test case many times (10, 100, sometimes more) and report success rate, not binary pass/fail. A test that passes 90% of the time is better than one that passes 60%; a test that passes 100% deterministically is suspicious because real production traffic will not.

Statistical regression detection. When you change the prompt, the model version, or the tool surface, measure the new success rate against the previous one. The change is acceptable if the success rate is statistically similar or better. The framework looks more like A/B testing than like CI.

Structured trace storage. Every agent invocation in production captures the full conversation: prompts, responses, tool calls, tool results, errors, latencies. When something fails, the trace is what makes debugging possible. The storage cost is real but the diagnostic value is impossible to replicate.

Tool-level testing with deterministic mocking. The tools the agent uses can be tested deterministically. The hard work is testing the agent’s tool-selection and argument-formation, which requires evaluation rather than traditional testing.

Human-in-the-loop sampling. A sample of production traffic gets reviewed by humans (or by larger evaluator models) for quality, with the results feeding back into ongoing measurement. This catches the silent regression failures that automated metrics miss.

Per-step success accounting. Long agent runs are made of many steps. Tracking which step typically fails, in which order, on which task type makes debugging tractable in ways that “the agent failed” does not.

The evaluation infrastructure

The 2026 agent-running team typically has more evaluation infrastructure than testing infrastructure. The shape looks roughly like:

  • A curated set of test scenarios, with expected behaviours documented at the trace level rather than just at the output level.
  • An evaluation runner that exercises agents against these scenarios at scale - hundreds or thousands of trials per scenario per change.
  • A regression detector that compares results across versions.
  • A production sampling system that captures real traffic for ongoing quality measurement.
  • A human-review interface for sampled traces, with structured feedback that goes back into the metrics.
  • A trace inspector for debugging specific failures.

This is meaningfully more infrastructure than a typical software service requires. The teams running serious agent products in production have invested heavily in this infrastructure because the alternative - shipping changes blindly and learning about problems from user complaints - is unworkable.

Where the hard failures still hide

A few categories of failure that resist all the above techniques:

Behavioural drift over time. The agent’s behaviour changes slowly because the upstream model has been updated, because the conversation patterns in production have shifted, because the tool responses are different. No specific failure occurs; the success rate just gradually declines.

Long-tail input categories. The 0.1% of user inputs that nobody anticipated. By definition these are not in the test set. They become visible only through production observation.

Compound failures across steps. Each individual step in an agent loop is fine; the combination fails in non-obvious ways. The failure surface compounds combinatorially as the agent loop gets longer.

Adversarial inputs. Users who deliberately probe for failures. The class of inputs that look like normal usage but are designed to elicit specific failures is hard to test against because the adversary is creative.

These remain genuinely hard problems. The state of the art in 2026 is to detect them quickly rather than to prevent them entirely.

What this implies for engineering teams

The implications for teams shipping agent products:

Hire for evaluation skill. The traditional QA role does not map directly. Teams need people who think statistically about behaviour, who can design evaluation scenarios, who can interpret trace data. This is different from classic test engineering.

Budget for trace storage and human review. The infrastructure costs are real and continuing. Treating them as an afterthought produces the failure modes the infrastructure is supposed to prevent.

Build the trace-debugging interface early. The first time a production failure cannot be diagnosed because traces were not captured will produce more pain than the infrastructure cost to prevent it would have.

Treat behavioural changes as deployments, with the same care. A prompt change is a deployment. A model version bump is a deployment. The success-rate measurement around them is the equivalent of the canary deployment for code.

Accept that some failures will reach users. The 100% reliability that classic systems aspire to is not available for agent systems. The goal is to minimise the rate and severity of failures, not to eliminate them, and to recover from them quickly when they occur.

The honest summary

Running agents reliably in production in 2026 is a real engineering discipline. It is meaningfully different from running classic systems. The teams doing it well have built specific infrastructure, hired specific skill sets, and accepted that the reliability profile of agent systems is fundamentally different from what they were used to.

The teams that have not done this are running agents in production while pretending they are running classic systems. The mismatch shows up in user complaints, in degraded experiences they cannot explain, and in failures they cannot reproduce. The gap between these two groups is one of the more interesting differentiators in AI engineering in 2026.