Flaky agent tests: the same LLM agent test passes sometimes and fails other times because output is non-deterministic. How do I make tests stable with recorded fixtures, seeds and tolerance-based evaluation?

Cause: Model outputs vary run to run, and an agent compounds this across multiple steps, tool calls and wording. Tests that compare exact strings, assume one fixed sequence of tool calls, or call a live model and live services on every run will fail intermittently. Anthropic's testing guidance is to define measurable success criteria and grade against them rather than expecting identical text. Fix (best first): 1. Split the layers. Unit-test everything deterministic (tool functions, parsers, routing, permission checks) with no model involved. These tests should never be flaky. 2. For the agent loop, replace the model and external services with recorded fixtures or fakes: store request/response pairs or scripted tool-call sequences, replay them in CI, and re-record deliberately when prompts or models change. Stub the clock, random ids and network. 3. Constrain output shape. Where the vendor offers it, use structured outputs for guaranteed schema conformance; otherwise specify an exact format and provide examples in the prompt. Test the shape strictly and the content leniently. 4. Grade with tolerances, not equality. Use exact match only for closed answers (labels, ids) after normalizing case and whitespace; use code-based checks for properties (the file was written, the right tool was called, no forbidden action); use similarity or a grader model for free text, ideally a different model than the one under test. 5. Run live-model evals separately from fast tests, over many cases that include edge cases, and assert on an aggregate pass rate with a threshold (for example at least 90 percent) instead of requiring every single case to pass. Prefer many automatically graded cases over a few hand-graded ones. 6. If your API exposes a sampling seed or low-randomness settings, use them to reduce variance, but do not rely on them for identical output across model versions. Pin the exact model version in eval runs. 7. Keep a record of failed outputs so flaky cases can be turned into new fixtures or tighter criteria. Verify: Run the deterministic suite and the fixture-based agent tests repeatedly (for example 20 times in a row) with zero failures, and confirm the live eval's pass rate stays inside the threshold across several runs. Sources: https://platform.claude.com/docs/en/test-and-evaluate/develop-tests https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/increase-consistency