AgentAssay replaces binary pass/fail testing with three-valued stochastic verdicts (Pass/Fail/Inconclusive) backed by SPRT, plus behavioral fingerprinting that detects regressions invisible to pass-rate testing (86% power vs 0%), achieving 78% trial reduction and 100% cost savings via trace-first offline analysis — validated across 7,605 trials costing $227.
AI agents exhibit inherent non-determinism: same prompt/tools/model produce divergent behaviors across runs. Traditional testing assumes deterministic outputs and binary verdicts — both fail for stochastic systems. Existing LLM eval tools (deepeval, promptfoo) evaluate single-turn quality but don't formalize regression detection for multi-step workflows. The closest work (agentrial) computes confidence intervals but lacks formal semantics, coverage metrics, mutation operators, or composition theory.
Stochastic test semantics (§3): Agent $A = (\pi, \mathcal{T}, \mu, \omega)$. Three-valued verdict via Wilson score CI: Pass if $\mathrm{CI}_{\text{lower}} \geq \theta$, Fail if $\mathrm{CI}_{\text{upper}} < \theta$, Inconclusive otherwise. Verdict soundness (Theorem 3.1): false positive rate $\leq \alpha$.
Regression detection (§3.4): Dual requirement of statistical significance ($p < \alpha$) AND practical significance ($|\hat{p}_b - \hat{p}_c| \geq \delta$), following Arcuri & Briand's best practices. Power guarantee (Theorem 3.2).
SPRT adaptation (§3.5): Wald's sequential test with log-likelihood ratio $\Lambda_k$ and boundaries $a = \log(\beta/(1-\alpha))$, $b = \log((1-\beta)/\alpha)$. Optimal by Wald-Wolfowitz theorem. Achieves 78% trial reduction.
Five coverage dimensions (§4): Tool coverage, decision-path coverage (Chao1 estimator), state-space coverage ($1 - e^{-|\pi|/\lambda}$), boundary coverage, model coverage. Geometric mean aggregate. Coverage monotonicity (Theorem 4.1).
Agent mutation testing (§5): 12 operators across 4 classes (prompt: synonym/reorder/noise/dropout; tool: remove/reorder/noise; model: swap/downgrade; context: truncate/noise/permute). Stochastic kill semantics requiring both verdict change and statistical significance. Mutation adequacy theorem (5.1).
Token-efficient testing (§7) — five pillars:
Combined theoretical reduction: $R \in [0.05, 0.20]$ ($5$–$20\times$ savings).
核心技术壁垒: Behavioral fingerprinting detects regressions that are completely invisible to pass-rate testing — when an agent changes its tool usage pattern or reasoning chain without affecting output correctness, traditional testing has exactly 0% power while fingerprinting achieves 86%.
| Metric | Value |
|---|---|
| SPRT trial reduction | 78% (22 trials vs 100 fixed) |
| Fingerprint detection power | 86% (vs 0% for pass/fail) |
| Full-pipeline cost savings | 100% (trace-first offline) |
| Total experimental cost | $227 across 7,605 trials |
| Token variation across models | 5.3× |
| Latency variation | 21× |
| Models tested | 5 (GPT-5.2, Sonnet 4.6, Mistral-Large-3, Llama-4-Maverick, Phi-4) |
| Scenarios | 3 (e-commerce, customer support, code generation) |
| SPRT savings consistency | 77.5–78.2% (σ < 1 pp across scenarios) |
The trace-first optimization enables coverage, contract, and metamorphic testing on stored traces at zero API cost. Live testing (SPRT + fingerprinting) is needed only for regression detection on candidate versions.
| Check | Status | Location |
|---|---|---|
| Verdict soundness (Theorem 3.1) | Proven via Clopper-Pearson CI coverage | §3.3, Appendix A.1 |
| Regression power (Theorem 3.2) | Proven via Neyman-Pearson + FKG inequality | §3.4, Appendix A.1 |
| SPRT optimality (Proposition 3.3) | Proven via Wald-Wolfowitz theorem | §3.5, Appendix A.1 |
| Verdict-contract correspondence (Prop. 3.4) | Proven via Hoeffding's inequality | §3.7, Appendix A.1 |
| Coverage monotonicity (Theorem 4.1) | Proven via Chao1 consistency | §4.3, Appendix A.2 |
| Mutation adequacy (Theorem 5.1) | Proven by contrapositive + coupling effect | §5.4, Appendix A.3 |
| Fingerprint power advantage (Theorem 7.1) | Proven via Hotelling's $T^2$ | §7.1, Appendix A.4 |
| Combined efficiency (Theorem 7.6) | Proven | §7.6 |
All major theorems have full proofs. The coupling effect assumption (Theorem 5.1) is well-established for source code but its applicability to agent mutations is not empirically validated.
E1–E6 Behavioral Characterization (Table 4): 5,250 trials across 5 models × 3 scenarios × 50 trials. Token generation varies 5.3× across models (Mistral 561±346 vs Llama 106±20). Latency varies 21× (Phi-4 12,874 ms vs Llama 613 ms). 100% trace collection success.
E7 Token-Efficient Testing (Table 5, 1,250 data points):
The key insight: pass rates remained at 100% across all conditions — "regressions" were entirely behavioral/distributional. Traditional testing is completely blind to this class of regression.
Cost consistency (Table 7): SPRT savings 77.7% (e-commerce), 77.5% (customer support), 78.2% (code generation) — variance < 1 pp across scenarios despite 5.3× token variation.
| Step | Claim | Evidence | Depends on |
|---|---|---|---|
| 1 | Agent non-determinism breaks testing assumptions (determinism, binary verdicts) | Motivating scenario + Kapoor et al. (2025) confirming stochastic drift | — |
| 2 | Three-valued stochastic verdict restores soundness | Theorem 3.1 (false positive control $\leq \alpha$) | Step 1 |
| 3 | SPRT reduces trial count by 78% with identical guarantees | Proposition 3.3 + E7 empirical (22 vs 100 trials) | Step 2 |
| 4 | Behavioral fingerprinting detects regressions invisible to pass-rate | Theorem 7.1 + E7: 86% power vs 0% | Step 2 |
| 5 | Trace-first analysis achieves zero-cost testing for coverage/contracts/MR | Theorem 7.3 + E7: $0.000 cost, 94% power | Steps 3, 4 |
| 6 | Combined system achieves 5–20× theoretical cost reduction | Theorem 7.6 (R ∈ [0.05, 0.20]) | Steps 3, 4, 5 |
~20,000 LoC Python, 751 tests, Apache 2.0 license. pytest plugin with @agentassay.test decorator. 10 built-in framework adapters (LangGraph, CrewAI, AutoGen, OpenAI Agents SDK, smolagents, Semantic Kernel, Amazon Bedrock, MCP, Vertex AI, Generic). Zenodo DOI: 10.5281/zenodo.18842011.
关键实现细节: