Version v0.3.0 of Prism-Eval has been released — the first public open-source framework for unit testing non-deterministic AI agents. The tool by Amin Parva (Insight IT Solutions) allows running adversarial tests against agents working with data extraction, document processing, and tool calling, and integrates with existing CI/CD pipelines via pytest, JUnit, and SARIF.
What happened
Prism-Eval v0.3.0 was published under the Apache 2.0 license, available via pip install prism-eval. The framework detects four classes of non-deterministic errors in LLM agents: digit drop — missing digits when extracting sums from documents, prompt injection — embedding malicious prompts, layout shifts — document layout shifts that disrupt extraction, and OCR noise — optical character recognition noise. The framework works with any agents — LangGraph, CrewAI, HTTP endpoints — without being tied to a specific orchestration stack. The companion package Prism-Shield adds runtime protection for agents in production.
Context
Standard testing frameworks, such as pytest, assume deterministic code outputs and cannot test the stochastic behavior of LLM agents. Testing agent systems was in demo mode: teams ran agents on examples and hoped for no critical failures. Prism-Eval introduces a formalized G4 classification of adversarial cases — the first systematic division of vulnerability types specific to agent systems, not generative models in general. The key principle of the framework is the zero critical false accepts invariant, which requires that an agent never misses a critical error under any adversarial conditions.
Why this matters for the industry
For the industry, Prism-Eval closes the gap between demo mode and engineering process. Integration with pytest and export to JUnit/SARIF makes the tool compatible with existing DevOps pipelines without rewriting infrastructure. Formalization of G4 cases creates a foundation for future benchmarks of agent systems, similar to HELM or BigBench for generative models. The zero critical false accepts invariant could become a standard metric in pitch presentations and technical specifications for agent products. For startups, the tool reduces the cost of QA for agent systems and accelerates production deployment.
Why this matters for users
Developers can install Prism-Eval with the pip install prism-eval command and run the built-in adversarial corpus against their own agent in 30 seconds. This provides immediate feedback on vulnerabilities before release, especially critical for financial operations, where digit drop — missing digits in sums — can lead to direct financial losses. Product teams will receive a SARIF report as proof of agent reliability for customers. Existing engineering skills and processes cover the new testing task — no training on new tools is required.
What is not yet known / limitations
There is no empirical data on the effectiveness of G4 corpora — it is unknown what share of real agent failures they actually detect. There is no comparative analysis with other solutions for testing AI agents. The methodological rigor of the G4 case classification has not been confirmed by independent research. Prism-Shield as a companion package for runtime protection has no public effectiveness metrics.
Sources
Author
Look at AI, editorial team