Openlayer has released jevals, an open-source Python library that replaces expensive calls to a frontier model judge with typed questions to the Jev decision model — the same model available via the TypeSafe API and Vercel AI Gateway. Evaluating a single agent trace costs a fraction of a cent and takes hundreds of milliseconds, and in the authors' benchmark, agent quality evaluation is approximately two orders of magnitude cheaper than using a generative judge like Ragas. The project is one week old, is in alpha status, and according to the independent JevBench benchmark, Jev's accuracy is currently on par with the smallest LLMs, so thresholds will need to be calibrated to your own data.

image
image

What Happened

Openlayer published the jevals Python library on GitHub and announced it on Hacker News as a Show HN. The library works on top of the Jev decision model: instead of generating text, the model returns calibrated probabilities for each typed question in a single forward pass — yes/no, multiple choice, or rubric-based scoring. Nine agent checks, including ToolChoice, Grounded, StayedInScope, AnswerRelevancy, Completeness, IndirectInjection, and PHI, are performed in a single HTTP request in approximately 0.5 seconds and $0.00006 per trace. It comes with 37 ready-made evaluation metrics, YAML descriptions of checks, three-level allow/escalate/block gates before tool calls in production, and adapters for the OpenAI Agents SDK and LangGraph.

Context

To evaluate the quality of AI agent responses, teams typically use an LLM judge: an expensive frontier model that reads the trace and generates a text verdict. This approach is expensive, slow, and non-deterministic: according to LangChain measurements, the variance in judge scores based on GPT and Claude is 92–913 times higher than for Jev, and the Ragas framework makes 6 to 11 LLM calls per sample for four metrics. Due to these costs, evaluation metrics remained an offline stage in CI rather than part of a running service. Interest in alternatives is also evident at the platform level: LangChain published a separate article on whether Jev can be an agent evaluator, meaning the class of typed decision models is being discussed within major LLMOps ecosystems.

Why This Matters for the Industry

For the industry, this is a signal of a price shift in LLMOps: according to the authors' benchmark on the gpt-4.1-mini model, evaluating 1,000 samples via jevals costs $0.03 compared to $2.60 for Ragas, and latency drops from 22–35 seconds to 0.8 seconds for 20 samples. At this price, evaluations and guardrails transform from an expensive CI add-on into cheap per-call infrastructure within the agent loop. Three-level allow/escalate/block gates allow building a hybrid architecture where a cheap typed judge checks every tool call, and an expensive frontier model is only connected for escalations. jevals directly competes with Ragas, DeepEval, and paid LLM judges, and if independent measurements confirm the numbers, pressure on the pricing of these tools and LangSmith will increase.

Why This Matters for Users

For those building their own AI agent, the library is available with a single command: pip install jevals. It can be used to check every step of the agent: tool selection, grounding in sources, staying within task scope, answer completeness, personal data leaks, and indirect injections in tool results. Quality policy is described in YAML, there is a backend="mock" for unit tests, and for data that cannot be sent to the cloud, there are local Kev or Laya models on Mac. A reasonable starting scheme is to wrap tool calls with gates and initially operate in shadow logging mode with escalation, calibrating thresholds on your own data using the jevals calibrate command.

What Is Still Unknown / Limitations

All impressive benchmark numbers — cost per trace, comparison with Ragas, and latency — are claimed by the vendor itself, the methodology for selecting tasks and metrics is not disclosed, and the measurement against Ragas on gpt-4.1-mini measures price and speed on a narrow set of scenarios. According to the independent JevBench benchmark, Jev's accuracy is 83–87 percent on Banking77 and CLINC150, meaning on par with the smallest LLMs, and this translates to up to approximately 15 percent classification errors at threshold boundaries; the authors themselves advise not to trust the classifier with authorization of irreversible actions. The project is one week old, is in alpha status with three stars on GitHub, and is currently more suitable for experiments, non-critical checks, and gates with escalation, while the question of whether the claimed numbers will be reproduced in independent measurements and whether the library will withstand production remains open.

Sources

Author

Look at AI, editorial team