Openlayer releases jevals — a fraction-of-a-cent replacement for...

Openlayer releases jevals — a fraction-of-a-cent replacement for LLM judges

Openlayer has released jevals, an open-source Python library that replaces expensive LLM judge calls with typed questions to the Jev decision model, checking an AI agent on 9 metrics in about 0.5 seconds and $0.00006 per trace.

Compiled by Sergey KostenchukPublished 2026-09-20Updated 2026-09-21

2026-09-20 Research OpenAI

🤖 Openlayer releases jevals — a replacement for LLM judges

The open-source jevals library replaces expensive LLM judge calls with typed questions to the Jev decision model, returning calibrated probabilities in a single forward pass. 9 agent checks — from tool selection to injections and PHI — fit into a single HTTP request: ~0.5 seconds and $0.00006 per trace.

🌍 Cost and non-determinism are the main complaints about LLM judges: according to LangChain measurements, the variance in GPT/Claude scores is 92–913 times higher than that of Jev. Checking every trace and allow/escalate/block gates before tool calls become practically free — a direct competitor to Ragas and DeepEval.

👤 pip install jevals checks every step of an agent for hallucinations, tool selection, and injections for a fraction of a cent. Nuance: the project is in alpha, Jev's accuracy is like that of small LLMs (83–87%) — calibrate the thresholds.

Source 1: https://github.com/openlayer-ai/jevals Source 2: https://news.ycombinator.com/item?id=49780849

Sources