The history of working with coding agents turns out to be not a consumable, but a dataset: Nebius developer Ibragim Badertdinov published the open-source evals-from-sessions skill, which builds private eval tasks with automatic verification from past sessions in Claude Code, Codex, and Pi. On his own task corpus, such a measurement showed that setups with identical results differ in price by a factor of two — and that choosing an agent is better done based on your own data, not public benchmarks.

image
image

What happened

Developer Ibragim Badertdinov, known for his work on RL environments and the SWE-rebench dataset at Nebius, published a guide and the agent skill evals-from-sessions under the MIT license. The skill takes session exports from Claude Code, Codex, and Pi and turns them into private eval tasks with automatic verification in the Harbor framework format; it is installed with a single command, npx skills add ibragim-bad/evals-from-sessions, or as a plugin for the same three agents. On a corpus of approximately 1,270 of his own sessions, the author built 10 tasks and ran different setups on them three times each: the lightweight CLI agent Pi on the GPT-6.1 Sol model with the medium setting solved all tasks on average for $0.070, while on Codex it took an average of $0.138. Security and correctness are built into the pipeline itself: exports are scanned with gitleaks, the verifier is protected from bypass paths, and each task undergoes human review before entering the set.

Context

Public benchmarks are built on other people's projects and give a weak idea of how a model and agent will behave on a specific stack and team work style, so it is difficult to transfer their conclusions to your own tasks. The answer was the idea of hill climbing on internal benchmarks, which Satya Nadella publicly promoted, and in the Claude Code ecosystem, build-eval and hillclimb teams have already appeared for building your own measurements. The remaining barrier is the cost of manually writing tasks, which most teams do not have the resources for; the evals-from-sessions approach closes exactly this gap, auto-generating tasks from work already done. The author's engineering culture is visible in the pipeline design: export privacy is checked with the gitleaks scanner, the verifier is isolated from potential deception, and final human review leaves the quality control of the set with a responsible person, not automation.

Why this matters for the industry

For the industry, the main shift is that comparing models, skills, and MCP servers becomes measurable on your own team's tasks — by price and solution time, not by other people's projects from public benchmarks. In the author's case, this immediately converted into money: a setup with the lightweight CLI agent Pi on GPT-6.1 Sol in medium mode cost twice as little as Codex with the same quality on his tasks, and the private eval corpus turns into an asset on which you can hill-climb and check model updates and new skills on the same tasks. If the pattern takes hold, the pipeline "session export, eval tasks, setup leaderboard" is expected to appear as a native function of agent products and as separate private benchmarking services, and private evals will become a routine hygiene practice for teams working with coding agents.

Why this matters for users

You can use this today: the skill is installed with the command npx skills add ibragim-bad/evals-from-sessions or as a plugin for Claude Code, Codex, and Pi, the license is MIT, and the author's guide and public leaderboard help understand the expected result. A practical route for the first run — run your session history through task collection and in one evening get the first version of a personal measurement, while also checking whether a cheaper model or another agent will solve your tasks with the same quality: the author found a setup twice as cheap this way and started spending twice as little on tokens. Expectations should be calibrated in advance: a dozen tasks will not give a strict verdict on models, but the set is enough to catch gross price and quality imbalances between setups and cut out obviously excessively expensive options.

What is still unknown / limitations

The author's conclusions are valid for his 10 tasks, not for models in general — he himself notes that he showed the best setup for his tasks, not a model rating, and three runs on a small private corpus do not give a correct statistical comparison. The conversion from sessions to tasks is less than one percent: from approximately 1,270 sessions, 10 tasks were obtained, each after human review, so the real cost of the project is hidden not in installation, but in filtering and checking, and for teams with less history, the set will be even smaller. In an enterprise, a pilot requires permission from the security team, because session exports may contain proprietary data, and gitleaks scanning does not replace internal security policies.

Sources

Author

Look at AI, editorial team