The cheap open model Gemma 4 31B gave students a GRE score gain statistically equivalent to lessons with a live human expert tutor — at a cost 918x lower. That is the bottom line of the large randomized StudentBench study, published on September 23, 2026, by the Handshake AI team in preprint arXiv:2609.28470. Code, de-anonymized data, and the reproduce.py script are open, so every number in the paper can be verified independently.

image
image

What happened

The randomized study covered 2,383 students and 2,469 prep sessions: 2,139 with AI tutors, 140 with a live expert, and 190 in the control group. The GRE score gain for those who studied with an LLM tutor was statistically indistinguishable from results with a human (p = .015), and in 5 of 7 GRE domains the best AI tutor beat the live mentor. The cost difference is even more striking: Gemma 4 31B cost $0.0052 per percentage-point gain versus $4.81 for a tutor with a $75/hour rate. The authors also traced the mechanism: instant AI responses were accompanied by more student messages, more solved practice problems, and a final gain, with all correlations significant at p < .002.

Context

Until now, models in education were compared mainly by exam metrics, answering the question “what does the model know,” while the question “what can it teach” was not systematically tested. StudentBench shifts the focus to real learning gain and, at this scale, for the first time addresses the “AI effect” — the illusion of learning, where a student feels they have mastered the material, but objective testing does not confirm it. The study design is unusually rigorous for a field where marketing case studies usually suffice: a full randomized control instead of isolated cases and self-reports. Notably, learning effectiveness is weakly tied to model class: the compact Gemma 4 31B was enough, meaning the task of “teaching” is less demanding than “knowing,” and for edtech, model mass and price matter more than the frontier. The authors’ reputation adds weight to the work: Curtis Northcutt and Jonas Mueller are known for the label-errors benchmark on annotation errors in datasets.

Why this matters for the industry

For the industry, the value is not in the formula “AI = tutor,” but in a data-confirmed value loop: instant feedback from a cheap model keeps the student in practice, and practice delivers a measurable gain at a cost three orders of magnitude lower than a live expert. The barrier to entry for personalized learning drops to near zero: Gemma 4 31B open weights, the public StudentBench protocol, and the reproducible reproduce.py pipeline allow a small team to build an AI tutor in a quarter and sell exam prep an order of magnitude cheaper than human. Competition shifts from model size to latency, the UX engagement loop, and practice quality — meaning the winner is not the best engineer, but the best product loop. The open dataset with 176,614 messages, 45,463 practice problems, and 2,028 paired ratings from 51 expert tutors is already suitable for fine-tuning and eval tuning, and StudentBench itself can be adopted as an internal eval with a testable hypothesis: does minimum response time reduce the number of solved problems. If independent replications confirm equivalence, a wave of tutoring-eval benchmarks and the inclusion of learning gain in model leaderboards alongside exam metrics are likely.

Why this matters for users

The practical takeaway for readers is direct: if you need to close a gap in math or verbal before an exam, an AI tutor already competes with a human on feedback speed and is unbeatable on price. Instant response is not a trivial detail: according to the authors, feedback speed drives the volume of practice that delivers the gain. Every number can be verified yourself: the code is on GitHub under the MIT license, de-anonymized student data is on Hugging Face under CC BY 4.0, the reproduce.py script verifies every chart and table with one command, and the StudentBench platform is available at studentbench.org. A live mentor is not replaced where motivation, mentorship, and complex case analysis are needed, but for routine practice, the alternative has become real and cheap.

What is still unknown / limitations

The effect mechanism is described by correlations, not established causality: response latency was not randomized, and students who write more messages may simply be more motivated than others. All data were collected only on GRE, so generalizing the findings to other subjects, languages, and professional skills remains an assumption. The work is still in preprint status and has barely been discussed by the community: on Hacker News it has 1 point and 0 comments. There are no independent replications yet, and their outcome will determine whether this result becomes an industry benchmark.

Sources

Author

Look at AI, editorial team