Using Large Language Models (LLMs) to generate their own confidence scores is a scientifically invalid practice that creates only an illusion of system reliability.
What Happened
Author Justin Flick argues that asking an LLM for a direct confidence assessment is a psychological trick rather than a calibration tool. Models tend to cluster answers around round numbers and are unable to distinguish between confidence in factual correctness and confidence in linguistic coherence.
Context
To obtain a real assessment of reliability, researchers recommend using methods such as semantic entropy or training separate classifiers on data obtained from LLMs, instead of attempting to extract a numerical indicator from the generative output itself.
Why It Matters for the Industry
AI service developers should stop adding a "confidence" field to model JSON responses for decision-making logic, as this does not increase actual accuracy and can lead to systematic errors in agents. Instead, the industry needs to focus on retrieval quality (RAG) and the integration of specialized evaluation tools (evals).
Why It Matters for Users
Users should not trust digital confidence metrics in neural network responses unless they are backed by technical calibration. These numbers reflect the model's readiness to continue the text rather than the probability that the stated facts are correct.
Sources
Author
Look at AI, Editorial Team