A new study by scientists from French and Italian universities has identified a dangerous effect known as "cognitive surrender": when using AI recommendations, the accuracy of human judgment drops from 27% to 9%, while users' false confidence in the correctness of their answers increases by 2.5 times.
What Happened
During an experiment using the GPT-4o mini model (referred to in source as Step 3.5 Flash), which intentionally made errors in details, scientists recorded a sharp decline in the accuracy of human decisions. Parallel to the drop in accuracy, human confidence in their answers jumped from 30% to 76%. Furthermore, participants' willingness to admit their own ignorance decreased from 44% to a critical 3%.
Context
The research focuses on the phenomenon where the excessive confidence of neural networks suppresses critical thinking. Models that provide answers in a directive and confident manner cause users to shift from active information verification to passive acceptance of AI assertions as truth.
Why It Matters for the Industry
For AI system developers, this is a signal to rethink approaches to interface design and response delivery. The current trend of maximizing model confidence may harm users. The industry may need to transition from direct-answer patterns to "calibrated confidence" mechanisms and the implementation of elements that stimulate critical reflection, such as Socratic prompting or the display of uncertainty.
Why It Matters for Users
Users should exercise caution and not use AI as a sole source of truth. There is a risk of creating an illusion of competence: by relying on erroneous neural network data, you may be extremely confident in the correctness of your actions while making gross errors.
Sources
Author
Look at AI, Editorial Staff