Lorin Hochstein of Surfing Complexity warns of a hidden threat in using LLMs to write incident reports. In his view, delegating the synthesis of findings to artificial intelligence can lead to the creation of "simulacra"—documents that look professional but completely ignore the deep systemic causes of failures.
What Happened
Surfing Complexity blog author Lorin Hochstein published a critical analysis of the practice of using LLMs to automate the writing of incident reports. He argues that the process of writing text is not merely a formality, but a critical stage of cognitive processing and comprehension of complex systemic problems. Using AI to create the final narrative creates a risk of documents that mimic an understanding of the situation without performing actual analysis.
Context
Unlike software development, where written code can be verified with tests, or SRE tasks, the results of which are verified by reality, incident reports lack an explicit mechanism for verification against the ground truth. This makes them the most vulnerable area for LLM application, as the system has no way to objectively confirm the correctness of the conclusions drawn.
Why It Matters for the Industry
For the industry, there is a risk of systemic degradation of engineering culture and the loss of systems thinking skills. The mass use of LLMs to create "drafts" or final versions of documentation could lead to an accumulation of "understanding technical debt." Companies that prioritize speed and automation without strict verification protocols risk a situation where incidents are formally closed, but their root causes remain unidentified.
Why It Matters for Users
It is important for engineers to realize that while AI can be useful for gathering primary data, handing over the synthesis and conclusion-forming stage deprives the specialist of the opportunity to truly understand the complexity of the system. This creates an illusion of productivity that, in reality, leads to a decrease in the quality of incident post-mortems and an inability to prevent the recurrence of errors in the future.
Sources
Author
Look at AI, Editorial Staff