Risks of Using LLMs for Writing Incident Reports

💻 Risks of Using LLMs for Writing Incident Reports

Lorin Hochstein from Surfing Complexity expresses concerns regarding the automation of incident report writing using LLMs. He argues that the process of writing is a critical stage of making sense of a problem, and using AI to synthesize the final narrative creates "simulacra" of reports—texts that look convincing but lack a deep understanding of systemic causes.

🌍 Risk of degrading engineering culture and losing systemic understanding. Using LLMs to synthesize reports without human involvement in the reasoning process can lead to the creation of documents that look correct but fail to uncover the actual causes of failures.

👤 It is important to remember that AI can help with data collection, but delegating the synthesis and conclusion stages to it deprives the engineer of the opportunity to truly understand the complexity of the system and prevent the recurrence of the incident.

Source 1: https://surfingcomplexity.blog/2026/06/19/i-am-dreading-our-llm-written-incident-report-future/