Post-Mortem Intelligence: Using Large Language Models to Build Proactive Reliability Knowledge Graphs from Incident Documentation

Abstract

Production incident post-mortem documentation represents one of the most underutilized assets in site reliability engineering. Organizations that conduct thorough post-mortem analyses accumulate structured knowledge about failure patterns, root causes, contributing factors, and remediation actions   knowledge stored as unstructured text rarely accessed after the specific incident it describes. This paper presents a post-mortem intelligence system that applies Large Language Models to extract structured knowledge from incident documentation and organize it into a searchable reliability knowledge graph, enabling on-call engineers to retrieve historically analogous incidents and their resolution pathways in real time during active investigations. The system was implemented and validated against a corpus of over 300 post-mortem documents accumulated across six credit union banking applications. LLM-based entity extraction using the Claude API identifies services, failure modes, contributing factors, timeline events, and remediation actions from unstructured post-mortem text. Extracted entities and relationships are stored in Amazon Neptune as a directed graph, with semantic similarity search enabling retrieval of analogous historical incidents based on current alert context. Analysis of the accumulated knowledge graph identified 14 systemic anti-patterns recurring across the incident corpus, and proactive infrastructure improvements derived from these patterns are estimated to prevent approximately 6 high-severity incidents annually. This paper presents the extraction methodology, graph schema design, retrieval architecture, and the organizational implications of treating post-mortem documentation as a continuously compounding engineering asset.