Unobservable Performance: Storage Failures That Leave No Metrics Behind

Abstract

Modern computer systems increasingly rely on intricate storage stack layers, and the performance of these layers can fail in ways that are not always easy to tell from error reports or measurable faults. A new category of silent or unobservable storage performance failures even trend to the point of causing the degradation of application throughput, latency, and reliability but without conventional alerts, counters, or health metrics being triggered. The root of these failures is the subtle coordination between the firmware, the controllers, caching layers, and the operating system which leave the system administrators little or no visibility into the real reason for the performance anomalies. Traditional monitoring frameworks largely rely on explicit error reporting, averaged latency metrics, or device-level statistics, which most of the time cannot pick up on transient stalls, internal throttling, or firmware-induced behaviours that are below the interface of the device and not directly observable. This paper tackles the problem of how to detect and characterize invisible performance failures which it does by offering a method that integrates workload-aware probing, cross-layer correlation, and anomaly-driven inference to bring to light hidden storage inefficiencies. Instead of just relying on the device-reported metrics, the method makes use of the end-to-end application behaviour, temporal patterns, and indirect performance signatures to deduce the storage pathologies. The real-life example in the paper shows a production system that was severely degraded in terms of its performance while its health indicators showed that it was in normal state. Later, it was confirmed that the problem was a silent internal storage bottleneck through the diagnostic procedure that was carried out by the proposed technique. Their studies expose the fact that system performance can deteriorate significantly without being noticed compromise because that the loss is undetected for a very long period which thus leads to misdiagnosis, poor scaling decisions, and reduced system reliability.