The Collapse of Predictability in Large-Pool Redundant Storage

Abstract

Hyperscale data centers have grown so fast that storage architecture has drastically changed from being isolated, well-defined systems to those extremely large, interconnected pools composed of thousands of drives, controllers, and failure domains. Redundancy mechanisms including replication and erasure coding have been the main ways to ensure reliability, but their example at such a scale is not clear anymore. This paper is about the unpredictability of large-pool redundant storage systems which grow so much that the traditional assumptions about independent failures and linear scaling no longer hold. With the expansion of storage infrastructures, there is a growing influence of correlated failures, repair traffic amplification, rebuild contention, and thermal or power-domain coupling on the overall dynamics, thus eliminating the possibility of using deterministic methods for capacity planning and availability forecasting. The main point which this paper tries to illustrate is that when the scale and redundancy density go beyond a certain limit, it is no longer possible to sum up the individual components of the system's reliability profile, but it becomes a property of the whole system, that is to say, increasing redundancy does not correspond to an increase in resilience and, it is even possible that systemic risk is heightened. In order to confirm this discovery, the authors rely on analytical reliability modeling in conjunction with large-scale simulation and patterns of real-world operational telemetry to measure failure propagation, rebuild times, and resource contention under various redundancy configurations. Some effects that contribute to failure clustering and repair amplification have been studied through the help of mean-time-to-data-loss predictions, while background recovery traffic is responsible for fluctuations in performance stability and these effects only serve to worsen one another under stress.