Federated Edge-Centric Observability for Multi-Cloud Healthcare Data Pipelines

Abstract

Healthcare organizations increasingly operate data pipelines that span multiple public clouds and on-premise environments, integrating electronic health records (EHRs), claims, and Internet of Medical Things (IoMT) telemetry into shared analytic platforms. In this context, traditional centralized monitoring and data quality approaches—which assume that logs and data can be aggregated into a single platform—are costly and fragile. Moving protected health information (PHI) across cloud boundaries for inspection inflates egress spend and raises regulatory concerns, while platform-specific observability tools provide limited visibility into how failures propagate end-to-end across heterogeneous environments. This paper proposes a federated, edge-centric observability architecture for multi-cloud healthcare data pipelines. ssssLightweight observability agents are deployed near data sources in each cloud and on-premise region to instrument pipelines, compute local health and anomaly metrics, and evaluate policies. Above them, a cloud-agnostic Quality-as-a-Service (QaaS) control plane aggregates derived signals, manages models and policies, and orchestrates human-in-the-loop remediation using secure pointers rather than PHI replication. The architecture explicitly decouples the data plane from a quality and observability plane, enabling cross-cloud visibility while preserving data sovereignty. We evaluate the design on a synthetic, FHIR-compatible multi-cloud healthcare workload comprising 47 pipelines across four environments over 30 simulated days. Using chaos engineering, we inject realistic failure modes, including schema changes, delayed or partial loads, cross-cloud link failures, and backlogged batch processing. We compare the proposed federated approach to a representative centralized monitoring baseline along three dimensions: time-to-detection and localization of failures, monitoring-related inter-cloud data transfer costs, and resource overhead of agents and control plane. Across 216 chaos injections, the edge-centric architecture reduces median time-to-detection and localization while lowering monitoring-related egress volume by roughly 25% in this synthetic workload, with agent overhead of about 4–7% CPU and ~3% memory on batch pipelines. Given the limitations of synthetic data and simplified cost modeling, these results should be interpreted as early-stage validation of a promising architectural pattern rather than definitive benchmarks.