KFP v2 Artifact-Centric ML Pipeline Governance

Abstract

The second version of Kubeflow Pipelines (KFP) represents a significant evolutionary step, which creates a window of opportunity to cleverly solve one of MLOps’ most challenging problems the governance of (ML) artifacts, which in turn drive the lifecycles of (ML) models. The paper under review introduces an artifact-centric governance framework engineered to elevate the traceability, reproducibility, compliance, and auditability of KFP v2-based ML workflows. Typically, pipeline tracking systems tend to focus on execution metadata at the expense of the detailed relationships that link datasets, models, and metrics and thereby define the lineage of a pipeline. Moving governance focus to the level of artifacts, the proposed solution thus allows for exact versioning, dependency mapping, and integrity checking of the pipeline at any component level. The intention of this research is to put forward a reference model for the governance of artifacts stretching from data ingestion to model deployment and embedding metadata management, lineage visualization, and compliance controls in the KFP ecosystem. The framework furthers the establishment of policy-driven artifact tracking, enriched metadata schema, and provenance-aware logging mechanisms, which inter alia ensure that an artifact’s creation, transformation, and consumption are documented in detail. This, in turn, not only facilitates collaboration and audit readiness but also aids the implementation of organizational AI governance and regulatory compliance initiatives. The experimental validation shows that implementing an artifact-centric governance model leads to a decrease in the complexity of governance tasks, while at the same time the reproducibility and the speed of audit response increase.