CNCF White Paper: Data Storage Remains the Main Barrier to Cloud-Native AI at Scale

The CNCF TAG Infrastructure published a white paper titled “Data Storage in Cloud Native AI” — a community-authored document not sponsored by any single vendor. The internal working title of the project was “Data On Kubernetes – Data Analytics and AI/ML Workloads,” which more precisely describes the practical focus: how does data storage behave when AI workloads run through Kubernetes?

The answer the document provides is not optimistic. Traditional storage architectures, optimized for standard microservices, show serious limitations when tasked with feeding massive datasets into parallelized, high-performance GPU hardware infrastructure.

What Are the Three Core Bottlenecks?

The document identifies three systemic barriers that cloud-native AI infrastructure must overcome to function at scale.

The small-file problem is perhaps the most demanding. Training datasets routinely contain millions of small files — images, audio clips, text documents. Each file requires a separate metadata operation. The aggregate of millions of such operations creates enormous metadata server load, resulting in GPU nodes waiting for data instead of computing. GPU utilization drops, costs rise, and productivity stagnates.

The compute-storage separation bottleneck is the second systemic challenge. Separating compute and storage resources is a foundational architectural principle of cloud-native environments — and it generally works well for standard workloads. But for AI training and inference, the API overhead that accumulates with every access to remote data compounds into a visible performance penalty and reduced utilization of expensive accelerators.

Variable requirements across different phases are the third bottleneck, this time of an organizational nature: training, inference, and agentic AI have fundamentally different storage requirement profiles, and a one-size-fits-all approach cannot optimally serve all three phases.

Three AI Lifecycle Phases with Different Needs

The white paper structures cloud-native AI storage around three phases that each require separate architectural consideration.

The training phase primarily demands high throughput. The system must be able to deliver enormous quantities of data continuously and sequentially to GPU clusters, with efficient checkpoint management enabling training resumption after interruption. Apache Parquet format dominates here due to its columnar reads and compression.

The inference phase places emphasis on latency. Requests arrive in bursts — traffic is not uniform — and the model must be available quickly. KV caching and fast model loading become critical. Apache Iceberg as a table format brings time travel and schema evolution, which are useful for managing model versions.

Agentic AI presents the most complex requirements. Agents executing iterative reasoning loops need a combination of short-term memory for active context within a session and long-term memory for knowledge that persists between sessions. They also generate artifacts — intermediate results, references, planned steps — that need to be stored and retrievable. This is where vector databases come into play, with Milvus receiving particular attention as an open-source solution for efficient vector search at scale.

Key Technologies and CNCF Projects

The document maps the ecosystem of technologies addressing the identified bottlenecks. CSI (Container Storage Interface) and COSI (Container Object Storage Interface) standardize storage access from Kubernetes workloads, independent of the specific vendor.

Fluid, an active CNCF project, implements a distributed caching layer that keeps data closer to compute nodes. This directly addresses the overhead of compute-storage separation — data is cached locally on the nodes actively using it. For streaming and event-driven pipeline architectures, Apache Kafka remains the reference implementation.

Why This Is Relevant for Platform Engineers

The white paper is not an academic document — it is a practical reference for teams making decisions today about how to architect Kubernetes clusters for AI workloads. The guidance is concrete and actionable: which data format to choose for which use case, how to structure the caching layer, which CNCF projects to integrate.

The community-authored approach with no vendor sponsorship gives the document a credibility that marketing white papers lack. CNCF TAG Infrastructure invites the community to participate in further development — reviewing the PDF, contributing to TAG Infrastructure charter discussions, and the #tag-infrastructure Slack channel.