Apache Kafka Embraces Cloud-Native Efficiency: Diskless Topics

Diskless Topics (KIP-1150) represent a major evolution in Apache Kafka's architecture, introducing a topic type that writes data directly to remote object storage, bypassing broker-local disks entirely.

Up to 97%

Cost reduction for storage and IOPS

Typical TCO reductions of 80-90%

What are Diskless Topics?

Unlike traditional Kafka topics that rely on broker-local storage and replication, Diskless Topics leverage external durability through cloud object storage (Amazon S3, GCS, Azure Blob). This makes brokers more stateless—any broker can serve any partition, and intra-cluster replication is no longer needed.

Cloud-Native Benefits

Dramatic Cost Savings

Eliminate local storage and cross-AZ replication costs

Simplified Infrastructure

Stateless brokers enable easier scaling and failovers

Multi-Cloud Ready

Fewer cross-zone traffic costs for geo-distributed setups

Data Lake Integration

Direct writes to Iceberg, Delta Lake, or similar formats

Architecture Details

Key Components

Batch Coordinator: Manages metadata, offsets, and object/batch mapping

Leaderless Ingestion: Any broker can accept writes for any partition

Object Storage: Messages are batched and stored in cloud storage

Mixed Clusters: Run both classic and diskless topics together

Current Limitations

As of 2025, Diskless Topics have some feature gaps:

No support for transactions

Compacted topics not supported

Kafka Streams state stores unavailable

Auto-created topics not supported

Higher tail latencies vs local SSD

Best Fit Use Cases

Ideal for workloads where ultra-low latency (sub-100ms p99) is not critical but cost efficiency is:

Log aggregation and analytics pipelines

Event sourcing with flexible replay

Data lake ingestion workflows

High-volume, cost-sensitive streaming

Diskless 2.0

Later in 2025, "Diskless 2.0" combines Diskless Topics (KIP-1150) with Tiered Storage (KIP-405) into a unified path. The newer design aims for zero-copy migration between topic types, unified storage formats, and simplified architecture—including reclaiming features like transactions and queues.