Awesome Kafka

A curated list of awesome Apache Kafka resources, tools, libraries, and applications.

Last verified: December 31, 2025 · Legend: ⚠️ Inactive (no commits in 2+ years) · 📦 Archived

See also: awesome-kafka-connect and Kafka Security Controls to help with CWE, NIST 800-53, and PCI-DSS compliance.

• Awesome Kafka
  • Kafka
  • Clients
  • Stream Processing
  • Kafka Connect
  • Schema Registry
  • Management & Monitoring
  • CLI Tools
  • Security
  • Testing & Development
  • Observability & Tracing
  • Kafka on Kubernetes
  • Use Cases & Case Studies
  • Best Practices & Patterns
  • Troubleshooting & Performance
  • Internals & Architecture
  • Disaster Recovery
  • Migration Guides
  • AI/ML Integration
  • Data Lakehouse Integration
  • Notable KIPs
  • Kafka-Compatible Alternatives
  • Managed Services
  • Learning Resources
  • Conferences & Events
  • Newsletters & Community
  • Books

Kafka

• Apache Kafka - Distributed event streaming platform capable of handling trillions of events a day.
• Kafka Documentation - Official documentation covering architecture, APIs, and operations.
• Kafka Quickstart - Get up and running with Kafka in minutes.
• KIPs - Kafka Improvement Proposals tracking upcoming features and changes.

Clients

Java

• Apache Kafka Client - Official Java client with Producer, Consumer, Streams, and Admin APIs.
• Spring for Apache Kafka - Spring Boot integration for Kafka producers, consumers, and transactions.
• Vert.x Kafka Client - Reactive Kafka client for Vert.x applications.
• SmallRye Reactive Messaging - MicroProfile reactive messaging with first-class Kafka connector used by Quarkus and Helidon.

C/C++

• librdkafka - High-performance C/C++ library, foundation for many language bindings.
• cppkafka - Modern C++11 wrapper for librdkafka.

Python

• confluent-kafka-python - High-performance client based on librdkafka with AsyncIO support.
• aiokafka - Native asyncio client for Python async applications.
• kafka-python - Pure Python client (not actively maintained).
• Faust - Stream processing library, Python equivalent of Kafka Streams.

Go

• franz-go - Feature-complete high-performance pure Go client.
• kafka-go - Go-idiomatic Kafka library by Segment.
• Sarama - Most popular Go client, pure Go implementation.
• confluent-kafka-go - CGo wrapper around librdkafka with official Confluent support.

Rust

• rust-rdkafka - Async futures-based client built on librdkafka.
• kafka-rust - Pure Rust Kafka protocol implementation.

Node.js / TypeScript

• confluent-kafka-javascript - Official Confluent client with TypeScript support.
• KafkaJS - Pure JavaScript client with clean API (not actively maintained).
• node-rdkafka - Node.js bindings for librdkafka.

.NET / C#

• confluent-kafka-dotnet - Official .NET client based on librdkafka.
• KafkaFlow - High-level .NET framework for building Kafka applications.
• Streamiz - Kafka Streams implementation for .NET.

Ruby

• rdkafka-ruby - Modern Ruby client based on librdkafka.
• Karafka - Ruby and Rails framework for Kafka with Web UI.
• ruby-kafka - Pure Ruby client (deprecated).

PHP

• php-rdkafka - PHP extension wrapping librdkafka.
• enqueue/rdkafka - Queue Interop compliant PHP wrapper.

Scala

• Alpakka Kafka - Reactive Streams connector for Akka Streams.
• ZIO Kafka - ZIO-based Kafka client for functional Scala.
• ⚠️ Greyhound - Wix's high-level Kafka SDK with ZIO, used in 1,500+ microservices.

Kotlin

• kotlin-kafka - Kotlin Coroutines and Arrow integration for Kafka.

Elixir / Erlang

• brod - Robust Erlang/Elixir client with consumer groups support.
• Kaffe - Opinionated Elixir wrapper around brod.
• KafkaEx - Pure Elixir Kafka client.

Swift

• swift-kafka-client - Swift Server Working Group client with modern concurrency.

Clojure

• Jackdaw - Comprehensive Kafka library for Clojure.
• ⚠️ kafka.clj - Clojure wrapper with Kafka Streams support.

Haskell

• hw-kafka-client - Haskell client based on librdkafka.

HTTP / REST

• Kafka REST Proxy - RESTful interface to produce and consume messages via HTTP.
• Strimzi Kafka Bridge - HTTP and AMQP bridge for Kubernetes.

Stream Processing

Kafka-Native

• Kafka Streams - Client library for building stream processing applications.
• ksqlDB - Event streaming database with SQL interface built on Kafka Streams.
• Interactive Queries - Query state stores in Kafka Streams applications.
• StreamT - Conduktor's dbt for streaming, transforming data in Kafka with SQL.

Frameworks

• Apache Flink - Distributed stream processing framework with exactly-once semantics.
• Apache Spark Streaming - Micro-batch stream processing on Spark.
• Apache Samza - Stream processing framework with deep Kafka integration.
• Apache Storm - Distributed real-time computation system.
• Apache Beam - Unified batch and streaming API portable across runners.

Streaming Databases

• RisingWave - PostgreSQL-compatible streaming database with materialized views.
• Materialize - Streaming database with incremental view maintenance.
• Apache Pinot - Real-time OLAP datastore with sub-second queries on Kafka topics.
• Apache Druid - Distributed data store for real-time analytics with Kafka ingestion.
• Timeplus Proton - Fast streaming SQL engine for real-time analytics.
• DeltaStream - Serverless stream processing using Flink SQL (commercial).
• StarRocks - MPP database with continuous Kafka routine load for lakehouse analytics.

Python Libraries

• Faust - Stream processing library porting Kafka Streams to Python.
• Quix Streams - Python library for high-volume time-series streaming.
• Bytewax - Python stream processing with Rust performance.
• Pathway - Python ETL framework with Rust engine for real-time processing.
• kstreams - Lightweight Kafka Streams implementation for Python.

Other

• Hazelcast - In-memory data grid with Jet stream processing engine.
• Redpanda Connect - Declarative stream processor with 100+ connectors (formerly Benthos).

Kafka Connect

CDC (Change Data Capture)

• Debezium - CDC platform for MySQL, PostgreSQL, MongoDB, SQL Server, Oracle, and more.

Databases

• JDBC Connector - Source and sink for JDBC-compatible databases.
• MongoDB Connector - Official MongoDB source and sink connector.
• Neo4j Connector - Source and sink for Neo4j graph database.
• Redis Sink - Write data to Redis.
• ClickHouse Sink - Official ClickHouse connector.

Cloud Storage

• S3 Sink - Export data to Amazon S3 in Avro, JSON, or Parquet.

Data Warehouses

• BigQuery Sink - Stream data to Google BigQuery with upsert support.
• Snowflake Sink - Official Snowflake connector with Iceberg support.

Message Queues

• JMS Source & Sink - Bridge JMS providers to Kafka topics.

IoT & Protocols

• MQTT Connector - Community MQTT source and sink connector for brokers and devices.

File Transfer

• SFTP/FTP Source - Stream files from SFTP/FTP servers into Kafka.

HTTP & APIs

• HTTP Sink - Send data to HTTP endpoints.
• HTTP Source - Poll HTTP APIs as a Kafka source.
• WebSocket Source - Stream messages from WebSocket endpoints into Kafka.
• gRPC Source - Consume gRPC server-streaming endpoints directly into Kafka topics.

Observability

• Splunk Sink - Send events to Splunk.
• OpenTelemetry OTLP Source - Receive OTLP traces/metrics/logs and publish them to Kafka topics.

Tools & Utilities

• Confluent Hub - Official connector marketplace.
• Lenses Stream Reactor - Collection of 25+ open-source connectors.
• Kafka Connect Datagen - Generate mock data for testing.
• Voluble - Realistic data generator for Kafka Connect.
• Spooldir Source - Monitor directories for new files.

Schema Registry

Implementations

• Confluent Schema Registry - Industry standard with Avro, Protobuf, and JSON Schema support.
• Karapace - Open-source drop-in replacement for Confluent Schema Registry (Apache 2.0).
• Apicurio Registry - Multi-format registry supporting OpenAPI, AsyncAPI, GraphQL, Avro, Protobuf.
• AWS Glue Schema Registry - Serverless registry for AWS with Avro and JSON Schema.
• Redpanda Schema Registry - Built-in schema registry in Redpanda.

Serialization

• Apache Avro - Compact binary format with rich schema evolution support.
• Protocol Buffers - Google's binary serialization with strong typing.
• Buf - Modern Protobuf toolchain with linting and breaking change detection.

Management & Monitoring

Web UIs

• Kafka UI - Free open-source UI for managing Kafka clusters, topics, and consumers.
• Kafbat UI - Community fork of Kafka UI with active development.
• Redpanda Console - Developer-friendly UI with time-travel debugging.
• AKHQ - Kafka GUI for topics, consumer groups, Schema Registry, and Connect.
• Kafdrop - Lightweight web UI for viewing Kafka topics and consumer groups.
• ⚠️ CMAK - Cluster Manager for Apache Kafka by Yahoo.
• Kouncil - Modern web interface with advanced message browsing.
• Conduktor Console - Enterprise control plane for Kafka with access control (commercial).
• Lenses - DataOps platform with SQL Studio and data policies (commercial).
• kPow - Enterprise Kafka monitoring with RBAC and audit logging (commercial).
• Confluent Control Center - Monitoring and management for Confluent Platform (commercial).

Metrics & Exporters

• Burrow - LinkedIn's consumer lag checking and monitoring service.
• Kafka Exporter - Prometheus exporter for Kafka broker and consumer group metrics.
• JMX Exporter - Prometheus exporter for JMX metrics from Kafka brokers.
• KMinion - Prometheus exporter for consumer lag and log directory sizes.
• 📦 Kafka Lag Exporter - Consumer group latency exporter for Kubernetes.
• Grafana Kafka Dashboards - Curated Grafana dashboards for Kafka broker, topic, and consumer metrics.

Cluster Management

• Strimzi - Kubernetes operator for running Apache Kafka (CNCF Incubating).
• Cruise Control - LinkedIn's automated workload rebalancer and self-healing.
• Cruise Control UI - Web interface for Cruise Control operations.
• MirrorMaker 2 - Built-in cross-cluster replication using Kafka Connect.
• Jikkou - GitOps tool for managing Kafka resources as code.
• Kafka-Kit - Datadog's tools for partition mapping, rebalancing, and auto-throttling.

Infrastructure as Code

• Terraform Provider for Kafka - Manage Kafka topics, ACLs, and quotas with Terraform.
• Confluent Terraform Provider - Terraform provider for Confluent Cloud resources including Kafka clusters.
• Conduktor Terraform Provider - Provision Conduktor Kafka Console resources (workspaces, RBAC, environments) via Terraform.

Interactive Tools

• Kafka Options Explorer - Browse and compare Kafka configuration options across versions.

CLI Tools

• kcat - Swiss army knife for Kafka, formerly kafkacat.
• kafkactl - Command-line tool inspired by kubectl with auto-completion.
• kaf - Modern CLI for Kafka inspired by kubectl and docker.
• Zoe - CLI for humans with time-based consumption and filtering.
• ⚠️ kcli - Simple Kafka command line browser.
• trubka - CLI tool for Kafka with Protobuf support.
• topicctl - Tool for managing Kafka topics with YAML configs.
• ⚠️ kafka-shell - Interactive shell for Apache Kafka.

Security

Authentication & Authorization

• Strimzi OAuth - OAuth2/OIDC authentication for Kafka.
• Apache Ranger - Fine-grained authorization and centralized policy management.
• ⚠️ Kafka Security Manager - Git-based ACL management with auto-revert by Conduktor.
• Julie - GitOps for Kafka RBAC and topologies.
• ⚠️ kafka-gitops - Manage Kafka resources as code with Git.

Encryption

• ⚠️ kafka-end-2-end-encryption - Transparent AES encryption for Kafka messages.
• kafkacrypto - End-to-end encryption library for Python and Java.

Governance

• Apache Atlas - Metadata management and data lineage for Kafka.
• DataHub - Modern data catalog with Kafka metadata and lineage support.
• OpenMetadata - Open-source metadata platform with Kafka integration.

Proxies & Gateways

• Conduktor Gateway - Kafka proxy with encryption, masking, and policy enforcement (commercial).
• Kroxylicious - Java framework for building Kafka protocol proxies.
• Gloo Gateway - Kubernetes-native API gateway with Kafka support.
• Apache APISIX - API gateway with kafka-proxy plugin.

Testing & Development

Local Development

• Kafka Docker - Docker Compose for Kafka ecosystem by Conduktor.
• cp-all-in-one - Confluent Platform Docker Compose files.
• kafka-local - Minimal Docker setup for local Kafka development.
• Redpanda - Kafka-compatible broker ideal for local development (no JVM).

Testing Frameworks

• Testcontainers Kafka - Automated Kafka containers for integration tests.
• Spring Kafka Test - Embedded Kafka for Spring Boot applications.
• kafka-junit - JUnit 4 and 5 support for embedded Kafka tests.
• embedded-kafka - Embedded Kafka for Scala testing with ScalaTest support.
• TopologyTestDriver - In-memory testing for Kafka Streams topologies.
• Toxiproxy - Simulate network conditions for resilience testing.

Data Generation

• Kafka Connect Datagen - Generate mock data based on Avro schemas.
• ⚠️ Voluble - Realistic relational data generator for Kafka.
• 📦 Datagen - Multi-format data generator by Materialize.
• Mockingbird - Mock streaming data generator.
• DataFaker - Java library for generating realistic fake data.

Benchmarking

• kafka-producer-perf-test - Built-in producer performance testing tool.
• kafka-consumer-perf-test - Built-in consumer performance testing tool.
• OpenMessaging Benchmark - Cross-platform messaging system benchmarking.
• ⚠️ Sangrenel - Kafka cluster load testing tool.

Chaos Engineering

• LitmusChaos Kafka Experiments - Chaos experiments for Kafka brokers, ZooKeeper/KRaft, and clients.
• Steadybit Kafka Attacks - Fault injection scenarios targeting Kafka brokers and client latency.
• Conduktor Chaos Testing via Gateway - Inject latency and failures through Conduktor Kafka proxy to validate resilience.

IDE Plugins

• IntelliJ Kafka Plugin - Official JetBrains plugin for Kafka.
• Kafkalytic - IntelliJ plugin with message search and filtering.
• Confluent for VS Code - Official Confluent extension for VS Code.
• Tools for Apache Kafka - VS Code extension for Kafka development.

Observability & Tracing

OpenTelemetry

• Instrumenting Kafka Clients with OpenTelemetry - Official OpenTelemetry guide for Kafka instrumentation.
• Kafka with OpenTelemetry Guide - Comprehensive distributed tracing setup guide.
• Tracing Kafka with OpenTelemetry - Practical instrumentation tutorial.
• Distributed Tracing for Kafka - New Relic's OpenTelemetry integration guide.
• Strimzi OpenTelemetry Support - From OpenTracing to OpenTelemetry in Strimzi.

Collectors & Pipelines

• OpenTelemetry Collector Kafka Receiver/Exporter - Ingest and ship traces, metrics, and logs through Kafka pipelines.
• Vector - High-performance observability pipeline with Kafka sources and sinks for logs, metrics, and traces.

Tracing Backends

• Jaeger - Open-source distributed tracing platform.
• Zipkin - Distributed tracing system for gathering timing data.
• Grafana Tempo - High-scale distributed tracing backend.

Kafka on Kubernetes

Operators

• Strimzi - CNCF Incubating Kubernetes operator for Apache Kafka.
• Confluent for Kubernetes - Official Confluent operator for deploying Confluent Platform.
• 📦 Koperator - Banzai Cloud's Kubernetes operator for Kafka (formerly Kafka Operator).

Helm Charts

• 📦 Confluent Platform Helm Charts - Helm charts for Confluent Platform components.
• Bitnami Kafka Chart - Production-ready Kafka Helm chart.
• Strimzi Helm Charts - Helm-based Strimzi installation.

Resources

• Running Kafka on Kubernetes at Shopify - Shopify's Kubernetes migration journey.
• Kafka on Kubernetes Best Practices - Confluent's production recommendations.

Use Cases & Case Studies

Enterprise Scale

• PayPal: Scaling Kafka to 1.3 Trillion Messages/Day - How PayPal runs 85+ clusters with 1,500 brokers processing trillions of messages.
• LinkedIn: 7 Trillion Messages Per Day - LinkedIn's Kafka customizations with 100+ clusters and 4,000+ brokers.
• Cloudflare: 1 Trillion Inter-Service Messages - How Cloudflare uses Kafka for analytics, DDoS mitigation, and logging.
• Netflix: Keystone Pipeline - Netflix's real-time personalization processing 2 trillion messages/day.
• Uber: Trillion Events Per Day - Uber's real-time infrastructure for driver-rider matching and fraud detection.
• Pinterest: 1.2 Petabytes Per Day - Running 2,000+ brokers for real-time streaming applications.
• Airbnb: Riverbed Framework - Processing 2.4 billion events daily with Kafka and Spark.
• Twitter: Kafka Adoption Story - Twitter's migration from EventBus to Kafka with 68-75% resource savings.
• Stripe: 6 Nines Availability - How Stripe achieves 99.9999% uptime with 700TB daily throughput.
• Slack: Self-Driving Kafka Clusters - Slack's Kafka powering job queues, logging, and analytics at 6.5Gbps.
• Shopify: Billions of Events - CDC with Debezium handling 66M messages/second at peak.
• DoorDash: 100 Billion Events - Iguazu pipeline processing billions of events with Kafka and Flink.
• Datadog: Trillions of Datapoints - Running 40+ clusters with petabytes of NVMe storage.
• Instacart: COVID-19 Scale - 10 years of growth in 6 weeks during pandemic surge.
• Lyft: Real-Time Pricing - Kafka for map-matching, ETA, and dynamic pricing.

Architecture Patterns

• Wix: 6 Event-Driven Architecture Patterns (Part 1) - Patterns from running 1,500 microservices on Kafka.
• Wix: 6 Event-Driven Architecture Patterns (Part 2) - Advanced patterns including retry strategies and dead-letter queues.
• Kai Waehner: Use Cases Across Industries - Comprehensive overview of Kafka use cases by industry.
• Kafka in Production - Curated collection of tech blogs and talks from companies running Kafka.

Best Practices & Patterns

Event Sourcing & CQRS

• Event Sourcing, CQRS, and Kafka - How Kafka enables event sourcing and CQRS patterns.
• CQRS in Event Sourcing Patterns - Confluent Developer course on implementing CQRS with Kafka.
• Microservices: Event Sourcing & CQRS - Pattern catalog entry for event sourcing.

Exactly-Once Semantics

• Exactly-Once Semantics in Kafka - Deep dive into how Kafka achieves exactly-once.
• Exactly-Once Processing with Kafka - Baeldung guide with Java code examples.
• Idempotent Consumer Pattern - Implementing idempotency for exactly-once semantics.
• What Exactly-Once Really Means - Clarifying the semantics and limitations.

Partitioning Strategies

• Kafka Topic Partitioning Strategies - Best practices from New Relic.
• Partition Strategy Guide - Confluent's comprehensive partitioning guide.
• Ultimate Comparison of Partition Strategies - Detailed comparison of all partitioning approaches.

Transactional Outbox Pattern

• Reliable Microservices Data Exchange with the Outbox Pattern - Debezium's foundational outbox pattern guide.
• Debezium Outbox Event Router - Official Debezium outbox SMT documentation.
• Implementing Outbox Pattern with CDC - Practical implementation guide.
• Transactional Outbox at SeatGeek - Real-world production implementation.
• Outbox with Debezium: Hidden Challenges - Lessons learned at Yotpo.

Schema Evolution

• Schema Evolution and Compatibility - Official Confluent schema evolution guide.
• Schema Registry Best Practices - Confluent's recommendations for schema management.
• Schema Evolution in Kafka - Comprehensive guide to backward and forward compatibility.
• Schema Compatibility Testing - Confluent course on testing schema compatibility.
• Handling Schema Evolution in Kafka Connect - Patterns and pitfalls guide.

Troubleshooting & Performance

Consumer Lag

• Kafka Consumer Lag Explained - Understanding and monitoring consumer lag.
• Fixing Consumer Lag Guide - Practical guide to diagnosing and fixing lag.
• Kafka Lag Causes and Solutions - Comprehensive lag troubleshooting.
• Consumer Lag Monitoring - Setting up lag monitoring with offsets.

Performance Tuning

• 7 Critical Kafka Performance Best Practices - Essential tuning recommendations.
• Top 10 Tips for Tuning Performance - meshIQ's performance optimization guide.
• Kafka Performance Tuning - Redpanda's tuning strategies.
• Kafka Optimization Theorem - Red Hat's mathematical approach to tuning.
• Producer and Consumer Tuning - Fine-tuning client configurations.

Debugging & Troubleshooting

• Troubleshooting and Debugging in Kafka - Comprehensive debugging guide.
• Common Kafka Performance Issues - Diagnosing and fixing common issues.

Internals & Architecture

KRaft (ZooKeeper Replacement)

• Deep Dive into KRaft Protocol - Red Hat's technical deep dive based on Kafka 4.1.0.
• KRaft Overview - Confluent's guide to Kafka Raft mode.
• Evolution from ZooKeeper to KRaft - Architectural evolution explained.

Replication & Consistency

• Kafka Replication Deep Dive - Technical deep dive into replication internals.
• ISR, High Watermark & Leader Epoch - Understanding ISR and consistency guarantees.
• Replication Documentation - Official Confluent replication docs.

Log Compaction

• Log Compaction Design - Official log compaction documentation.
• Introduction to Log Compaction - Beginner-friendly explanation.
• Log Compaction Troubleshooting - Configuration and troubleshooting guide.
• Compaction in Kafka Topics - Baeldung's practical guide.

How Kafka Works

• How Kafka Works - System design newsletter deep dive.
• Kafka Architecture Deep Dive - Confluent's architecture course.

Disaster Recovery

Multi-Datacenter Patterns

• Disaster Recovery for Multi-Datacenter Deployments - Confluent's comprehensive DR guide.
• Active-Passive vs Active-Active Topologies - Comparison of replication patterns.
• Cross-Data-Center Replication Playbook - Decision framework for geo-replication.
• Multi-Region Kafka with RPO=0 - Achieving zero data loss with synchronous replication.
• Hitchhiker's Guide to DR and Multi-Region Kafka - WarpStream's comprehensive guide.
• Building Bulletproof DR for Kafka - Field-tested architecture patterns.

Replication Tools

• MirrorMaker 2 - Built-in cross-cluster replication.
• Confluent Replicator - Enterprise replication with conflict detection.
• Cluster Linking - Byte-for-byte topic mirroring in Confluent Cloud.

Migration Guides

ZooKeeper to KRaft

• KIP-866: ZooKeeper to KRaft Migration - Official migration KIP.
• Strimzi KRaft Migration - How the migration works technically.
• Confluent ZK to KRaft Migration - Confluent Platform migration guide.
• OSO ZooKeeper to KRaft Guide - Step-by-step migration guide.
• Strimzi Cluster Migration - Kubernetes-specific migration with Strimzi.

Cross-Cluster Replication

• MirrorMaker 2 Guide - How to replicate data across clusters.
• MirrorMaker 2 Best Practices - Usages and best practices.
• Migration with MM2 and Strimzi - Kubernetes migration patterns.
• AWS MSK Migration Guide - Migrating to Amazon MSK.

AI/ML Integration

Feature Stores

• Online Feature Store with Kafka and Flink - Building real-time feature stores.
• Kafka as a Feature Store - Using Kafka for ML feature delivery.

ML Pipelines

• Kafka-ML Framework - Open-source framework connecting Kafka with TensorFlow and PyTorch.
• Machine Learning in Kafka Applications - Deploying ML models with Kafka.
• ⚠️ Kafka Streams ML Examples - Examples with H2O, TensorFlow, Keras, and DeepLearning4J.
• Machine Learning with Kafka - Patterns for ML integration.

Online Learning

• Online Model Training and Model Drift - Continuous model training with Kafka and Flink.

Data Lakehouse Integration

Tableflow & Iceberg

• Confluent Tableflow - Convert Kafka topics to Iceberg and Delta tables.
• Tableflow GA Announcement - Real-time Kafka to Iceberg integration.
• Apache Iceberg for Lakehouse and Streaming - Iceberg as the bridge between streaming and lakehouse.
• Tableflow with Trino and Jupyter - Querying Kafka data with Trino.

Data Warehouse Integration

• Snowflake Kafka Integration Options - Snowflake integration patterns including Iceberg.

Notable KIPs

KIP-500: ZooKeeper Removal

• KIP-500: Replace ZooKeeper with Self-Managed Metadata Quorum - The foundational KIP for KRaft.
• 42 Ways ZooKeeper Removal Improves Kafka - Benefits of the ZooKeeper-less architecture.
• Prepare for KIP-500 - Migration preparation guide.

KIP-1150: Diskless Topics

• KIP-1150: Diskless Topics - Store data directly in object storage.
• Kafka Without Disks - Explanation of diskless topics.
• Hitchhiker's Guide to Diskless Kafka - Aiven's comprehensive guide.
• Benchmarking Diskless Topics - Performance benchmarks.

Other Notable KIPs

• KIP-848: Next-Gen Consumer Group Protocol - Improved consumer rebalancing.
• KIP-405: Tiered Storage - Move older data to cheaper object storage.
• Kafka Improvement Proposals Index - All KIPs.

Kafka-Compatible Alternatives

• Redpanda - C++ implementation with Kafka API compatibility and 10x lower latency.
• WarpStream - Diskless Kafka on object storage, acquired by Confluent (commercial).
• AutoMQ - Cloud-native Kafka with shared storage for 50-90% cost reduction.
• Bufstream - Kafka-compatible broker with Apache Iceberg integration (commercial).

Related Message Brokers

• Apache Pulsar - Distributed pub-sub with separated compute and storage.
• RabbitMQ Streams - Stream processing extension for RabbitMQ.
• NATS JetStream - Lightweight cloud-native messaging with persistence.
• Amazon Kinesis - Fully managed real-time data streaming on AWS.

Managed Services

• Confluent Cloud - Fully managed Kafka with Kora engine and stream processing.
• AWS MSK - Amazon Managed Streaming for Apache Kafka with serverless option.
• Azure Event Hubs - Kafka protocol compatible event streaming on Azure.
• Aiven for Apache Kafka - Multi-cloud managed Kafka with open-source focus.
• Instaclustr - Managed open-source Kafka with strong compliance.
• Redpanda Cloud - Managed Redpanda with BYOC and serverless options.

Learning Resources

Courses

• Kafkademy - Free Kafka learning platform by Conduktor and Stéphane Maarek.
• Confluent Developer - Free courses on Kafka, Flink, Kafka Streams, and ksqlDB.
• Stephane Maarek's Kafka Courses - Popular Udemy courses covering Kafka fundamentals to advanced topics.
• Confluent Certified Developer - Official Kafka certification exam.

Documentation & Tutorials

• Confluent Documentation - Comprehensive guides for Confluent Platform and Cloud.
• Cloudurable Kafka Tutorial - In-depth tutorial covering Kafka 4.0 and KRaft.
• IBM Kafka Streams Labs - Hands-on Kafka Streams labs.

Videos & Podcasts

• Confluent YouTube - Official Confluent channel with tutorials and talks.
• Kafka Summit / Current - Annual data streaming conference talks.
• Confluent Developer Podcast - Weekly podcast with Kafka experts.

Community

• Confluent Community Slack - Active community for Q&A and discussion.
• Apache Kafka Mailing Lists - Official user and developer mailing lists.
• Stack Overflow - Q&A for Kafka development.
• Reddit r/apachekafka - Community discussions and news.

Reference

• Conduktor Data Streaming Glossary - Comprehensive glossary of data streaming terminology and concepts.

Conferences & Events

Kafka Summit / Current

• Kafka Summit - The largest annual Apache Kafka conference by Confluent.
• Current - Confluent's expanded conference covering the broader streaming ecosystem.
• Kafka Summit London 2024 - Past sessions and slides.
• Confluent Events Calendar - Upcoming events and conferences.

Flink Forward

• Flink Forward - Conference dedicated to Apache Flink and stream processing.
• Ververica Academy - Training sessions from Flink Forward conferences.

Meetups

• Apache Kafka Meetups - Local Kafka meetup groups worldwide.
• Kafka Community Events - Official Apache Kafka events page.
• Meetup in a Box - Resources for hosting Kafka meetups.

Newsletters & Community

Newsletters

• Confluent Developer Newsletter - Bimonthly newsletter on Kafka, Flink, and streaming.
• Kai Waehner's Blog - Data streaming insights and architecture patterns.
• ByteByteGo Newsletter - System design newsletter with frequent Kafka content.
• System Design Newsletter - Deep dives on distributed systems including Kafka.

Podcasts

• Confluent Developer Podcast - Weekly podcast with Kafka experts and practitioners.
• Software Engineering Daily: Kafka - Technical interviews about Kafka.

Community Resources

• Apache Kafka Community - Official community resources and contribution guidelines.
• Confluent Developer - Developer portal with courses, tutorials, and community links.

Books

• Kafka: The Definitive Guide, 2nd Edition - Comprehensive guide by Confluent and LinkedIn engineers.
• Kafka in Action - Practical introduction with real-world examples.
• Kafka Streams in Action - Stream processing guide from basics to production.
• Designing Event-Driven Systems - Free eBook on event-driven architecture with Kafka.
• I Heart Logs - Free eBook by Jay Kreps on logs and stream processing.

---

Contributing

Contributions are welcome! Please read the contribution guidelines first.

License