kafka-learning-resources
README.md
Apache Kafka Learning Resources Repository
Hello and welcome to my comprehensive collection of Apache Kafka learning materials. This repository is designed to be a centralized resource for anyone looking to learn or deepen their knowledge of Apache Kafka, regardless of their experience level.
What You'll Find Here
This collection encompasses a wide range of materials organized by and suited to different learning preferences and skill levels. From in-depth guides and documentation to interactive exercises, I've gathered resources to cater to a variety of needs.
Contributing to the Collection
If you have a resource that could benefit others in their understanding of Apache Kafka, I encourage you to contribute. To submit a resource, please open an issue in the following format:
- Name: Title of the resource.
- Link: URL or location of the resource.
- Description: A brief summary of the resource.
- Source: The origin or creator of the resource.
- Skill Level: [Beginner, Intermediate, Advanced].
- Resource Type: [Video, Documentation, Guide or Tutorial, Blog Post, Book or Article, FAQ, Newsletter].
- Interactivity: [Hands-on Exercise, Quiz or Test, Community Forum].
- Language: [Java, Python, .NET, Go, JavaScript, Other].
I regularly review submissions to ensure the repository remains up-to-date and valuable for learners at all levels.
How I Manage the Resources
To keep things organized and easily accessible, I’ve implemented a simple, streamlined process. All the resources are stored in an AirTable spreadsheet, which acts as the central hub for all the content. This approach allows me to maintain and update the resources efficiently. Whenever there's a change in the contents of the spreadsheet, a script is executed to update this very README.md page. This process ensures that the information you see on this page is current and reflects the contents of the AirTable spreadsheet.
Your input and participation are what make this repository a valuable resource for everyone. If you have any questions or ideas, please don't hesitate to reach out. It is my goal to ensure that this repository remains a top-notch resource for Apache Kafka knowledge. Thank you for visiting and contributing to this collaborative effort. Together, we can make learning Apache Kafka accessible and effective for everyone.
Happy Learning!
Table of Contents
Skill Level
Beginner
- Apache Kafka Fundamentals: Apache Kafka Fundamentals Course with Tim Berglund. A great resource if you're a complete beginner. | 7 videos | ~90 min | Source: https://youtube.com
Tags: Beginner, Video, Hands-on Exercises - Apache Kafka 101: A complete end-to-end course with hands-on exercises taking you into the world of Kafka. | 18 videos, 6 hands-on exercises | ~90 min | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises - How do I get started building my first Kafka producer application?: An intuitive step-by-step tutorial on publishing Kafka messages. The tutorial offers two deployment options for you to choose from: pure Apache Kafka or Confluent Cloud. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises - How do I get started building my first Kafka consumer application?: An intuitive step-by-step tutorial on consuming Kafka messages. The tutorial offers two deployment options for you to choose from: pure Apache Kafka or Confluent Cloud. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - How can I count the number of messages in a Kafka topic?: Counting messages is the Hello World app of the Kafka world. This tutorial provides 3 options: plain vanilla Kafka, Confluent Cloud, and ksqlDB. Compare and contrast the three to learn the most! | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - Apache Kafka for .NET Developers: Learn how to use Kafka to build a streaming application with .NET. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, .NET - Apache Kafka for Python Developers: Building Kafka apps using Python. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Python - Top 7 Free Apache Kafka Tutorials and Courses for Beginners in 2023: A personal perspective on some great Kafka learning resources for beginners. | Source: https://developer.confluent.io
Tags: Beginner, Blog Post - Intro to Apache Kafka: How Kafka Works: Tim Berglund's intro to everything Kafka: events, topics, partitioning, brokers, replication, producers, consumers, and more. | Source: https://developer.confluent.io
Tags: Beginner, Blog Post - Apache Kafka in 6 Minutes: One of the most popular quick explainers on YouTube. | Source: https://youtube.com
Tags: Beginner, Video - Kafka in 100 Seconds: Although closer to 150 seconds, it's still quite impressive to give an idea about Kafka in such a short time. | Source: https://youtube.com
Tags: Beginner, Video - What is Kafka?: A great video explainer from IBM - just under 10 minutes. | Source: https://youtube.com
Tags: Beginner, Video - Apache Kafka Explained: Stephane Maarek's 15-part YouTube playlist on everything Kafka. | Source: https://youtube.com
Tags: Beginner, Video - Kafka: The Definitive Guide: Free 300+ page e-book (registration required), covering a full introduction to Apache Kafka , the distributed, publish-subscribe queue for handling real-time data feeds, Kafka internal architecture, what it's used for, and how to take full advantage of Kafka stream processing technology. | Source: https://www.confluent.io/
Tags: Beginner, Book or Article, Hands-on Exercises - Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale - 2nd Edition: This book was written for software engineers who develop applications that use Kafka’s APIs, and for production engineers (also called SREs, DevOps, or sysadmins) who install, configure, tune, and monitor Kafka in production. | Source: https://amazon.com
Tags: Beginner, Book or Article, Hands-on Exercises - What is a Kafka Cluster?: This video explains what an Apache Kafka cluster is, and why it's unique. Learn how Kafka supports speed, scalability, and durability through its cluster structure. | Source: https://youtube.com
Tags: Beginner, Video - Apache Kafka 3.6 New Features: Watch this sweet video (shot in an apiary) discussing the new features in Kafka 3.6 . | Source: https://youtube.com
Tags: Beginner, Video - Apache Kafka FAQ - Apache Software Foundation: Frequently Asked Questions about Apache Kafka by the Apache Software Foundation | Source: https://apache.org
Tags: Beginner, FAQ - Apache Kafka FAQ - Confluent: Frequently Asked Questions about Apache Kafka by Confluent | Source: https://developer.confluent.io
Tags: Beginner, FAQ - Apache Kafka FAQ - IBM: Frequently Asked Questions about Apache Kafka by IBM | Source: https://github.com
Tags: Beginner, FAQ - Learn Apache Kafka for Beginners v3: Intro course by Conduktor's Stephan Maarek: Learn Apache Kafka 3.0 Ecosystem, Core Concepts, Real World Java, Producers/Consumers & Big Data Architecture | For Fee | Source: https://udemy.com
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - Apache Kafka for absolute beginners: Learn Apache Kafka Ecosystem, Architecture, and Client API Programming in Java | For Fee | Source: https://udemy.com
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - Comprehensive Kafka Blog by Confluent: A comprehensive technical blog by Confluent. New posts on a regular basis - probably the best one out there. | Source: https://confluent.io
Tags: Beginner, Blog Post - Apache Kafka Documentation: The official Kafka documentation | Source: https://kafka.apache.org/
Tags: Beginner, Documentation - Amazon MSK Developer Guide: Amazon Managed Streaming for Apache Kafka | Source: https://docs.aws.amazon.com/
Tags: Beginner, Documentation - Confluent Documentation: Guides, samples, and references you need to use the streaming data platform based on Apache Kafka. | Source: https://docs.confluent.io
Tags: Beginner, Documentation - Confluent DevX Newsletter: This biweekly newsletter is a one-stop-shop for staying up to date on the latest Kafka and Flink learning materials, news, community meetups and events, useful terminal hacks, and some fun finds from around the web. Complete archive available. | Source: https://developer.confluent.io/newsletter/
Tags: Beginner, Newsletter
Intermediate
- Kafka Streams 101: Learn about the super powerful library to process and transform your data in the Kafka world. | 23 videos | 9 hands-on exercises | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - ksqlDB 101: Introduction to powerful event processing capabilities using declarative SQL statements. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Schema Registry 101: Everything you ever wanted to know about schema registry. If you're serious about your data, care about the message formats, or need versioning - this course is for you. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Kafka Connect 101: Need to integrate Kafka with your back-ends? There are hundreds of connectors readily available for you to choose from so you don’t have to reinvent the ‘data integration’ wheel. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Hybrid and Multicloud Architecture with Apache Kafka: Disaster recovery, cloud migration, and data sharing in a hybrid cloud world. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Mastering Production Data Streaming Systems with Apache Kafka: In this course, you learn how to avoid pitfalls when scaling your data streaming platform. You'll also delve deep into the GitOps framework helping you to deliver changes swiftly and securely. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Spring Framework and Apache Kafka: Building event streaming systems in Spring Boot. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Building Data Pipelines with Apache Kafka and Confluent: Build a scalable, streaming data pipeline in under 20 minutes using Kafka and Confluent. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Event Sourcing and Event Storage with Apache Kafka: In this course, you learn about how events can be used as the storage model for your applications, what event sourcing is, how it works, and how it relates to approaches like CQRS and CDC. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Apache Kafka Security: An introduction to Kafka security. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Hybrid and Multicloud Architecture with Apache Kafka: Explore hybrid cloud benefits and prepare for disaster recovery, cloud migration, and data sharing. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - What Are Apache Kafka Consumer Group IDs?: Setting the consumer Group ID determines what group a Kafka consumer belongs to, which has some major consequences. Read this blog post to learn more... | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post - Introduction to Consumer Group IDs: Lucia Cerchie explains what an Apache Kafka® Consumer Group ID is, and what role it plays in work sharing and rebalancing. | Source: https://youtube.com
Tags: Intermediate, Video - 5 Common Pitfalls When Using Apache Kafka: Blog post about with a list of five common pitfalls and tips for how to avoid them—from client and broker configurations to design and monitoring considerations—that are sure to save you time and effort down the road. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post - Diagnose and Debug Apache Kafka Issues: Understanding Increased Consumer Rebalance Time: Part 3 in the Diagnose and Debug Apache Kafka Issues blog post series by Confluent. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post - How to Run Confluent on Windows in Minutes: Learn how to set up the entire Confluent stack on Windos using WSL (Windows Subsystem for Linux 2). | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises - How to Run Apache Kafka on Windows: Learn how to run Apache Kafka on Windows using WSL (Windows Subsystem for Linux 2). | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises - My Python/Java/Spring/Go/Whatever Client Won’t Connect to My Apache Kafka Cluster in Docker/AWS/My Brother’s Laptop. Please Help!: A detailed look at 5 scenarios of what can go wrong with connections. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises, Python - Exactly-Once Semantics Are Possible: Here’s How Kafka Does It: In this blog post, Neha Narkhede explains what Kafka’s exactly-once semantics mean, why it is a hard problem, and how the new idempotence and transactions features in Kafka enable correct exactly-once stream processing using the Kafka Streams API. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post - Transactions in Apache Kafka: The goal of this post is to share the main concepts needed to use the transaction API in Apache Kafka effectively. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post - Kafka Connect Deep Dive – Converters and Serialization Explained: Kafka Connect is part of Apache Kafka®, providing streaming integration between data stores and Kafka. For data engineers, it just requires JSON configuration files to use. There are connectors for common (and not-so-common) data stores out there already, including JDBC, Elasticsearch, IBM MQ, S3 and BigQuery, to name but a few. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises - Amazon S3 Kafka Connector Setup & Configuration: Connecting your Kafka environment to AWS S3 using the S3 Connector. | Source: https://youtube.com
Tags: Intermediate, Video - Kafka Data Streaming Cheat Sheet: A multi-page cheat sheet for anybody deploying streaming applications to production. The cheat sheet focuses on topics, including Data, Platform, Applications, Automation, and Business Continuity. | Source: https://confluent.io
Tags: Intermediate, FAQ - Kafka Streams for Data Processing: Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8 | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Kafka Connect Hands-on Learning: Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8 | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Kafka Cluster Setup & Administration: Hands-On Training on ZooKeeper Quorum Setup, Kafka Cluster Setup and Administration in AWS. | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises
Advanced
- Streaming Data Governance: Learn about Confluent Cloud stream governance features. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - Data Mesh 101: Data mesh is a framework for decentralized domain-driven architectures, with data as a product, that is self-service, and with strong governance models. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - Apache Kafka Internal Architecture: This course describes the architecture of Apache Kafka with lectures from Jun Rao, one of its original authors. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - Inside ksqlDB: Learn about the internals of ksqlDB, how it was built, as well as security, scalability and performance. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - GPT-4 + Streaming Data = Real-Time Generative AI: Blog post about bringing real-time streaming data to LLMs. | Source: https://developer.confluent.io
Tags: Advanced, Blog Post - Apache Kafka Message Compression: In addition to topic partitions, message compression plays an important role in ensuring high Kafka throughput. | Source: https://developer.confluent.io
Tags: Advanced, Blog Post - Unknown Magic Byte! How to Address Magic Byte Errors in Apache Kafka: Everything you ever wanted to know about the Magic Byte in Kafka. | Source: https://developer.confluent.io
Tags: Advanced, Blog Post - Multi-Threaded Message Consumption with the Apache Kafka Consumer: Understanding Kafka consumer internals is important in implementing a successful multi-threaded solution. This blog post takes a look under the hood of the Kafka consumer as a first step.
| Source: https://developer.confluent.io
Tags: Advanced, Blog Post
Resource Type
Video
- Apache Kafka Fundamentals: Apache Kafka Fundamentals Course with Tim Berglund. A great resource if you're a complete beginner. | 7 videos | ~90 min | Source: https://youtube.com
Tags: Beginner, Video, Hands-on Exercises - Introduction to Consumer Group IDs: Lucia Cerchie explains what an Apache Kafka® Consumer Group ID is, and what role it plays in work sharing and rebalancing. | Source: https://youtube.com
Tags: Intermediate, Video - Apache Kafka in 6 Minutes: One of the most popular quick explainers on YouTube. | Source: https://youtube.com
Tags: Beginner, Video - Kafka in 100 Seconds: Although closer to 150 seconds, it's still quite impressive to give an idea about Kafka in such a short time. | Source: https://youtube.com
Tags: Beginner, Video - What is Kafka?: A great video explainer from IBM - just under 10 minutes. | Source: https://youtube.com
Tags: Beginner, Video - Apache Kafka Explained: Stephane Maarek's 15-part YouTube playlist on everything Kafka. | Source: https://youtube.com
Tags: Beginner, Video - Amazon S3 Kafka Connector Setup & Configuration: Connecting your Kafka environment to AWS S3 using the S3 Connector. | Source: https://youtube.com
Tags: Intermediate, Video - What is a Kafka Cluster?: This video explains what an Apache Kafka cluster is, and why it's unique. Learn how Kafka supports speed, scalability, and durability through its cluster structure. | Source: https://youtube.com
Tags: Beginner, Video - Apache Kafka 3.6 New Features: Watch this sweet video (shot in an apiary) discussing the new features in Kafka 3.6 . | Source: https://youtube.com
Tags: Beginner, Video
Book or Article
- Kafka: The Definitive Guide: Free 300+ page e-book (registration required), covering a full introduction to Apache Kafka , the distributed, publish-subscribe queue for handling real-time data feeds, Kafka internal architecture, what it's used for, and how to take full advantage of Kafka stream processing technology. | Source: https://www.confluent.io/
Tags: Beginner, Book or Article, Hands-on Exercises - Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale - 2nd Edition: This book was written for software engineers who develop applications that use Kafka’s APIs, and for production engineers (also called SREs, DevOps, or sysadmins) who install, configure, tune, and monitor Kafka in production. | Source: https://amazon.com
Tags: Beginner, Book or Article, Hands-on Exercises
Guide or Tutorial
- Apache Kafka 101: A complete end-to-end course with hands-on exercises taking you into the world of Kafka. | 18 videos, 6 hands-on exercises | ~90 min | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises - Kafka Streams 101: Learn about the super powerful library to process and transform your data in the Kafka world. | 23 videos | 9 hands-on exercises | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - How do I get started building my first Kafka producer application?: An intuitive step-by-step tutorial on publishing Kafka messages. The tutorial offers two deployment options for you to choose from: pure Apache Kafka or Confluent Cloud. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises - How do I get started building my first Kafka consumer application?: An intuitive step-by-step tutorial on consuming Kafka messages. The tutorial offers two deployment options for you to choose from: pure Apache Kafka or Confluent Cloud. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - How can I count the number of messages in a Kafka topic?: Counting messages is the Hello World app of the Kafka world. This tutorial provides 3 options: plain vanilla Kafka, Confluent Cloud, and ksqlDB. Compare and contrast the three to learn the most! | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - ksqlDB 101: Introduction to powerful event processing capabilities using declarative SQL statements. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Schema Registry 101: Everything you ever wanted to know about schema registry. If you're serious about your data, care about the message formats, or need versioning - this course is for you. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Kafka Connect 101: Need to integrate Kafka with your back-ends? There are hundreds of connectors readily available for you to choose from so you don’t have to reinvent the ‘data integration’ wheel. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Hybrid and Multicloud Architecture with Apache Kafka: Disaster recovery, cloud migration, and data sharing in a hybrid cloud world. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Mastering Production Data Streaming Systems with Apache Kafka: In this course, you learn how to avoid pitfalls when scaling your data streaming platform. You'll also delve deep into the GitOps framework helping you to deliver changes swiftly and securely. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Spring Framework and Apache Kafka: Building event streaming systems in Spring Boot. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Apache Kafka for .NET Developers: Learn how to use Kafka to build a streaming application with .NET. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, .NET - Apache Kafka for Python Developers: Building Kafka apps using Python. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Python - Building Data Pipelines with Apache Kafka and Confluent: Build a scalable, streaming data pipeline in under 20 minutes using Kafka and Confluent. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Event Sourcing and Event Storage with Apache Kafka: In this course, you learn about how events can be used as the storage model for your applications, what event sourcing is, how it works, and how it relates to approaches like CQRS and CDC. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Apache Kafka Security: An introduction to Kafka security. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Hybrid and Multicloud Architecture with Apache Kafka: Explore hybrid cloud benefits and prepare for disaster recovery, cloud migration, and data sharing. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Streaming Data Governance: Learn about Confluent Cloud stream governance features. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - Data Mesh 101: Data mesh is a framework for decentralized domain-driven architectures, with data as a product, that is self-service, and with strong governance models. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - Apache Kafka Internal Architecture: This course describes the architecture of Apache Kafka with lectures from Jun Rao, one of its original authors. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - Inside ksqlDB: Learn about the internals of ksqlDB, how it was built, as well as security, scalability and performance. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - Learn Apache Kafka for Beginners v3: Intro course by Conduktor's Stephan Maarek: Learn Apache Kafka 3.0 Ecosystem, Core Concepts, Real World Java, Producers/Consumers & Big Data Architecture | For Fee | Source: https://udemy.com
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - Kafka Streams for Data Processing: Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8 | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Kafka Connect Hands-on Learning: Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8 | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Kafka Cluster Setup & Administration: Hands-On Training on ZooKeeper Quorum Setup, Kafka Cluster Setup and Administration in AWS. | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Apache Kafka for absolute beginners: Learn Apache Kafka Ecosystem, Architecture, and Client API Programming in Java | For Fee | Source: https://udemy.com
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java
Documentation
- Apache Kafka Documentation: The official Kafka documentation | Source: https://kafka.apache.org/
Tags: Beginner, Documentation - Amazon MSK Developer Guide: Amazon Managed Streaming for Apache Kafka | Source: https://docs.aws.amazon.com/
Tags: Beginner, Documentation - Confluent Documentation: Guides, samples, and references you need to use the streaming data platform based on Apache Kafka. | Source: https://docs.confluent.io
Tags: Beginner, Documentation
Blog Post
- GPT-4 + Streaming Data = Real-Time Generative AI: Blog post about bringing real-time streaming data to LLMs. | Source: https://developer.confluent.io
Tags: Advanced, Blog Post - Apache Kafka Message Compression: In addition to topic partitions, message compression plays an important role in ensuring high Kafka throughput. | Source: https://developer.confluent.io
Tags: Advanced, Blog Post - Unknown Magic Byte! How to Address Magic Byte Errors in Apache Kafka: Everything you ever wanted to know about the Magic Byte in Kafka. | Source: https://developer.confluent.io
Tags: Advanced, Blog Post - Top 7 Free Apache Kafka Tutorials and Courses for Beginners in 2023: A personal perspective on some great Kafka learning resources for beginners. | Source: https://developer.confluent.io
Tags: Beginner, Blog Post - What Are Apache Kafka Consumer Group IDs?: Setting the consumer Group ID determines what group a Kafka consumer belongs to, which has some major consequences. Read this blog post to learn more... | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post - 5 Common Pitfalls When Using Apache Kafka: Blog post about with a list of five common pitfalls and tips for how to avoid them—from client and broker configurations to design and monitoring considerations—that are sure to save you time and effort down the road. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post - Diagnose and Debug Apache Kafka Issues: Understanding Increased Consumer Rebalance Time: Part 3 in the Diagnose and Debug Apache Kafka Issues blog post series by Confluent. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post - How to Run Confluent on Windows in Minutes: Learn how to set up the entire Confluent stack on Windos using WSL (Windows Subsystem for Linux 2). | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises - How to Run Apache Kafka on Windows: Learn how to run Apache Kafka on Windows using WSL (Windows Subsystem for Linux 2). | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises - Multi-Threaded Message Consumption with the Apache Kafka Consumer: Understanding Kafka consumer internals is important in implementing a successful multi-threaded solution. This blog post takes a look under the hood of the Kafka consumer as a first step.
| Source: https://developer.confluent.io
Tags: Advanced, Blog Post - My Python/Java/Spring/Go/Whatever Client Won’t Connect to My Apache Kafka Cluster in Docker/AWS/My Brother’s Laptop. Please Help!: A detailed look at 5 scenarios of what can go wrong with connections. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises, Python - Intro to Apache Kafka: How Kafka Works: Tim Berglund's intro to everything Kafka: events, topics, partitioning, brokers, replication, producers, consumers, and more. | Source: https://developer.confluent.io
Tags: Beginner, Blog Post - Exactly-Once Semantics Are Possible: Here’s How Kafka Does It: In this blog post, Neha Narkhede explains what Kafka’s exactly-once semantics mean, why it is a hard problem, and how the new idempotence and transactions features in Kafka enable correct exactly-once stream processing using the Kafka Streams API. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post - Transactions in Apache Kafka: The goal of this post is to share the main concepts needed to use the transaction API in Apache Kafka effectively. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post - Kafka Connect Deep Dive – Converters and Serialization Explained: Kafka Connect is part of Apache Kafka®, providing streaming integration between data stores and Kafka. For data engineers, it just requires JSON configuration files to use. There are connectors for common (and not-so-common) data stores out there already, including JDBC, Elasticsearch, IBM MQ, S3 and BigQuery, to name but a few. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises - Comprehensive Kafka Blog by Confluent: A comprehensive technical blog by Confluent. New posts on a regular basis - probably the best one out there. | Source: https://confluent.io
Tags: Beginner, Blog Post
FAQ
- Kafka Data Streaming Cheat Sheet: A multi-page cheat sheet for anybody deploying streaming applications to production. The cheat sheet focuses on topics, including Data, Platform, Applications, Automation, and Business Continuity. | Source: https://confluent.io
Tags: Intermediate, FAQ - Apache Kafka FAQ - Apache Software Foundation: Frequently Asked Questions about Apache Kafka by the Apache Software Foundation | Source: https://apache.org
Tags: Beginner, FAQ - Apache Kafka FAQ - Confluent: Frequently Asked Questions about Apache Kafka by Confluent | Source: https://developer.confluent.io
Tags: Beginner, FAQ - Apache Kafka FAQ - IBM: Frequently Asked Questions about Apache Kafka by IBM | Source: https://github.com
Tags: Beginner, FAQ
Newsletter
- Confluent DevX Newsletter: This biweekly newsletter is a one-stop-shop for staying up to date on the latest Kafka and Flink learning materials, news, community meetups and events, useful terminal hacks, and some fun finds from around the web. Complete archive available. | Source: https://developer.confluent.io/newsletter/
Tags: Beginner, Newsletter
Interactivity
Hands-on Exercises
- Apache Kafka Fundamentals: Apache Kafka Fundamentals Course with Tim Berglund. A great resource if you're a complete beginner. | 7 videos | ~90 min | Source: https://youtube.com
Tags: Beginner, Video, Hands-on Exercises - Apache Kafka 101: A complete end-to-end course with hands-on exercises taking you into the world of Kafka. | 18 videos, 6 hands-on exercises | ~90 min | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises - Kafka Streams 101: Learn about the super powerful library to process and transform your data in the Kafka world. | 23 videos | 9 hands-on exercises | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - How do I get started building my first Kafka producer application?: An intuitive step-by-step tutorial on publishing Kafka messages. The tutorial offers two deployment options for you to choose from: pure Apache Kafka or Confluent Cloud. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises - How do I get started building my first Kafka consumer application?: An intuitive step-by-step tutorial on consuming Kafka messages. The tutorial offers two deployment options for you to choose from: pure Apache Kafka or Confluent Cloud. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - How can I count the number of messages in a Kafka topic?: Counting messages is the Hello World app of the Kafka world. This tutorial provides 3 options: plain vanilla Kafka, Confluent Cloud, and ksqlDB. Compare and contrast the three to learn the most! | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - ksqlDB 101: Introduction to powerful event processing capabilities using declarative SQL statements. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Schema Registry 101: Everything you ever wanted to know about schema registry. If you're serious about your data, care about the message formats, or need versioning - this course is for you. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Kafka Connect 101: Need to integrate Kafka with your back-ends? There are hundreds of connectors readily available for you to choose from so you don’t have to reinvent the ‘data integration’ wheel. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Hybrid and Multicloud Architecture with Apache Kafka: Disaster recovery, cloud migration, and data sharing in a hybrid cloud world. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Mastering Production Data Streaming Systems with Apache Kafka: In this course, you learn how to avoid pitfalls when scaling your data streaming platform. You'll also delve deep into the GitOps framework helping you to deliver changes swiftly and securely. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Spring Framework and Apache Kafka: Building event streaming systems in Spring Boot. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Apache Kafka for .NET Developers: Learn how to use Kafka to build a streaming application with .NET. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, .NET - Apache Kafka for Python Developers: Building Kafka apps using Python. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Python - Building Data Pipelines with Apache Kafka and Confluent: Build a scalable, streaming data pipeline in under 20 minutes using Kafka and Confluent. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Event Sourcing and Event Storage with Apache Kafka: In this course, you learn about how events can be used as the storage model for your applications, what event sourcing is, how it works, and how it relates to approaches like CQRS and CDC. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Apache Kafka Security: An introduction to Kafka security. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Hybrid and Multicloud Architecture with Apache Kafka: Explore hybrid cloud benefits and prepare for disaster recovery, cloud migration, and data sharing. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Streaming Data Governance: Learn about Confluent Cloud stream governance features. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - Data Mesh 101: Data mesh is a framework for decentralized domain-driven architectures, with data as a product, that is self-service, and with strong governance models. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - Apache Kafka Internal Architecture: This course describes the architecture of Apache Kafka with lectures from Jun Rao, one of its original authors. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - Inside ksqlDB: Learn about the internals of ksqlDB, how it was built, as well as security, scalability and performance. | Source: https://developer.confluent.io
Tags: Advanced, Guide or Tutorial, Hands-on Exercises - How to Run Confluent on Windows in Minutes: Learn how to set up the entire Confluent stack on Windos using WSL (Windows Subsystem for Linux 2). | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises - How to Run Apache Kafka on Windows: Learn how to run Apache Kafka on Windows using WSL (Windows Subsystem for Linux 2). | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises - My Python/Java/Spring/Go/Whatever Client Won’t Connect to My Apache Kafka Cluster in Docker/AWS/My Brother’s Laptop. Please Help!: A detailed look at 5 scenarios of what can go wrong with connections. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises, Python - Kafka Connect Deep Dive – Converters and Serialization Explained: Kafka Connect is part of Apache Kafka®, providing streaming integration between data stores and Kafka. For data engineers, it just requires JSON configuration files to use. There are connectors for common (and not-so-common) data stores out there already, including JDBC, Elasticsearch, IBM MQ, S3 and BigQuery, to name but a few. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises - Kafka: The Definitive Guide: Free 300+ page e-book (registration required), covering a full introduction to Apache Kafka , the distributed, publish-subscribe queue for handling real-time data feeds, Kafka internal architecture, what it's used for, and how to take full advantage of Kafka stream processing technology. | Source: https://www.confluent.io/
Tags: Beginner, Book or Article, Hands-on Exercises - Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale - 2nd Edition: This book was written for software engineers who develop applications that use Kafka’s APIs, and for production engineers (also called SREs, DevOps, or sysadmins) who install, configure, tune, and monitor Kafka in production. | Source: https://amazon.com
Tags: Beginner, Book or Article, Hands-on Exercises - Learn Apache Kafka for Beginners v3: Intro course by Conduktor's Stephan Maarek: Learn Apache Kafka 3.0 Ecosystem, Core Concepts, Real World Java, Producers/Consumers & Big Data Architecture | For Fee | Source: https://udemy.com
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - Kafka Streams for Data Processing: Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8 | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Kafka Connect Hands-on Learning: Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8 | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Kafka Cluster Setup & Administration: Hands-On Training on ZooKeeper Quorum Setup, Kafka Cluster Setup and Administration in AWS. | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Apache Kafka for absolute beginners: Learn Apache Kafka Ecosystem, Architecture, and Client API Programming in Java | For Fee | Source: https://udemy.com
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java
Language
Java
- Kafka Streams 101: Learn about the super powerful library to process and transform your data in the Kafka world. | 23 videos | 9 hands-on exercises | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - How do I get started building my first Kafka consumer application?: An intuitive step-by-step tutorial on consuming Kafka messages. The tutorial offers two deployment options for you to choose from: pure Apache Kafka or Confluent Cloud. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - How can I count the number of messages in a Kafka topic?: Counting messages is the Hello World app of the Kafka world. This tutorial provides 3 options: plain vanilla Kafka, Confluent Cloud, and ksqlDB. Compare and contrast the three to learn the most! | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - Spring Framework and Apache Kafka: Building event streaming systems in Spring Boot. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Learn Apache Kafka for Beginners v3: Intro course by Conduktor's Stephan Maarek: Learn Apache Kafka 3.0 Ecosystem, Core Concepts, Real World Java, Producers/Consumers & Big Data Architecture | For Fee | Source: https://udemy.com
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java - Kafka Streams for Data Processing: Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8 | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Kafka Connect Hands-on Learning: Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8 | For Fee | Source: https://udemy.com
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises, Java - Apache Kafka for absolute beginners: Learn Apache Kafka Ecosystem, Architecture, and Client API Programming in Java | For Fee | Source: https://udemy.com
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Java
Python
- Apache Kafka for Python Developers: Building Kafka apps using Python. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, Python - My Python/Java/Spring/Go/Whatever Client Won’t Connect to My Apache Kafka Cluster in Docker/AWS/My Brother’s Laptop. Please Help!: A detailed look at 5 scenarios of what can go wrong with connections. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises, Python
.NET
- Apache Kafka for .NET Developers: Learn how to use Kafka to build a streaming application with .NET. | Source: https://developer.confluent.io
Tags: Beginner, Guide or Tutorial, Hands-on Exercises, .NET
Integration
Several Integrations
- Kafka Connect 101: Need to integrate Kafka with your back-ends? There are hundreds of connectors readily available for you to choose from so you don’t have to reinvent the ‘data integration’ wheel. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Hybrid and Multicloud Architecture with Apache Kafka: Disaster recovery, cloud migration, and data sharing in a hybrid cloud world. | Source: https://developer.confluent.io
Tags: Intermediate, Guide or Tutorial, Hands-on Exercises - Kafka Connect Deep Dive – Converters and Serialization Explained: Kafka Connect is part of Apache Kafka®, providing streaming integration between data stores and Kafka. For data engineers, it just requires JSON configuration files to use. There are connectors for common (and not-so-common) data stores out there already, including JDBC, Elasticsearch, IBM MQ, S3 and BigQuery, to name but a few. | Source: https://developer.confluent.io
Tags: Intermediate, Blog Post, Hands-on Exercises - Amazon S3 Kafka Connector Setup & Configuration: Connecting your Kafka environment to AWS S3 using the S3 Connector. | Source: https://youtube.com
Tags: Intermediate, Video