Description

A beginner-level course that follows a step-by-step approach to learning the fundamentals and core concepts of Apache Kafka 3.0. You will work through interesting activities such as programming a Twitter producer and Elasticsearch consumer to understand the various concepts.

The high throughput and low latency of Apache Kafka have made it one of the leading distributed data-streaming enterprise technologies. It is used by many Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart, and LinkedIn. If you want to develop Apache Kafka skills to stream data easily, then this course is for you. The course starts by explaining the architecture of the Apache Kafka ecosystem, before going on to cover the core concepts of Kafka such as topics, partitions, brokers, replicas, producers, consumers, and more. Next, you will use native Kafka binaries to launch your own Kafka cluster on Windows, Mac OS X, and Linux. As you advance, you will get hands-on experience with the Kafka command-line interface (CLI), along with an understanding of how to create producers and consumers in Java to interact with Kafka. Next, a real-world project using Wikimedia as a source of data for a producer and OpenSearch as a sink for our consumers. Moving on, you will get to grips with advanced APIs such as Kafka Connect and Kafka Streams and perform a case study on the real-world applications of Kafka. Finally, you will get an overview of advanced Kafka for administrators and understand advanced topic configurations. By the end of this course, you will be well-versed with how Apache Kafka 3.0 plays an important role in data-streaming applications.

What You Will Learn

Understand the Apache Kafka ecosystem, architecture, core concepts, and operations
Start a personal Kafka development environment
Create producers and consumers in Java to interact with Kafka
Master concepts such as topics, partitions, brokers, producers, consumers
Discover how to change a topic configuration
Practice and understand log compaction

Audience

This course is for developers who want to learn the fundamentals of Apache Kafka, software architects who want to understand how Kafka fits into their solution architecture or anyone who is looking to understand how Apache Kafka works as a distributed system. Basic knowledge of Java and the Linux command line will be beneficial to effectively understand the concepts covered in the course.

Approach

With this course, you will get hands-on experience with Apache Kafka. Through interesting examples and activities, you will learn how to use Apache Kafka to build robust distributed data-streaming applications.

Key Features

Find out why Apache Kafka is an excellent choice of distributed data-streaming platform * Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs * Get a solid grip on Apache Kafka by understanding various real-world examples

Github Repo

https://github.com/PacktPublishing/Apache-Kafka-Series---Learn-Apache-Kafka-for-Beginners-v3

About the Author

Stéphane Maarek

Stéphane Maarek is a solutions architect, consultant, and software developer who has a particular interest in all things related to big data and analytics. He is also a bestseller instructor on Udemy for his courses on Apache Kafka, Apache NiFi, and AWS Lambda. He loves Apache Kafka and regularly contributes to the Apache Kafka project. Stéphane has also written a guest blog post that was featured on the Confluent website, the company behind Apache Kafka. He is also an AWS Certified Solutions Architect and has many years of experience with technologies such as Apache Kafka, Apache NiFi, Apache Spark, Hadoop, PostgreSQL, Tableau, Spotfire, Docker, Ansible, and more.

Course Outline

1. Kafka Introduction

1. Course Introduction

In this video, the author introduces you to the course and his experience in this field.

2. Apache Kafka in Five Minutes

This video introduces Apache Kafka.

3. Course Objectives

In this section, you will learn about topics and partitions, how to replicate topics, and much more.

2. Kafka Theory

1. Topics, Partitions, and Offsets

This video explains topics, partitions, and offsets.

2. Producers and Message Keys

This video explains the concept of producers and message keys.

3. Consumers and Deserialization

This video explains the concept of consumers and deserialization.

4. Consumer Groups and Consumer Offsets

This video explains the consumer groups and consumer offsets.

5. Brokers and Topics

This video explains the concept of Kafka brokers and topics.

6. Topic Replication

This video explains the topic replication process in Kafka.

7. Producer Acknowledgments and Topic Durability

This video explains producer acknowledgments and topic durability.

8. Zookeeper

This video explains the concept of a Zookeeper in Kafka.

9. Kafka KRaft - Removing Zookeeper

This video explains Kafka KRaft. Kafka Raft (KRaft) is the consensus protocol that was introduced to remove Apache Kafka's dependency on Zookeeper for metadata management.

10. Theory Roundup

This video provides a roundup of the theory.

3. Starting Kafka

1. Important: Starting Kafka

This video demonstrates how to start Kafka in various operating systems and how we can overcome the challenge of starting Kafka.

2. Starting Kafka with Conduktor - Multi-Platform

This video demonstrates how to start Kafka with Conduktor.

3. Mac OS X - Download Kafka and Set Up PATH

This video demonstrates how to download Kafka and set up PATH on Mac OS X.

4. Mac OS X - Start Zookeeper and Kafka

This video demonstrates how to start a Zookeeper server and install Kafka on Mac OS X.

5. Mac OS X - Using Brew

This video demonstrates how to install Kafka on Mac OS X using Brew.

6. Linux - Download and Set Up Kafka in PATH

This video demonstrates how to download Kafka and set up PATH on a Linux system.

7. Linux - Start Zookeeper and Kafka

This video demonstrates how to start a Zookeeper server and install Kafka on a Linux server.

8. Windows WSL2 - Download Kafka and PATH Setup

This video demonstrates how to download Kafka and set up PATH on Windows.

9. Windows WSL2 - Start Zookeeper and Kafka

This video demonstrates how to start a Zookeeper server and install Kafka on Windows.

10. Windows WSL2 - How to Fix Problems

This video demonstrates the bug on WSL2 and how to fix the bug.

11. Windows Non-WSL2 - Start Zookeeper and Kafka

This video demonstrates how to start a Zookeeper server and install Kafka non-WSL2.

4. Starting Kafka without Zookeeper

1. Mac OS X - Start Kafka in KRaft Mode

This video explains how to start one Kafka Broker in KRaft mode without a Zookeeper on Mac OS.

2. Linux - Start Kafka in KRaft Mode

This video explains how to start one Kafka Broker using KRaft mode on Linux without a Zookeeper in this instance.

3. Windows WSL2 - Start Kafka in KRaft Mode

This video explains how to start one Kafka Broker with Zookeeper in KRaft Mode

5. Command Line Interface (CLI) 101

1. Command Line Interface (CLI) Introduction

This video introduces you to Kafka command line interface and how it comes bundled with Kafka binaries.

2. Kafka Topics Command Line Interface (CLI)

This video explains the CLI commands related to Kafka topics.

3. Kafka Console Producer Command Line Interface (CLI)

This video explains the kafka-console-producer CLI command to start producing data into our Kafka topics.

4. Kafka Console Consumer Command Line Interface (CLI)

This video explains the Kafka-console-consumer.sh CLI.

5. Kafka Consumers in a Group

This video explains the concept of Kafka consumers in a group and the -group parameter.

6. Kafka Consumer Groups Command Line Interface (CLI)

This video explains the consumer groups CLI, in which we can reset, delete, and manage the consumer groups.

7. Resetting Offsets

This video focuses on resetting offsets using the Kafka Offsets groups command.

6. Kafka UI

1. Conduktor - Demo

This video demonstrates Kafka UI Conduktor to get the full power of the UI.

7. Kafka Java Programming 101

1. Kafka SDK List

This video explains Kafka SDK list.

2. Creating Kafka Project

This video demonstrates how to create a Kafka project.

3. Java Producer

This video explains how to write a basic producer to send data to Kafka.

4. Java Producer Callback

This video explains how to create Kafka producer in Java using the callback function.

5. Java Producer with Keys

This video explains how to send messages in Kafka using keys.

6. Java Consumer

This video explains how to write a basic consumer to receive data from Kafka.

7. Java Consumer - Graceful Shutdown

This video explains a code to respond to termination signals and how to improve our Java code.

8. Java Consumer Inside a Consumer Group

This video explains the behavior of our consumers as part of our consumer groups.

9. Java Consumer Incremental Cooperative Rebalance and Static Group Membership

This video explains Java consumer incremental cooperative rebalance and static group membership.

10. Java Consumer Incremental Cooperative Rebalance - Practice

This video explains Java consumer incremental cooperative rebalance-practice.

11. Java Consumer Auto Offset Commit Behavior

This video explains the concept of Java consumer auto offset commit behavior.

12. Programming - Advanced Tutorials

This video explains programming-advanced tutorials.

8. Kafka Real-World Project

1. Real-World Project Overview

This video provides an overview of the real-world project.

9. Kafka Wikimedia Producer and Advanced Configurations

1. IMPORTANT: Start Local Kafka with Conduktor Using Docker

This video provides an overview of how to start Kafka with Conduktor using Docker Compose.

2. Wikimedia Producer Project Setup

This video provides an overview of the Wikimedia producer project setup.

3. Wikimedia Producer Implementation

This video explains Wikimedia producer implementation.

4. Wikimedia Producer Run

This video explains Wikimedia producer run using a demo.

5. Wikimedia Producer - Producer Config Intros

This video explains Wikimedia producer-producer Config Intros.

6. Producer Acknowledgements Deep Dive

This video explains Kafka producer acknowledgements.

7. Producer Retries

This video explains producer retries.

8. Idempotent Producer

This video explains idempotent producer.

9. Safe Kafka Producer Settings

This video explains the safe Kafka producer settings.

10. Wikimedia Producer Safe Producer Implementation

This video explains Wikimedia producer safe producer implementation.

11. Kafka Message Compression

This video explains Kafka message compression.

12. linger.ms and batch.size Producer Settings

This video explains how to implement a high throughput producer in Kafka.

13. Wikimedia Producer High Throughput Implementation

This video explains Wikimedia producer high throughput implementation.

14. Producer Default Partitioner and Sticky Partitioner

This video explains producer default partitioner and sticky partitioner.

15. [Advanced] max.block.ms and buffer.memory

This video explains max.block.ms and buffer.memory.

10. OpenSearch Consumer and Advanced Consumer Configurations

1. OpenSearch Consumer - Project Overview

This video provides an OpenSearch consumer-project overview.

2. OpenSearch Consumer - Project Setup

This video explains OpenSearch consumer-project setup.

3. Setting Up OpenSearch on Docker

This video focuses on setting up OpenSearch on Docker.

4. Setting Up OpenSearch on the Cloud

This video explains setting up OpenSearch on the Cloud.

5. OpenSearch 101

This video explains OpenSearch 101.

6. OpenSearch Consumer Implementation - Part 1

This video explains OpenSearch consumer implementation - part 1.

7. OpenSearch Consumer Implementation - Part 2

This video explains the setup of Kafka consumer and improving our code.

8. Consumer Delivery Semantics

This video explains consumer delivery semantics.

9. OpenSearch Consumer Implementation Part 3 - Idempotence

This video explains how to make our consumer idempotence.

10. Consumer Offsets Commit Strategies

This video explains consumer offsets commit strategies.

11. OpenSearch Consumer Implementation Part 4 - Delivery Semantics

This video explains OpenSearch consumer implementation delivery semantics.

12. OpenSearch Consumer Implementation Part 5 - Batching Data

This video explains OpenSearch consumer implementation part 5-batching data.

13. Consumer Offset Reset Behavior

This video explains consumer offset reset behavior.

14. OpenSearch Consumer Implementation Part 6 - Replaying Data

This video explains how the Kafka consumer is used to replay data.

15. Consumer Internal Threads

This video explains consumer internal threads.

16. Consumer Replica Fetching - Rack Awareness

This video explains consumer replica fetching-rack awareness.

11. Kafka Extended APIs for Developers

1. Kafka Extended APIs - Overview

This video explains the real-world applications in Kafka.

2. Kafka Connect Introduction

This video introduces you to Kafka Connect.

3. Kafka Connect Wikimedia and ElasticSearch Hands-On

This video explains Kafka Connect Wikimedia and ElasticSearch.

4. Kafka Streams Introduction

This video introduces you to Kafka Streams.

5. Kafka Streams Hands-On

This video demonstrates running a Kafka streams application.

6. Kafka Schema Registry Introduction

This video is an introduction to Kafka Schema Registry.

7. Kafka Schema Registry Hands-On

This video explains how to start Kafka with Schema Registry and Conduktor using Docker.

8. Which Kafka API Should I Use?

This video explains which Kafka API to use.

12. Real-World Insights and Case Studies (Big Data / Fast Data)

1. Choosing Partition Count and Replication Factor

This video explains how to determine the partition count and replication factor.

2. Kafka Topics Naming Convention

This video explains Kafka Topics naming convention.

3. Case Study - MovieFlix

This video explains the MovieFlix case study in Kafka.

4. Case Study - GetTaxi

This video explains the GetTaxi case study in Kafka.

5. Case Study - MySocialMedia

This video explains the MySocialMedia case study in Kafka.

6. Case Study - MyBank

This video explains the MyBank case study in Kafka.

7. Case Study - Big Data Ingestion

This video explains the big data ingestion case study in Kafka.

8. Case Study - Logging and Metrics Aggregation

This video explains the logging and metrics aggregation case study in Kafka.

13. Kafka in the Enterprise for Admins

1. Kafka Cluster Setup High-Level Architecture Overview

This video provides an overview of the Kafka cluster setup and high-level architecture.

2. Kafka Monitoring and Operations

This video explains the Kafka monitoring setup and operations.

3. Kafka Security

This video explains the concept of Kafka security.

4. Kafka Multi-Cluster and MirrorMaker

This video explains the concept of Kafka multi-cluster and MirrorMaker tool.

5. Advertised Listeners: Kafka Client and Server Communication Protocol

This video explains advertised listeners: Kafka client and server communication protocol.

14. Advanced Topics Configurations

1. Changing a Topic Configuration

This video explains how to change the topic configuration settings.

2. Segment and Indexes

This video explains the concept of segment and indexes files in Kafka.

3. Log Cleanup Policies

This video explains the Kafka log cleanup policies.

4. Log Cleanup Delete

This video explains how to delete the log cleanup policy.

5. Log Compaction Theory

This video explains the Kafka log compaction theory.

6. Log Compaction Practice

This video explains how to allow log compaction into practice.

7. Unclean Leader Election

This video explains how to enable an unclean leader election in Kafka.

8. Large Messages in Kafka

This video explains large messages in Kafka.

15. Next Steps

1. What's Next?

This video highlights the next concepts you can learn in the world of Apache Kafka.

Course Images

Apache Kafka Series - Learn Apache Kafka for Beginners v3

By Packt

Booking options

Highlights