Apache Kafka Essential Training: Building Scalable Applications

Course details

Scalable and distributed message queuing plays an important role in building real time big data pipelines. Asynchronous publisher/subscriber models are required to handle unpredictable loads in these pipelines. Apache Kafka is the leading technology today that provides these capabilities and is an essential skill for a big data professional. In this course, Kumaran Ponnambalam provides insights into the scalability and manageability aspects of Kafka and demonstrates how to build asynchronous applications with Kafka and Java. Kumaran starts by demonstrating how to set up a Kafka cluster and explores the basics of Java programming in Kafka. He then takes a deep dive into the various messaging and schema options available. Kumaran also goes over some best practices for designing Kafka applications before finishing with a use case project that applies the lessons covered in the course.

Concepts

0. Introduction

01 - Building robust Kafka applications

1. Introduction to Kafka

02 - What is Kafka
03 - Prerequisites for the course
04 - Kafka scaling and resiliency
05 - Setting up the exercise files

2. Kafka Scaling Concepts

06 - A Kafka cluster
07 - Kafka controllers
08 - Replication
09 - Partition leaders
10 - Security

3. Building a Kafka Cluster

11 - Kafka cluster setup
12 - Running the cluster
13 - Creating topics with replication
14 - Kafka clusters in action
15 - Kafka resiliency in action

4. Building Scalable Producers

16 - Producer internals
17 - Producer publishing options
18 - Acknowledgments in Kafka
19 - Additional producer parameters
20 - Java producer options example

5. Building Scalable Consumers

21 - How consumer works
22 - Batching message consumption
23 - Committing messages
24 - Java consumer example
25 - Multi-threaded consumers

6. Kafka Best Practices

26 - Managing partition counts
27 - Managing messages
28 - Managing consumer settings
29 - Managing resiliency

7. Use Case Project

30 - Kafka applications use case - Problem definition
31 - Setting up topics
32 - Producing data in Java
33 - Consuming data in Java

Conclusion

34 - How do you extend your Kafka learning journey