Real-Time Stream Processing Using Apache Spark 3 for Scala Developers

Description

Since its inception, Apache Spark has seen rapid adoption by enterprises across a wide range of industries. So, mastering Apache Spark opens a wide range of professional opportunities. If you are a software engineer or architect and want to design or build your own projects, then this is the right course for you. This is a hands-on, example-driven, advanced course with demonstrations and coding sessions. This course will help you understand real-time stream processing using Apache Spark and later, you will be able to apply that knowledge to build real-time stream processing solutions. This course covers everything from scratch, which involves installing Apache Spark and seeing how to set up and run Apache Kafka. Furthermore, it introduces stream processing and how to work with files and directories. You will also explore Kafka serialization and deserialization for Spark and how to work with Kafka AVRO Source. And finally, the course wraps up with streaming Watermark and outer joints. By the end of this course, you will be able to design and develop big data engineering projects. You will be able to create real-time stream processing applications with Apache Spark. This course will also help you further your growth in real-time stream processing. All resources and code files are placed here: https://github.com/PacktPublishing/Spark-Streaming-In-Scala

What You Will Learn

Create arbitrary streaming sinks
Explore Kafka Source and integrate Spark with Kafka
Learn state-less and state-full streaming transformations
Learn to handle memory problems with streaming joins
Learn to work with file streams
Explore windowing aggregates using Spark Stream

Audience

This course is designed for software engineers and architects who aspire to develop big data engineering projects using Apache Spark. Also, if you are a programmer and developer who wants to grow and learn data engineering using Apache Spark, then this course is for you. Another group of people that can opt for this course are the managers and architects who might not directly work with Spark implementation but still work with the people who implement Apache Spark at the ground level.

Approach

This is a comprehensive, hands-on, example-driven course, and it follows a working session-like approach. There is a combination of both theory and demonstration. The theory part explains all the concepts needed and then demonstrations to help you master real-time stream processing. It provides real-world examples to give you practical learning experience.

Key Features

Deep dive into Spark structured streaming APIs and architecture * Discover streaming joins and aggregation * Explore real-time stream processing concepts

Github Repo

https://github.com/PacktPublishing/Spark-Streaming-In-Scala

About the Author

Scholar Nest

ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects. It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.

Course Outline

1. Before you start

2. Setup your Environment

3. Getting started with Spark Structured Streaming

4. Spark Streaming with Kafka

5. Windowing and Aggregates

6. Stream Processing and Joins

7. Keep Learning