Spark Programming in Python for Beginners with Apache Spark 3

Description

If you are looking to expand your knowledge in data engineering or want to level up your portfolio by adding Spark programming to your skillset, then you are in the right place. This course will help you understand Spark programming and apply that knowledge to build data engineering solutions. This course is example-driven and follows a working session-like approach. We will be taking a live coding approach and explaining all the concepts needed along the way. In this course, we will start with a quick introduction to Apache Spark, then set up our environment by installing and using Apache Spark. Next, we will learn about Spark execution model and architecture, and about Spark programming model and developer experience. Next, we will cover Spark structured API foundation and then move towards Spark data sources and sinks. Then we will cover Spark Dataframe and dataset transformations. We will also cover aggregations in Apache Spark and finally, we will cover Spark Dataframe joins. By the end of this course, you will be able to build data engineering solutions using Spark structured API in Python. All the resources for the course are available at https://github.com/PacktPublishing/Spark-Programming-in-Python-for-Beginners-with-Apache-Spark-3

What You Will Learn

Learn Apache Spark Foundation and Spark architecture
Learn data engineering and data processing in Spark
Work with data sources and sinks
Work with data frames and Spark SQL
Use PyCharm IDE for Spark development and debugging
Learn unit testing, managing application logs, and cluster deployment

Audience

This course is designed for software engineers willing to develop a data engineering pipeline and application using Apache Spark; for data architects and data engineers who are responsible for designing and building the organization's data-centric infrastructure, for managers and architects who do not directly work with Spark implementation but work with the people who implement Apache Spark at the ground level.

This course does not require any prior knowledge of Apache Spark or Hadoop; only programming knowledge using Python programming language is required.

Approach

This course is example-driven and follows a working session-like approach. The course delivers live coding sessions and explains the concepts along the way.

Key Features

Build your own data engineering solutions using Spark structured API in Python * Gain an in-depth understanding of the Apache Hadoop architecture, ecosystem, and practices * Learn to apply Spark programming basics

Github Repo

https://github.com/PacktPublishing/Spark-Programming-in-Python-for-Beginners-with-Apache-Spark-3

About the Author

ScholarNest

ScholarNest is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. Together, they have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. They have worked with international software services organizations on various data-centric and Big Data projects. It is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of continuous learning, they started publishing free training videos on their YouTube channel. They conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.

Course Outline

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">

In this section, we will cover an introduction to Apache Spark. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover big data history and primer.

2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will understand the data lake landscape.

3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will understand what Apache Spark is and get a quick introduction and overview of Apache Spark.

2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">

In this section, you will learn how to install and use Apache Spark. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Spark development environments.

2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Mac users to set-up Apache Spark in Local Mode Command Line REPL.

3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Windows users to set-up Apache Spark in local mode command line REPL.

4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Mac users to set-up Apache Spark in the PyCharm IDE.

5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Windows users to set-up Apache Spark in the PyCharm IDE.

6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Apache Spark in cloud and Databricks community and notebooks.

7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will explore Apache Spark in Anaconda and Jupyter notebook.

3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">

In this section, you will learn about Spark execution model and architecture. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover execution methods and learn how to run Spark programs.

2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will understand Spark distributed processing model and how your program runs?

3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Spark execution modes and cluster managers.

4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we willsummarize Spark execution models and understand when to use what?

5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this demo video, you will learn how to work with PySpark shell.

6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this demo video, you will learn how to install multi-node Spark cluster.

7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this demo video, you will learn how to work with notebooks in the cluster.

8. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this demo video, you will learn how to work with Spark submit.

9. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will summarize our learning from this section.

4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">

In this section, you will learn about Spark programming model and developer experience. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, youwill learn how tocreate Spark project build configuration.

2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, youwill learn how toconfigure Spark project application logs.

3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, youwill learn how tocreate Spark session.

4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, youwill learn how toconfigure Spark session.

5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover anintroductionto data frame.

6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover data frame partitions and executors.

7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Spark transformations and actions.

8. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Spark jobs stages and task.

9. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, youwill learn how tounderstand your execution plan.

10. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover unit testing Spark application.

11. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will summarize our learning from this section.

5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">

In this section, you will learn about Spark structured API foundation. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover an introduction to Spark APIs.

2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover an introduction to Spark RDD API.

3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this demo video, you will learn how to work with Spark SQL.

4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Spark SQL engine and catalyst optimizer.

5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will summarize our learning from this section.

6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">

In this section, you will learn about Spark data sources and sinks. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Spark data sources and sinks.

2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Spark DataFrameReader API.

3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, youwill learn how toread CSV, JSON and Parquet files.

4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, youwill learn how tocreate Spark DataFrame schema.

5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Spark DataFrameWriter API.

6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, youwill learn how towrite your data and manage layout.

7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Spark databases and tables.

8. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this demo video, you will learn how to work with Spark SQL tables.

7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">

In this section, you will learn about Spark Dataframe and Dataset transformations. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover an introduction to data transformation.

2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this demo video, you will learn how to work with Dataframe rows.

3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover DataFrame rows and unit testing.

4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Dataframe rows and unstructured data.

5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this demo video, you will learn how to work with Dataframe columns.

6. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, youwill learn how tocreate and use UDF.

7. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Misc transformations.

8. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">

In this section, you will learn about aggregations in Apache Spark. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Aggregating Dataframes.

2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Grouping aggregations.

3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Windowing aggregations.

9. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">

In this section, you will learn about Spark Dataframe joins. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover Dataframe joins and column name ambiguity.

2. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover outer joins in Dataframe.

3. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, we will cover the internals of Spark join and shuffle.

4. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, you will learn how to optimize your joins.

5. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> In this video, you will learn how to implement Bucket joins.

10. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;">

Welcome to the concluding section of the course. 

1. 0px;padding:0px;clear:both;font-family:'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif;font-size:12px;"> Thank you for joining. Happy learning!