36 Apache Spark courses

Get hands-on with Kafka monitoring setup with Prometheus and Grafana, Kafka operations and Kafka cluster upgrades Setup in AWS.

Elasticsearch and Elastic Stack are important tools for managing massive data. You need to know the problems it solves and how it works to design the best systems and be the most valuable engineer you can be. Explore Elasticsearch 8 and learn to manage operations on your Elastic Stack with this comprehensive course. This course covers it all, from installation to operations.

This course brings together all the important topics related to modern distributed applications and systems in one place. Explore the common challenges that appear while designing and implementing large-scale distributed systems, and how big-tech companies solve those problems. Throughout the course, we are going to build a distributed URL shortening service.

Better than REST APIs! Build a fast and scalable HTTP/2 API for a Go microservice with gRPC and protocol buffers (protobufs)

Duration 4 Days 24 CPD hours This course is intended for This class is intended for experienced developers who are responsible for managing big data transformations including: Extracting, loading, transforming, cleaning, and validating data. Designing pipelines and architectures for data processing. Creating and maintaining machine learning and statistical models. Querying datasets, visualizing query results and creating reports Overview Design and build data processing systems on Google Cloud Platform. Leverage unstructured data using Spark and ML APIs on Cloud Dataproc. Process batch and streaming data by implementing autoscaling data pipelines on Cloud Dataflow. Derive business insights from extremely large datasets using Google BigQuery. Train, evaluate and predict using machine learning models using TensorFlow and Cloud ML. Enable instant insights from streaming data Get hands-on experience with designing and building data processing systems on Google Cloud. This course uses lectures, demos, and hand-on labs to show you how to design data processing systems, build end-to-end data pipelines, analyze data, and implement machine learning. This course covers structured, unstructured, and streaming data. INTRODUCTION TO DATA ENGINEERING * Explore the role of a data engineer. * Analyze data engineering challenges. * Intro to BigQuery. * Data Lakes and Data Warehouses. * Demo: Federated Queries with BigQuery. * Transactional Databases vs Data Warehouses. * Website Demo: Finding PII in your dataset with DLP API. * Partner effectively with other data teams. * Manage data access and governance. * Build production-ready pipelines. * Review GCP customer case study. * Lab: Analyzing Data with BigQuery. BUILDING A DATA LAKE * Introduction to Data Lakes. * Data Storage and ETL options on GCP. * Building a Data Lake using Cloud Storage. * Optional Demo: Optimizing cost with Google Cloud Storage classes and Cloud Functions. * Securing Cloud Storage. * Storing All Sorts of Data Types. * Video Demo: Running federated queries on Parquet and ORC files in BigQuery. * Cloud SQL as a relational Data Lake. * Lab: Loading Taxi Data into Cloud SQL. BUILDING A DATA WAREHOUSE * The modern data warehouse. * Intro to BigQuery. * Demo: Query TB+ of data in seconds. * Getting Started. * Loading Data. * Video Demo: Querying Cloud SQL from BigQuery. * Lab: Loading Data into BigQuery. * Exploring Schemas. * Demo: Exploring BigQuery Public Datasets with SQL using INFORMATION_SCHEMA. * Schema Design. * Nested and Repeated Fields. * Demo: Nested and repeated fields in BigQuery. * Lab: Working with JSON and Array data in BigQuery. * Optimizing with Partitioning and Clustering. * Demo: Partitioned and Clustered Tables in BigQuery. * Preview: Transforming Batch and Streaming Data. INTRODUCTION TO BUILDING BATCH DATA PIPELINES * EL, ELT, ETL. * Quality considerations. * How to carry out operations in BigQuery. * Demo: ELT to improve data quality in BigQuery. * Shortcomings. * ETL to solve data quality issues. EXECUTING SPARK ON CLOUD DATAPROC * The Hadoop ecosystem. * Running Hadoop on Cloud Dataproc. * GCS instead of HDFS. * Optimizing Dataproc. * Lab: Running Apache Spark jobs on Cloud Dataproc. SERVERLESS DATA PROCESSING WITH CLOUD DATAFLOW * Cloud Dataflow. * Why customers value Dataflow. * Dataflow Pipelines. * Lab: A Simple Dataflow Pipeline (Python/Java). * Lab: MapReduce in Dataflow (Python/Java). * Lab: Side Inputs (Python/Java). * Dataflow Templates. * Dataflow SQL. MANAGE DATA PIPELINES WITH CLOUD DATA FUSION AND CLOUD COMPOSER * Building Batch Data Pipelines visually with Cloud Data Fusion. * Components. * UI Overview. * Building a Pipeline. * Exploring Data using Wrangler. * Lab: Building and executing a pipeline graph in Cloud Data Fusion. * Orchestrating work between GCP services with Cloud Composer. * Apache Airflow Environment. * DAGs and Operators. * Workflow Scheduling. * Optional Long Demo: Event-triggered Loading of data with Cloud Composer, Cloud Functions, Cloud Storage, and BigQuery. * Monitoring and Logging. * Lab: An Introduction to Cloud Composer. INTRODUCTION TO PROCESSING STREAMING DATA * Processing Streaming Data. SERVERLESS MESSAGING WITH CLOUD PUB/SUB * Cloud Pub/Sub. * Lab: Publish Streaming Data into Pub/Sub. CLOUD DATAFLOW STREAMING FEATURES * Cloud Dataflow Streaming Features. * Lab: Streaming Data Pipelines. HIGH-THROUGHPUT BIGQUERY AND BIGTABLE STREAMING FEATURES * BigQuery Streaming Features. * Lab: Streaming Analytics and Dashboards. * Cloud Bigtable. * Lab: Streaming Data Pipelines into Bigtable. ADVANCED BIGQUERY FUNCTIONALITY AND PERFORMANCE * Analytic Window Functions. * Using With Clauses. * GIS Functions. * Demo: Mapping Fastest Growing Zip Codes with BigQuery GeoViz. * Performance Considerations. * Lab: Optimizing your BigQuery Queries for Performance. * Optional Lab: Creating Date-Partitioned Tables in BigQuery. INTRODUCTION TO ANALYTICS AND AI * What is AI?. * From Ad-hoc Data Analysis to Data Driven Decisions. * Options for ML models on GCP. PREBUILT ML MODEL APIS FOR UNSTRUCTURED DATA * Unstructured Data is Hard. * ML APIs for Enriching Data. * Lab: Using the Natural Language API to Classify Unstructured Text. BIG DATA ANALYTICS WITH CLOUD AI PLATFORM NOTEBOOKS * What's a Notebook. * BigQuery Magic and Ties to Pandas. * Lab: BigQuery in Jupyter Labs on AI Platform. PRODUCTION ML PIPELINES WITH KUBEFLOW * Ways to do ML on GCP. * Kubeflow. * AI Hub. * Lab: Running AI models on Kubeflow. CUSTOM MODEL BUILDING WITH SQL IN BIGQUERY ML * BigQuery ML for Quick Model Building. * Demo: Train a model with BigQuery ML to predict NYC taxi fares. * Supported Models. * Lab Option 1: Predict Bike Trip Duration with a Regression Model in BQML. * Lab Option 2: Movie Recommendations in BigQuery ML. CUSTOM MODEL BUILDING WITH CLOUD AUTOML * Why Auto ML? * Auto ML Vision. * Auto ML NLP. * Auto ML Tables.

Duration 2 Days 12 CPD hours This course is intended for Audience: Data Scientists, Software Developers, IT Architects, and Technical Managers. Participants should have the general knowledge of statistics and programming Also familiar with Python Overview ? NumPy, pandas, Matplotlib, scikit-learn ? Python REPLs ? Jupyter Notebooks ? Data analytics life-cycle phases ? Data repairing and normalizing ? Data aggregation and grouping ? Data visualization ? Data science algorithms for supervised and unsupervised machine learning Covers theoretical and technical aspects of using Python in Applied Data Science projects and Data Logistics use cases. PYTHON FOR DATA SCIENCE * ? Using Modules * ? Listing Methods in a Module * ? Creating Your Own Modules * ? List Comprehension * ? Dictionary Comprehension * ? String Comprehension * ? Python 2 vs Python 3 * ? Sets (Python 3+) * ? Python Idioms * ? Python Data Science ?Ecosystem? * ? NumPy * ? NumPy Arrays * ? NumPy Idioms * ? pandas * ? Data Wrangling with pandas' DataFrame * ? SciPy * ? Scikit-learn * ? SciPy or scikit-learn? * ? Matplotlib * ? Python vs R * ? Python on Apache Spark * ? Python Dev Tools and REPLs * ? Anaconda * ? IPython * ? Visual Studio Code * ? Jupyter * ? Jupyter Basic Commands * ? Summary APPLIED DATA SCIENCE * ? What is Data Science? * ? Data Science Ecosystem * ? Data Mining vs. Data Science * ? Business Analytics vs. Data Science * ? Data Science, Machine Learning, AI? * ? Who is a Data Scientist? * ? Data Science Skill Sets Venn Diagram * ? Data Scientists at Work * ? Examples of Data Science Projects * ? An Example of a Data Product * ? Applied Data Science at Google * ? Data Science Gotchas * ? Summary DATA ANALYTICS LIFE-CYCLE PHASES * ? Big Data Analytics Pipeline * ? Data Discovery Phase * ? Data Harvesting Phase * ? Data Priming Phase * ? Data Logistics and Data Governance * ? Exploratory Data Analysis * ? Model Planning Phase * ? Model Building Phase * ? Communicating the Results * ? Production Roll-out * ? Summary REPAIRING AND NORMALIZING DATA * ? Repairing and Normalizing Data * ? Dealing with the Missing Data * ? Sample Data Set * ? Getting Info on Null Data * ? Dropping a Column * ? Interpolating Missing Data in pandas * ? Replacing the Missing Values with the Mean Value * ? Scaling (Normalizing) the Data * ? Data Preprocessing with scikit-learn * ? Scaling with the scale() Function * ? The MinMaxScaler Object * ? Summary DESCRIPTIVE STATISTICS COMPUTING FEATURES IN PYTHON * ? Descriptive Statistics * ? Non-uniformity of a Probability Distribution * ? Using NumPy for Calculating Descriptive Statistics Measures * ? Finding Min and Max in NumPy * ? Using pandas for Calculating Descriptive Statistics Measures * ? Correlation * ? Regression and Correlation * ? Covariance * ? Getting Pairwise Correlation and Covariance Measures * ? Finding Min and Max in pandas DataFrame * ? Summary DATA AGGREGATION AND GROUPING * ? Data Aggregation and Grouping * ? Sample Data Set * ? The pandas.core.groupby.SeriesGroupBy Object * ? Grouping by Two or More Columns * ? Emulating the SQL's WHERE Clause * ? The Pivot Tables * ? Cross-Tabulation * ? Summary DATA VISUALIZATION WITH MATPLOTLIB * ? Data Visualization ? What is matplotlib? ? Getting Started with matplotlib ? The Plotting Window ? The Figure Options ? The matplotlib.pyplot.plot() Function ? The matplotlib.pyplot.bar() Function ? The matplotlib.pyplot.pie () Function ? Subplots ? Using the matplotlib.gridspec.GridSpec Object ? The matplotlib.pyplot.subplot() Function ? Hands-on Exercise ? Figures ? Saving Figures to File ? Visualization with pandas ? Working with matplotlib in Jupyter Notebooks ? Summary DATA SCIENCE AND ML ALGORITHMS IN SCIKIT-LEARN * ? Data Science, Machine Learning, AI? * ? Types of Machine Learning * ? Terminology: Features and Observations * ? Continuous and Categorical Features (Variables) * ? Terminology: Axis * ? The scikit-learn Package * ? scikit-learn Estimators * ? Models, Estimators, and Predictors * ? Common Distance Metrics * ? The Euclidean Metric * ? The LIBSVM format * ? Scaling of the Features * ? The Curse of Dimensionality * ? Supervised vs Unsupervised Machine Learning * ? Supervised Machine Learning Algorithms * ? Unsupervised Machine Learning Algorithms * ? Choose the Right Algorithm * ? Life-cycles of Machine Learning Development * ? Data Split for Training and Test Data Sets * ? Data Splitting in scikit-learn * ? Hands-on Exercise * ? Classification Examples * ? Classifying with k-Nearest Neighbors (SL) * ? k-Nearest Neighbors Algorithm * ? k-Nearest Neighbors Algorithm * ? The Error Rate * ? Hands-on Exercise * ? Dimensionality Reduction * ? The Advantages of Dimensionality Reduction * ? Principal component analysis (PCA) * ? Hands-on Exercise * ? Data Blending * ? Decision Trees (SL) * ? Decision Tree Terminology * ? Decision Tree Classification in Context of Information Theory * ? Information Entropy Defined * ? The Shannon Entropy Formula * ? The Simplified Decision Tree Algorithm * ? Using Decision Trees * ? Random Forests * ? SVM * ? Naive Bayes Classifier (SL) * ? Naive Bayesian Probabilistic Model in a Nutshell * ? Bayes Formula * ? Classification of Documents with Naive Bayes * ? Unsupervised Learning Type: Clustering * ? Clustering Examples * ? k-Means Clustering (UL) * ? k-Means Clustering in a Nutshell * ? k-Means Characteristics * ? Regression Analysis * ? Simple Linear Regression Model * ? Linear vs Non-Linear Regression * ? Linear Regression Illustration * ? Major Underlying Assumptions for Regression Analysis * ? Least-Squares Method (LSM) * ? Locally Weighted Linear Regression * ? Regression Models in Excel * ? Multiple Regression Analysis * ? Logistic Regression * ? Regression vs Classification * ? Time-Series Analysis * ? Decomposing Time-Series * ? Summary LAB EXERCISES * Lab 1 - Learning the Lab Environment * Lab 2 - Using Jupyter Notebook * Lab 3 - Repairing and Normalizing Data * Lab 4 - Computing Descriptive Statistics * Lab 5 - Data Grouping and Aggregation * Lab 6 - Data Visualization with matplotlib * Lab 7 - Data Splitting * Lab 8 - k-Nearest Neighbors Algorithm * Lab 9 - The k-means Algorithm * Lab 10 - The Random Forest Algorithm

Better than REST APIs! Build a fast and scalable HTTP/2 API for your microservice with gRPC and protocol buffers (protobufs).

OVERVIEW This comprehensive course on SQL NoSQL Big Data and Hadoop will deepen your understanding on this topic. After successful completion of this course you can acquire the required skills in this sector. This SQL NoSQL Big Data and Hadoop comes with accredited certification from CPD, which will enhance your CV and make you worthy in the job market. So enrol in this course today to fast track your career ladder. HOW WILL I GET MY CERTIFICATE? At the end of the course there will be an online written test, which you can take either during or after the course. After successfully completing the test you will be able to order your certificate, these are included in the price. WHO IS THIS COURSE FOR? There is no experience or previous qualifications required for enrolment on this SQL NoSQL Big Data and Hadoop. It is available to all students, of all academic backgrounds. REQUIREMENTS Our SQL NoSQL Big Data and Hadoop is fully compatible with PC's, Mac's, Laptop, Tablet and Smartphone devices. This course has been designed to be fully compatible with tablets and smartphones so you can access your course on Wi-Fi, 3G or 4G. There is no time limit for completing this course, it can be studied in your own time at your own pace. CAREER PATH Learning this new skill will help you to advance in your career. It will diversify your job options and help you develop new techniques to keep up with the fast-changing world. This skillset will help you to- * Open doors of opportunities * Increase your adaptability * Keep you relevant * Boost confidence And much more! COURSE CURRICULUM 14 sections • 130 lectures • 22:34:00 total length •Introduction: 00:07:00 •Building a Data-driven Organization - Introduction: 00:04:00 •Data Engineering: 00:06:00 •Learning Environment & Course Material: 00:04:00 •Movielens Dataset: 00:03:00 •Introduction to Relational Databases: 00:09:00 •SQL: 00:05:00 •Movielens Relational Model: 00:15:00 •Movielens Relational Model: Normalization vs Denormalization: 00:16:00 •MySQL: 00:05:00 •Movielens in MySQL: Database import: 00:06:00 •OLTP in RDBMS: CRUD Applications: 00:17:00 •Indexes: 00:16:00 •Data Warehousing: 00:15:00 •Analytical Processing: 00:17:00 •Transaction Logs: 00:06:00 •Relational Databases - Wrap Up: 00:03:00 •Distributed Databases: 00:07:00 •CAP Theorem: 00:10:00 •BASE: 00:07:00 •Other Classifications: 00:07:00 •Introduction to KV Stores: 00:02:00 •Redis: 00:04:00 •Install Redis: 00:07:00 •Time Complexity of Algorithm: 00:05:00 •Data Structures in Redis : Key & String: 00:20:00 •Data Structures in Redis II : Hash & List: 00:18:00 •Data structures in Redis III : Set & Sorted Set: 00:21:00 •Data structures in Redis IV : Geo & HyperLogLog: 00:11:00 •Data structures in Redis V : Pubsub & Transaction: 00:08:00 •Modelling Movielens in Redis: 00:11:00 •Redis Example in Application: 00:29:00 •KV Stores: Wrap Up: 00:02:00 •Introduction to Document-Oriented Databases: 00:05:00 •MongoDB: 00:04:00 •MongoDB Installation: 00:02:00 •Movielens in MongoDB: 00:13:00 •Movielens in MongoDB: Normalization vs Denormalization: 00:11:00 •Movielens in MongoDB: Implementation: 00:10:00 •CRUD Operations in MongoDB: 00:13:00 •Indexes: 00:16:00 •MongoDB Aggregation Query - MapReduce function: 00:09:00 •MongoDB Aggregation Query - Aggregation Framework: 00:16:00 •Demo: MySQL vs MongoDB. Modeling with Spark: 00:02:00 •Document Stores: Wrap Up: 00:03:00 •Introduction to Search Engine Stores: 00:05:00 •Elasticsearch: 00:09:00 •Basic Terms Concepts and Description: 00:13:00 •Movielens in Elastisearch: 00:12:00 •CRUD in Elasticsearch: 00:15:00 •Search Queries in Elasticsearch: 00:23:00 •Aggregation Queries in Elasticsearch: 00:23:00 •The Elastic Stack (ELK): 00:12:00 •Use case: UFO Sighting in ElasticSearch: 00:29:00 •Search Engines: Wrap Up: 00:04:00 •Introduction to Columnar databases: 00:06:00 •HBase: 00:07:00 •HBase Architecture: 00:09:00 •HBase Installation: 00:09:00 •Apache Zookeeper: 00:06:00 •Movielens Data in HBase: 00:17:00 •Performing CRUD in HBase: 00:24:00 •SQL on HBase - Apache Phoenix: 00:14:00 •SQL on HBase - Apache Phoenix - Movielens: 00:10:00 •Demo : GeoLife GPS Trajectories: 00:02:00 •Wide Column Store: Wrap Up: 00:05:00 •Introduction to Time Series: 00:09:00 •InfluxDB: 00:03:00 •InfluxDB Installation: 00:07:00 •InfluxDB Data Model: 00:07:00 •Data manipulation in InfluxDB: 00:17:00 •TICK Stack I: 00:12:00 •TICK Stack II: 00:23:00 •Time Series Databases: Wrap Up: 00:04:00 •Introduction to Graph Databases: 00:05:00 •Modelling in Graph: 00:14:00 •Modelling Movielens as a Graph: 00:10:00 •Neo4J: 00:04:00 •Neo4J installation: 00:08:00 •Cypher: 00:12:00 •Cypher II: 00:19:00 •Movielens in Neo4J: Data Import: 00:17:00 •Movielens in Neo4J: Spring Application: 00:12:00 •Data Analysis in Graph Databases: 00:05:00 •Examples of Graph Algorithms in Neo4J: 00:18:00 •Graph Databases: Wrap Up: 00:07:00 •Introduction to Big Data With Apache Hadoop: 00:06:00 •Big Data Storage in Hadoop (HDFS): 00:16:00 •Big Data Processing : YARN: 00:11:00 •Installation: 00:13:00 •Data Processing in Hadoop (MapReduce): 00:14:00 •Examples in MapReduce: 00:25:00 •Data Processing in Hadoop (Pig): 00:12:00 •Examples in Pig: 00:21:00 •Data Processing in Hadoop (Spark): 00:23:00 •Examples in Spark: 00:23:00 •Data Analytics with Apache Spark: 00:09:00 •Data Compression: 00:06:00 •Data serialization and storage formats: 00:20:00 •Hadoop: Wrap Up: 00:07:00 •Introduction Big Data SQL Engines: 00:03:00 •Apache Hive: 00:10:00 •Apache Hive : Demonstration: 00:20:00 •MPP SQL-on-Hadoop: Introduction: 00:03:00 •Impala: 00:06:00 •Impala : Demonstration: 00:18:00 •PrestoDB: 00:13:00 •PrestoDB : Demonstration: 00:14:00 •SQL-on-Hadoop: Wrap Up: 00:02:00 •Data Architectures: 00:05:00 •Introduction to Distributed Commit Logs: 00:07:00 •Apache Kafka: 00:03:00 •Confluent Platform Installation: 00:10:00 •Data Modeling in Kafka I: 00:13:00 •Data Modeling in Kafka II: 00:15:00 •Data Generation for Testing: 00:09:00 •Use case: Toll fee Collection: 00:04:00 •Stream processing: 00:11:00 •Stream Processing II with Stream + Connect APIs: 00:19:00 •Example: Kafka Streams: 00:15:00 •KSQL : Streaming Processing in SQL: 00:04:00 •KSQL: Example: 00:14:00 •Demonstration: NYC Taxi and Fares: 00:01:00 •Streaming: Wrap Up: 00:02:00 •Database Polyglot: 00:04:00 •Extending your knowledge: 00:08:00 •Data Visualization: 00:11:00 •Building a Data-driven Organization - Conclusion: 00:07:00 •Conclusion: 00:03:00 •Assignment -SQL NoSQL Big Data and Hadoop: 00:00:00

With this course, you will master all CloudFormation concepts, and become confident in writing CloudFormation templates using YAML. Throughout the course, you will encounter various interesting examples and activities that will help you to consolidate your learning.

This course will show you why Hadoop is one of the best tools to work with big data. With the help of some real-world data sets, you will learn how to use Hadoop and its distributed technologies, such as Spark, Flink, Pig, and Flume, to store, analyze, and scale big data.