93 Data Engineering courses

Duration 4 Days 24 CPD hours This course is intended for The primary audience for this course is data professionals, data architects, and business intelligence professionals who want to learn about data engineering and building analytical solutions using data platform technologies that exist on Microsoft Azure. The secondary audience for this course includes data analysts and data scientists who work with analytical solutions built on Microsoft Azure. In this course, the student will learn how to implement and manage data engineering workloads on Microsoft Azure, using Azure services such as Azure Synapse Analytics, Azure Data Lake Storage Gen2, Azure Stream Analytics, Azure Databricks, and others. The course focuses on common data engineering tasks such as orchestrating data transfer and transformation pipelines, working with data files in a data lake, creating and loading relational data warehouses, capturing and aggregating streams of real-time data, and tracking data assets and lineage. Prerequisites Successful students start this course with knowledge of cloud computing and core data concepts and professional experience with data solutions. AZ-900T00 Microsoft Azure Fundamentals DP-900T00 Microsoft Azure Data Fundamentals 1 - INTRODUCTION TO DATA ENGINEERING ON AZURE * What is data engineering * Important data engineering concepts * Data engineering in Microsoft Azure 2 - INTRODUCTION TO AZURE DATA LAKE STORAGE GEN2 * Understand Azure Data Lake Storage Gen2 * Enable Azure Data Lake Storage Gen2 in Azure Storage * Compare Azure Data Lake Store to Azure Blob storage * Understand the stages for processing big data * Use Azure Data Lake Storage Gen2 in data analytics workloads 3 - INTRODUCTION TO AZURE SYNAPSE ANALYTICS * What is Azure Synapse Analytics * How Azure Synapse Analytics works * When to use Azure Synapse Analytics 4 - USE AZURE SYNAPSE SERVERLESS SQL POOL TO QUERY FILES IN A DATA LAKE * Understand Azure Synapse serverless SQL pool capabilities and use cases * Query files using a serverless SQL pool * Create external database objects 5 - USE AZURE SYNAPSE SERVERLESS SQL POOLS TO TRANSFORM DATA IN A DATA LAKE * Transform data files with the CREATE EXTERNAL TABLE AS SELECT statement * Encapsulate data transformations in a stored procedure * Include a data transformation stored procedure in a pipeline 6 - CREATE A LAKE DATABASE IN AZURE SYNAPSE ANALYTICS * Understand lake database concepts * Explore database templates * Create a lake database * Use a lake database 7 - ANALYZE DATA WITH APACHE SPARK IN AZURE SYNAPSE ANALYTICS * Get to know Apache Spark * Use Spark in Azure Synapse Analytics * Analyze data with Spark * Visualize data with Spark 8 - TRANSFORM DATA WITH SPARK IN AZURE SYNAPSE ANALYTICS * Modify and save dataframes * Partition data files * Transform data with SQL 9 - USE DELTA LAKE IN AZURE SYNAPSE ANALYTICS * Understand Delta Lake * Create Delta Lake tables * Create catalog tables * Use Delta Lake with streaming data * Use Delta Lake in a SQL pool 10 - ANALYZE DATA IN A RELATIONAL DATA WAREHOUSE * Design a data warehouse schema * Create data warehouse tables * Load data warehouse tables * Query a data warehouse 11 - LOAD DATA INTO A RELATIONAL DATA WAREHOUSE * Load staging tables * Load dimension tables * Load time dimension tables * Load slowly changing dimensions * Load fact tables * Perform post load optimization 12 - BUILD A DATA PIPELINE IN AZURE SYNAPSE ANALYTICS * Understand pipelines in Azure Synapse Analytics * Create a pipeline in Azure Synapse Studio * Define data flows * Run a pipeline 13 - USE SPARK NOTEBOOKS IN AN AZURE SYNAPSE PIPELINE * Understand Synapse Notebooks and Pipelines * Use a Synapse notebook activity in a pipeline * Use parameters in a notebook 14 - PLAN HYBRID TRANSACTIONAL AND ANALYTICAL PROCESSING USING AZURE SYNAPSE ANALYTICS * Understand hybrid transactional and analytical processing patterns * Describe Azure Synapse Link 15 - IMPLEMENT AZURE SYNAPSE LINK WITH AZURE COSMOS DB * Enable Cosmos DB account to use Azure Synapse Link * Create an analytical store enabled container * Create a linked service for Cosmos DB * Query Cosmos DB data with Spark * Query Cosmos DB with Synapse SQL 16 - IMPLEMENT AZURE SYNAPSE LINK FOR SQL * What is Azure Synapse Link for SQL? * Configure Azure Synapse Link for Azure SQL Database * Configure Azure Synapse Link for SQL Server 2022 17 - GET STARTED WITH AZURE STREAM ANALYTICS * Understand data streams * Understand event processing * Understand window functions 18 - INGEST STREAMING DATA USING AZURE STREAM ANALYTICS AND AZURE SYNAPSE ANALYTICS * Stream ingestion scenarios * Configure inputs and outputs * Define a query to select, filter, and aggregate data * Run a job to ingest data 19 - VISUALIZE REAL-TIME DATA WITH AZURE STREAM ANALYTICS AND POWER BI * Use a Power BI output in Azure Stream Analytics * Create a query for real-time visualization * Create real-time data visualizations in Power BI 20 - INTRODUCTION TO MICROSOFT PURVIEW * What is Microsoft Purview? * How Microsoft Purview works * When to use Microsoft Purview 21 - INTEGRATE MICROSOFT PURVIEW AND AZURE SYNAPSE ANALYTICS * Catalog Azure Synapse Analytics data assets in Microsoft Purview * Connect Microsoft Purview to an Azure Synapse Analytics workspace * Search a Purview catalog in Synapse Studio * Track data lineage in pipelines 22 - EXPLORE AZURE DATABRICKS * Get started with Azure Databricks * Identify Azure Databricks workloads * Understand key concepts 23 - USE APACHE SPARK IN AZURE DATABRICKS * Get to know Spark * Create a Spark cluster * Use Spark in notebooks * Use Spark to work with data files * Visualize data 24 - RUN AZURE DATABRICKS NOTEBOOKS WITH AZURE DATA FACTORY * Understand Azure Databricks notebooks and pipelines * Create a linked service for Azure Databricks * Use a Notebook activity in a pipeline * Use parameters in a notebook ADDITIONAL COURSE DETAILS: Nexus Humans DP-203T00 Data Engineering on Microsoft Azure training program is a workshop that presents an invigorating mix of sessions, lessons, and masterclasses meticulously crafted to propel your learning expedition forward. This immersive bootcamp-style experience boasts interactive lectures, hands-on labs, and collaborative hackathons, all strategically designed to fortify fundamental concepts. Guided by seasoned coaches, each session offers priceless insights and practical skills crucial for honing your expertise. Whether you're stepping into the realm of professional skills or a seasoned professional, this comprehensive course ensures you're equipped with the knowledge and prowess necessary for success. While we feel this is the best course for the DP-203T00 Data Engineering on Microsoft Azure course and one of our Top 10 we encourage you to read the course outline to make sure it is the right content for you. Additionally, private sessions, closed classes or dedicated events are available both live online and at our training centres in Dublin and London, as well as at your offices anywhere in the UK, Ireland or across EMEA.

Duration 4 Days 24 CPD hours This course is intended for This course is designed for data scientists with existing knowledge of Python and machine learning frameworks like Scikit-Learn, PyTorch, and Tensorflow, who want to build and operate machine learning solutions in the cloud. Overview Learn how to operate machine learning solutions at cloud scale using Azure Machine Learning. This course teaches you to leverage your existing knowledge of Python and machine learning to manage data ingestion and preparation, model training and deployment, and machine learning solution monitoring in Microsoft Azure. Learn how to operate machine learning solutions at cloud scale using Azure Machine Learning. This course teaches you to leverage your existing knowledge of Python and machine learning to manage data ingestion and preparation, model training and deployment, and machine learning solution monitoring with Azure Machine Learning and MLflow. Prerequisites Creating cloud resources in Microsoft Azure. Using Python to explore and visualize data. Training and validating machine learning models using common frameworks like Scikit-Learn, PyTorch, and TensorFlow. Working with containers AI-900T00: Microsoft Azure AI Fundamentals is recommended, or the equivalent experience. 1 - DESIGN A DATA INGESTION STRATEGY FOR MACHINE LEARNING PROJECTS * Identify your data source and format * Choose how to serve data to machine learning workflows * Design a data ingestion solution 2 - DESIGN A MACHINE LEARNING MODEL TRAINING SOLUTION * Identify machine learning tasks * Choose a service to train a machine learning model * Decide between compute options 3 - DESIGN A MODEL DEPLOYMENT SOLUTION * Understand how model will be consumed * Decide on real-time or batch deployment 4 - DESIGN A MACHINE LEARNING OPERATIONS SOLUTION * Explore an MLOps architecture * Design for monitoring * Design for retraining 5 - EXPLORE AZURE MACHINE LEARNING WORKSPACE RESOURCES AND ASSETS * Create an Azure Machine Learning workspace * Identify Azure Machine Learning resources * Identify Azure Machine Learning assets * Train models in the workspace 6 - EXPLORE DEVELOPER TOOLS FOR WORKSPACE INTERACTION * Explore the studio * Explore the Python SDK * Explore the CLI 7 - MAKE DATA AVAILABLE IN AZURE MACHINE LEARNING * Understand URIs * Create a datastore * Create a data asset 8 - WORK WITH COMPUTE TARGETS IN AZURE MACHINE LEARNING * Choose the appropriate compute target * Create and use a compute instance * Create and use a compute cluster 9 - WORK WITH ENVIRONMENTS IN AZURE MACHINE LEARNING * Understand environments * Explore and use curated environments * Create and use custom environments 10 - FIND THE BEST CLASSIFICATION MODEL WITH AUTOMATED MACHINE LEARNING * Preprocess data and configure featurization * Run an Automated Machine Learning experiment * Evaluate and compare models 11 - TRACK MODEL TRAINING IN JUPYTER NOTEBOOKS WITH MLFLOW * Configure MLflow for model tracking in notebooks * Train and track models in notebooks 12 - RUN A TRAINING SCRIPT AS A COMMAND JOB IN AZURE MACHINE LEARNING * Convert a notebook to a script * Run a script as a command job * Use parameters in a command job 13 - TRACK MODEL TRAINING WITH MLFLOW IN JOBS * Track metrics with MLflow * View metrics and evaluate models 14 - PERFORM HYPERPARAMETER TUNING WITH AZURE MACHINE LEARNING * Define a search space * Configure a sampling method * Configure early termination * Use a sweep job for hyperparameter tuning 15 - RUN PIPELINES IN AZURE MACHINE LEARNING * Create components * Create a pipeline * Run a pipeline job 16 - REGISTER AN MLFLOW MODEL IN AZURE MACHINE LEARNING * Log models with MLflow * Understand the MLflow model format * Register an MLflow model 17 - CREATE AND EXPLORE THE RESPONSIBLE AI DASHBOARD FOR A MODEL IN AZURE MACHINE LEARNING * Understand Responsible AI * Create the Responsible AI dashboard * Evaluate the Responsible AI dashboard 18 - DEPLOY A MODEL TO A MANAGED ONLINE ENDPOINT * Explore managed online endpoints * Deploy your MLflow model to a managed online endpoint * Deploy a model to a managed online endpoint * Test managed online endpoints 19 - DEPLOY A MODEL TO A BATCH ENDPOINT * Understand and create batch endpoints * Deploy your MLflow model to a batch endpoint * Deploy a custom model to a batch endpoint * Invoke and troubleshoot batch endpoints

24 Hour Flash Deal **25-in-1 Data Engineering Advanced Diploma Mega Bundle** Data Engineering Advanced Diploma Enrolment Gifts **FREE PDF Certificate**FREE PDF Transcript ** FREE Exam** FREE Student ID ** Lifetime Access **FREE Enrolment Letter ** Take the initial steps toward a successful long-term career by studying the Data Engineering Advanced Diploma package online with Studyhub through our online learning platform. The Data Engineering Advanced Diploma bundle can help you improve your CV, wow potential employers, and differentiate yourself from the mass. This Data Engineering Advanced Diploma course provides complete 360-degree training on Data Engineering Advanced Diploma. You'll get not one, not two, not three, but twenty-five Data Engineering Advanced Diploma courses included in this course. Plus Studyhub's signature Forever Access is given as always, meaning these Data Engineering Advanced Diploma courses are yours for as long as you want them once you enrol in this course This Data Engineering Advanced Diploma Bundle consists the following career oriented courses: * Course 01: Data Analysis * Course 02: Data Science & Machine Learning with R * Course 03: Data Analytics with Tableau * Course 04: SQL Programming * Course 05: Excel Pivot Tables, Pivot Charts, Slicers, and Timelines * Course 06: Data Analytics * Course 07: Python Data Science with Numpy, Pandas and Matplotlib * Course 08: Google Analytics * Course 09: Business Intelligence and Data Mining * Course 10: Microsoft Excel: Automated Dashboard Using Advanced Formula, VBA, Power Query * Course 11: JavaScript Functions * Course 12: Data Protection and Data Security (GDPR) Training Certification * Course 13: GDPR Training * Course 14: Diploma in Python Programming * Course 15: Data Center Training Essentials: General Introduction * Course 16: Quick Data Science Approach from Scratch * Course 17: Diploma in Statistics & Probability for Data Science & Machine Learning * Course 18: Master JavaScript with Data Visualization * Course 19: Data Analysis and Forecasting in Excel * Course 20: AWS Certified Solutions Architect Associate Preparation * Course 21: Statistical Analysis Course * Course 22: SQL for Data Science, Data Analytics and Data Visualization * Course 23: Learn MySQL from Scratch for Data Science and Analytics * Course 24: Data Analysis & Forecasting * Course 25: Data Structure The Data Engineering Advanced Diploma course has been prepared by focusing largely on Data Engineering Advanced Diploma career readiness. It has been designed by our Data Engineering Advanced Diploma specialists in a manner that you will be likely to find yourself head and shoulders above the others. For better learning, one to one assistance will also be provided if it's required by any learners. The Data Engineering Advanced Diploma Bundle is one of the most prestigious training offered at StudyHub and is highly valued by employers for good reason. This Data Engineering Advanced Diploma bundle course has been created with twenty-five premium courses to provide our learners with the best learning experience possible to increase their understanding of their chosen field. This Data Engineering Advanced Diploma Course, like every one of Study Hub's courses, is meticulously developed and well researched. Every one of the topics is divided into Data Engineering Advanced Diploma Elementary modules, allowing our students to grasp each lesson quickly. The Data Engineering Advanced Diploma course is self-paced and can be taken from the comfort of your home, office, or on the go! With our Student ID card you will get discounts on things like music, food, travel and clothes etc. In this exclusive Data Engineering Advanced Diploma bundle, you really hit the jackpot. Here's what you get: * Step by step Data Engineering Advanced Diploma lessons * One to one assistance from Data Engineering Advanced Diplomaprofessionals if you need it * Innovative exams to test your knowledge after the Data Engineering Advanced Diplomacourse * 24/7 customer support should you encounter any hiccups * Top-class learning portal * Unlimited lifetime access to all twenty-five Data Engineering Advanced Diploma courses * Digital Certificate, Transcript and student ID are all included in the price * PDF certificate immediately after passing * Original copies of your Data Engineering Advanced Diploma certificate and transcript on the next working day * Easily learn the Data Engineering Advanced Diploma skills and knowledge you want from the comfort of your home CPD 250 CPD hours / points Accredited by CPD Quality Standards WHO IS THIS COURSE FOR? This Data Engineering Advanced Diploma training is suitable for - * Students * Recent graduates * Job Seekers * Individuals who are already employed in the relevant sectors and wish to enhance their knowledge and expertise in Data Engineering Advanced Diploma REQUIREMENTS To participate in this Data Engineering Advanced Diploma course, all you need is - * A smart device * A secure internet connection * And a keen interest in Data Engineering Advanced Diploma CAREER PATH You will be able to kickstart your Data Engineering Advanced Diploma career because this course includes various courses as a bonus. This Data Engineering Advanced Diploma is an excellent opportunity for you to learn multiple skills from the convenience of your own home and explore Data Engineering Advanced Diploma career opportunities. CERTIFICATES CPD ACCREDITED CERTIFICATE Digital certificate - Included * CPD Accredited e-Certificate - Free * CPD Accredited Hardcopy Certificate - Free * Enrolment Letter - Free * Student ID Card - Free

If you aim to enhance your Data Engineering skills, our comprehensive Data Engineering course is perfect for you. Designed for success, this Data Engineering course covers everything from basics to advanced topics in Data Engineering. Each lesson in this Data Engineering course is crafted for easy understanding, enabling you to become proficient in Data Engineering. Whether you are a beginner or looking to sharpen your existing skills, this Data Engineering is the ideal choice. With our Data Engineering exclusive bundle, you will get a PDF Certificate, PDF Transcript and Digital Student ID Card (worth £50) Absolutely FREE. Courses are Included in This Data Engineering Bundle: * Course 01: Diploma in Data Analysis Fundamentals * Course 02: Python for Data Analysis * Course 03: Data Analytics with Tableau * Course 04: SQL Masterclass: SQL For Data Analytics * Course 05: Basic Google Data Studio * Course 06: Data Analysis and Forecasting in Excel Why Choose Our Data Engineering Course? * FREE Data Engineering certificate accredited * Get a free student ID card with Data Engineering Training * Get instant access to this Data Engineering course. * Learn Data Engineering from anywhere in the world * The Data Engineering is affordable and simple to understand * The Data Engineering is an entirely online, interactive lesson with voiceover audio * Lifetime access to the Data Engineering course materials * The Data Engineering comes with 24/7 tutor support So enrol now in this Data Engineering Today to advance your career! Start your learning journey straightaway! This Data Engineering's curriculum has been designed by Data Engineering experts with years of Data Engineering experience behind them. The Data Engineering course is extremely dynamic and well-paced to help you understand Data Engineering with ease. You'll discover how to master the Data Engineering skill while exploring relevant and essential topics. Assessment Process Once you have completed all the courses in the Data Engineering bundle, you can assess your skills and knowledge with an optional assignment. Our expert trainers will assess your assignment and give you feedback afterwards. CPD 60 CPD hours / points Accredited by CPD Quality Standards WHO IS THIS COURSE FOR? This Data Engineering bundle is suitable for everyone. REQUIREMENTS You will not need any prior background or expertise. CAREER PATH This Data Engineering bundle will allow you to kickstart or take your career in the related sector to the next stage. CERTIFICATES CPD ACCREDITED DIGITAL CERTIFICATE Digital certificate - Included CPD ACCREDITED HARD COPY CERTIFICATE Hard copy certificate - £29 If you are an international student, you will be required to pay an additional fee of 10 GBP for international delivery, and 4.99 GBP for delivery within the UK, for each certificate

Gain the solid skills and knowledge to kickstart a successful career and learn from the experts with this

QLS Endorsed + CPD QS Accredited - Dual Certification | Instant Access | 24/7 Tutor Support | All-Inclusive Cost

  The comprehensive Diploma in Data Engineering with Google BigQuery & Google Cloud has been designed by industry experts to provide learners with everything they need to enhance their skills and knowledge in their chosen area of study. Enrol on the Diploma in Data Engineering with Google BigQuery & Google Cloud today, and learn from the very best the industry has to offer! This best selling Diploma in Data Engineering with Google BigQuery & Google Cloud has been developed by industry professionals and has already been completed by hundreds of satisfied students. This in-depth Diploma in Data Engineering with Google BigQuery & Google Cloud is suitable for anyone who wants to build their professional skill set and improve their expert knowledge. The Diploma in Data Engineering with Google BigQuery & Google Cloud is CPD-accredited, so you can be confident you're completing a quality training course  will boost your CV and enhance your career potential. The Diploma in Data Engineering with Google BigQuery & Google Cloud is made up of several information-packed modules which break down each topic into bite-sized chunks to ensure you understand and retain everything you learn. After successfully completing the Diploma in Data Engineering with Google BigQuery & Google Cloud, you will be awarded a certificate of completion  as proof of your new skills. If you are looking to pursue a new career and want to build your professional skills to excel in your chosen field, the certificate of completion from the Diploma in Data Engineering with Google BigQuery & Google Cloud will help you stand out from the crowd. You can also validate your certification on our website. We know that you are busy and that time is precious, so we have designed the Diploma in Data Engineering with Google BigQuery & Google Cloud to be completed at your own pace, whether that's part-time or full-time. Get full course access upon registration and access the course materials from anywhere in the world, at any time, from any internet-enabled device.  Our experienced tutors are here to support you through the entire learning process and answer any queries you may have via email.

Register on the Data Engineering with Google BigQuery & Google Cloud today and build the experience, skills and knowledge you need to enhance your professional development and work towards your dream job. Study this course through online learning and take the first steps towards a long-term career. The course consists of a number of easy to digest, in-depth modules, designed to provide you with a detailed, expert level of knowledge. Learn through a mixture of instructional video lessons and online study materials. Receive online tutor support as you study the course, to ensure you are supported every step of the way. Get an e-certificate as proof of your course completion. The Data Engineering with Google BigQuery & Google Cloud is incredibly great value and allows you to study at your own pace. Access the course modules from any internet-enabled device, including computers, tablet, and smartphones. The course is designed to increase your employability and equip you with everything you need to be a success. Enrol on the now and start learning instantly! WHAT YOU GET WITH THE DATA ENGINEERING WITH GOOGLE BIGQUERY & GOOGLE CLOUD * Receive a e-certificate upon successful completion of the course * Get taught by experienced, professional instructors * Study at a time and pace that suits your learning style * Get instant feedback on assessments  * 24/7 help and advice via email or live chat * Get full tutor support on weekdays (Monday to Friday) COURSE DESIGN The course is delivered through our online learning platform, accessible through any internet-connected device. There are no formal deadlines or teaching schedules, meaning you are free to study the course at your own pace. You are taught through a combination of * Video lessons * Online study materials CERTIFICATION Upon successful completion of the course, you will be able to obtain your course completion e-certificate free of cost. Print copy by post is also available at an additional cost of £9.99 and PDF Certificate at £4.99. WHO IS THIS COURSE FOR: The course is ideal for those who already work in this sector or are an aspiring professional. This course is designed to enhance your expertise and boost your CV. Learn key skills and gain a professional qualification to prove your newly-acquired knowledge. REQUIREMENTS: The online training is open to all students and has no formal entry requirements. To study the Data Engineering with Google BigQuery & Google Cloud, all your need is a passion for learning, a good understanding of English, numeracy, and IT skills. You must also be over the age of 16. COURSE CONTENT Unit 01: Introduction Module 01: Welcome to This Course 00:01:00 Module 02: How to Get Maximum Value from This Course 00:06:00 Module 03: Course Structure & Coverage 00:02:00 Module 04: Technology in This Course 00:02:00 Unit 02: Introducing Data Warehouse & BigQuery Module 01: Data Warehouse 00:07:00 Module 02: Start With BigQuery 00:05:00 Module 03: BigQuery Web User Interface 00:04:00 Unit 03: First Exploration Module 01: First Data 00:04:00 Module 02: Basic Exploration 00:16:00 Module 03: Functions 00:14:00 Module 04: Common Data Types 00:07:00 Module 05: Different Query 00:01:00 Module 06: Exploring Common Data Types 00:25:00 Module 07: Converting Data Types 00:07:00 Unit 04: Data Flow Basic Module 01: Data Quality 00:06:00 Module 02: Clean & Transform 00:13:00 Module 03: Store Data 00:06:00 Module 04: Upgrading From Sandbox Account 00:01:00 Module 05: Clean & Transform With Dataprep 00:25:00 Module 06: Scheduled Query 00:04:00 Module 07: Analyze Data 00:05:00 Module 08: Data Visualization 00:06:00 Unit 08: Intermediate Query Module 01: Essential BigQuery 00:15:00 Module 02: Load Data into BigQuery (Part 1) - The Basic 00:36:00 Module 03: Tip: Mock Data 00:01:00 Module 04: Load Data into BigQuery (Part 2) - Handling Errors 00:22:00 Module 05: Load Data into BigQuery (Part 3) - Efficient Load 00:14:00 Module 06: Load Data into BigQuery (Part 4) - From Your Data to BigQuery 00:23:00 Module 07: Load Data into BigQuery (Part 5) - In Microservice Architecture 00:20:00 Module 08: Tip: Message Broker Overview 00:08:00 Module 09: Load Data into BigQuery (Part 6) - Recurring Load 00:12:00 Unit 05: Diving into BigQuery Module 01: BigQuery View 00:06:00 Unit 06: Virtual Data using View Module 01: What We Will Learn 00:01:00 Module 02: Google Sheets & BigQuery 00:10:00 Module 03: Google Data Studio 00:13:00 Unit 07: Data Visualization Module 01: Using Join - Theory 00:04:00 Module 02: Using Join - Hands On 00:16:00 Module 03: Union & Intersect 00:06:00 Module 04: Basic Statistical Functions 00:05:00 FREQUENTLY ASKED QUESTIONS Are there any prerequisites for taking the course? There are no specific prerequisites for this course, nor are there any formal entry requirements. All you need is an internet connection, a good understanding of English and a passion for learning for this course. Can I access the course at any time, or is there a set schedule? You have the flexibility to access the course at any time that suits your schedule. Our courses are self-paced, allowing you to study at your own pace and convenience. How long will I have access to the course? For this course, you will have access to the course materials for 1 year only. This means you can review the content as often as you like within the year, even after you've completed the course. However, if you buy Lifetime Access for the course, you will be able to access the course for a lifetime. Is there a certificate of completion provided after completing the course? Yes, upon successfully completing the course, you will receive a certificate of completion. This certificate can be a valuable addition to your professional portfolio and can be shared on your various social networks. Can I switch courses or get a refund if I'm not satisfied with the course? We want you to have a positive learning experience. If you're not satisfied with the course, you can request a course transfer or refund within 14 days of the initial purchase. How do I track my progress in the course? Our platform provides tracking tools and progress indicators for each course. You can monitor your progress, completed lessons, and assessments through your learner dashboard for the course. What if I have technical issues or difficulties with the course? If you encounter technical issues or content-related difficulties with the course, our support team is available to assist you. You can reach out to them for prompt resolution.

Duration 4 Days 24 CPD hours This course is intended for This class is intended for experienced developers who are responsible for managing big data transformations including: Extracting, loading, transforming, cleaning, and validating data. Designing pipelines and architectures for data processing. Creating and maintaining machine learning and statistical models. Querying datasets, visualizing query results and creating reports Overview Design and build data processing systems on Google Cloud Platform. Leverage unstructured data using Spark and ML APIs on Cloud Dataproc. Process batch and streaming data by implementing autoscaling data pipelines on Cloud Dataflow. Derive business insights from extremely large datasets using Google BigQuery. Train, evaluate and predict using machine learning models using TensorFlow and Cloud ML. Enable instant insights from streaming data Get hands-on experience with designing and building data processing systems on Google Cloud. This course uses lectures, demos, and hand-on labs to show you how to design data processing systems, build end-to-end data pipelines, analyze data, and implement machine learning. This course covers structured, unstructured, and streaming data. INTRODUCTION TO DATA ENGINEERING * Explore the role of a data engineer. * Analyze data engineering challenges. * Intro to BigQuery. * Data Lakes and Data Warehouses. * Demo: Federated Queries with BigQuery. * Transactional Databases vs Data Warehouses. * Website Demo: Finding PII in your dataset with DLP API. * Partner effectively with other data teams. * Manage data access and governance. * Build production-ready pipelines. * Review GCP customer case study. * Lab: Analyzing Data with BigQuery. BUILDING A DATA LAKE * Introduction to Data Lakes. * Data Storage and ETL options on GCP. * Building a Data Lake using Cloud Storage. * Optional Demo: Optimizing cost with Google Cloud Storage classes and Cloud Functions. * Securing Cloud Storage. * Storing All Sorts of Data Types. * Video Demo: Running federated queries on Parquet and ORC files in BigQuery. * Cloud SQL as a relational Data Lake. * Lab: Loading Taxi Data into Cloud SQL. BUILDING A DATA WAREHOUSE * The modern data warehouse. * Intro to BigQuery. * Demo: Query TB+ of data in seconds. * Getting Started. * Loading Data. * Video Demo: Querying Cloud SQL from BigQuery. * Lab: Loading Data into BigQuery. * Exploring Schemas. * Demo: Exploring BigQuery Public Datasets with SQL using INFORMATION_SCHEMA. * Schema Design. * Nested and Repeated Fields. * Demo: Nested and repeated fields in BigQuery. * Lab: Working with JSON and Array data in BigQuery. * Optimizing with Partitioning and Clustering. * Demo: Partitioned and Clustered Tables in BigQuery. * Preview: Transforming Batch and Streaming Data. INTRODUCTION TO BUILDING BATCH DATA PIPELINES * EL, ELT, ETL. * Quality considerations. * How to carry out operations in BigQuery. * Demo: ELT to improve data quality in BigQuery. * Shortcomings. * ETL to solve data quality issues. EXECUTING SPARK ON CLOUD DATAPROC * The Hadoop ecosystem. * Running Hadoop on Cloud Dataproc. * GCS instead of HDFS. * Optimizing Dataproc. * Lab: Running Apache Spark jobs on Cloud Dataproc. SERVERLESS DATA PROCESSING WITH CLOUD DATAFLOW * Cloud Dataflow. * Why customers value Dataflow. * Dataflow Pipelines. * Lab: A Simple Dataflow Pipeline (Python/Java). * Lab: MapReduce in Dataflow (Python/Java). * Lab: Side Inputs (Python/Java). * Dataflow Templates. * Dataflow SQL. MANAGE DATA PIPELINES WITH CLOUD DATA FUSION AND CLOUD COMPOSER * Building Batch Data Pipelines visually with Cloud Data Fusion. * Components. * UI Overview. * Building a Pipeline. * Exploring Data using Wrangler. * Lab: Building and executing a pipeline graph in Cloud Data Fusion. * Orchestrating work between GCP services with Cloud Composer. * Apache Airflow Environment. * DAGs and Operators. * Workflow Scheduling. * Optional Long Demo: Event-triggered Loading of data with Cloud Composer, Cloud Functions, Cloud Storage, and BigQuery. * Monitoring and Logging. * Lab: An Introduction to Cloud Composer. INTRODUCTION TO PROCESSING STREAMING DATA * Processing Streaming Data. SERVERLESS MESSAGING WITH CLOUD PUB/SUB * Cloud Pub/Sub. * Lab: Publish Streaming Data into Pub/Sub. CLOUD DATAFLOW STREAMING FEATURES * Cloud Dataflow Streaming Features. * Lab: Streaming Data Pipelines. HIGH-THROUGHPUT BIGQUERY AND BIGTABLE STREAMING FEATURES * BigQuery Streaming Features. * Lab: Streaming Analytics and Dashboards. * Cloud Bigtable. * Lab: Streaming Data Pipelines into Bigtable. ADVANCED BIGQUERY FUNCTIONALITY AND PERFORMANCE * Analytic Window Functions. * Using With Clauses. * GIS Functions. * Demo: Mapping Fastest Growing Zip Codes with BigQuery GeoViz. * Performance Considerations. * Lab: Optimizing your BigQuery Queries for Performance. * Optional Lab: Creating Date-Partitioned Tables in BigQuery. INTRODUCTION TO ANALYTICS AND AI * What is AI?. * From Ad-hoc Data Analysis to Data Driven Decisions. * Options for ML models on GCP. PREBUILT ML MODEL APIS FOR UNSTRUCTURED DATA * Unstructured Data is Hard. * ML APIs for Enriching Data. * Lab: Using the Natural Language API to Classify Unstructured Text. BIG DATA ANALYTICS WITH CLOUD AI PLATFORM NOTEBOOKS * What's a Notebook. * BigQuery Magic and Ties to Pandas. * Lab: BigQuery in Jupyter Labs on AI Platform. PRODUCTION ML PIPELINES WITH KUBEFLOW * Ways to do ML on GCP. * Kubeflow. * AI Hub. * Lab: Running AI models on Kubeflow. CUSTOM MODEL BUILDING WITH SQL IN BIGQUERY ML * BigQuery ML for Quick Model Building. * Demo: Train a model with BigQuery ML to predict NYC taxi fares. * Supported Models. * Lab Option 1: Predict Bike Trip Duration with a Regression Model in BQML. * Lab Option 2: Movie Recommendations in BigQuery ML. CUSTOM MODEL BUILDING WITH CLOUD AUTOML * Why Auto ML? * Auto ML Vision. * Auto ML NLP. * Auto ML Tables.

This course primarily focuses on explaining the concepts of Python and PySpark. It will help you enhance your data analysis skills using structured Spark DataFrames APIs.