[Hindi] End to End Project | A pipeline with Medallion Architecture

About Course

Pre Requisites :

Python
SQL
PySpark
PySpark Optimizations
Big Data Foundations
Hadoop

Course Content

M1 – Project Architcture and Flow

Project Session 1 – Project Flow and Project Architecture Overview

01:06:00
Project Session 2 – Ingestion Tool Choices

32:31
Project Session 3 – ETL Tool Choices

18:04

M2 – Data Ingestion Using Glue

M3 – Data Ingestion Using Nifi

M4 – Spark App Developmet Phase

Kafka Essentials

Airflow Essentials

MongoDB Essentials

AWS for Data Engineering

Big Data With AWS || Demo Session

36:43
AWS IAM User, Group and Policies

01:26:35
Big Data With AWS | IAM Roles

01:23:47
AWS Cloud Infrastructure

31:39
S3 Session 1

01:40:25
S3 Session 2

01:00:49
S3 Session 3

01:09:45
AWS glue | Crawlers And Jobs

01:23:58
BDA | Glue Scenarios | Glue Workflows

01:24:01
Glue Scenarios

01:15:03
Glue Project : CDC

01:46:21
EMR basics EC2 introduction

23:36
EMR_basics_IAM_Role

24:26
Starting EMR cluster | Connecting to EMR Cluster

58:25
AWS EMR | Starting and Deploying Spark Application on EMR

58:04
Cluster Mode Deployment | Accessing Web UIs | Steps Introduction

53:48
Deploying Spark Application in Cluster Mode on EMR

11:15
Deploying Spark Application using Steps on AWS EMR

01:02:03
Athena Basics

01:23:48
Athena on Command Line

59:43
Using Athena through python code

41:27
Redshift And Data Warehousing Introduction

41:34
Redshift Clusters | Snapshots | S3 Copy

59:16
Creating redshift cluster and making it publicly accessible

13:58
Redshift connect using sql workbench and python script

38:55
dist keys and sort keys in redshift

24:30
DIST Keys hands on

30:38
Redshift Federated Queries

40:58
What is Streaming Data | Streaming Data Terminologies

23:24
Kinesis Data Streams | Kinesis Architecture and terminologies

40:05
Streaming Data using Console Producer | Python Producer and Python Consumer

39:48

Code Walkthrough

Development Phase

Deployment Phase

Project Installations and setups

Student Ratings & Reviews

5.0

Total 1 Rating

1 Rating

0 Rating

Shubham

3 weeks ago

Hello Sandip sir, I just wanted to say how genuinely impressed I am with your teaching. One random day I was searching for an Airflow tutorial on YouTube, and honestly, there are barely any good ones that explain things the way they’re actually used in the real world. Then I found your video and from that moment I became a fan of your teaching style.

I went through all of your YouTube tutorials, and I even purchased your end-to-end project pipeline with the medallion architecture. The way you explain concepts is just amazing. It feels real, practical, and so easy to connect with. Truly appreciate your work.

About Course

What Will You Learn?

Course Content

M1 – Project Architcture and Flow

Project Session 1 – Project Flow and Project Architecture Overview

Project Session 2 – Ingestion Tool Choices

Project Session 3 – ETL Tool Choices

M2 – Data Ingestion Using Glue

Project Session 4 – Incremental Ingestion | CDC | Deltas

Project Session 5 – Ingestion Code with Incremental Logic

Project Session 6 – Glue Job Demo

Project Session 7 – Nifi Basic Ingestion

M3 – Data Ingestion Using Nifi

Project Session 8 – Nifi Concepts

Project Session 9 – FTP to S3 Using Nifi (Basic)

Project Session 10 – FTP to S3 Ingestion (Industry Level)

M4 – Spark App Developmet Phase

Project Session 11 – Dev Phase Warm Up

Project Session 12 – Dev Phase Setup

Project Session 13 – Spark Application Development Key Strategies

Project Session 14 – Importance and Foundations of Logging

Project Session 15 – Step Up Logging

Project Session 16 – Integrating Logging and Validations

Project Session 17 – Testing Session 1

Project Session 18 – Testing Session 2

Kafka Essentials

Streaming Basics

Kafka session 1

Kafka session 2

kafka session 3

Kafka session 4

Kafka session 5

Airflow Essentials

Self Learning : Airflow Foundations

MongoDB Essentials

Introduction to NoSQL | Use Cases | Types

MongoDB Essential Elements | CRUD Operations

Indexing in MongoDB

Data Modeling in MongoDB

Sharding in MongoDB

AWS for Data Engineering

Big Data With AWS || Demo Session

AWS IAM User, Group and Policies

Big Data With AWS | IAM Roles

AWS Cloud Infrastructure

S3 Session 1

S3 Session 2

S3 Session 3

AWS glue | Crawlers And Jobs

BDA | Glue Scenarios | Glue Workflows

Glue Scenarios

Glue Project : CDC

EMR basics EC2 introduction

EMR_basics_IAM_Role

Starting EMR cluster | Connecting to EMR Cluster

AWS EMR | Starting and Deploying Spark Application on EMR

Cluster Mode Deployment | Accessing Web UIs | Steps Introduction

Deploying Spark Application in Cluster Mode on EMR

Deploying Spark Application using Steps on AWS EMR

Athena Basics

Athena on Command Line

Using Athena through python code

Redshift And Data Warehousing Introduction

Redshift Clusters | Snapshots | S3 Copy

Creating redshift cluster and making it publicly accessible

Redshift connect using sql workbench and python script

dist keys and sort keys in redshift

DIST Keys hands on

Redshift Federated Queries

What is Streaming Data | Streaming Data Terminologies

Kinesis Data Streams | Kinesis Architecture and terminologies

Streaming Data using Console Producer | Python Producer and Python Consumer

Code Walkthrough

End to End Spark App Overview

Understanding Code Flow | Bronze Layer Flow

Understanding Code Flow | Silver Layer Flow

Understanding Code Flow | Gold Layer Flow

Development Phase

Modularizing your code

logging, Validations, Environments, Scripts