5.00
(2 Ratings)
[Hindi] End to End Project | A pipeline with Medallion Architecture
About Course
Pre Requisites :
- Python
- SQL
- PySpark
- PySpark Optimizations
- Big Data Foundations
- Hadoop
What Will You Learn?
- A Complete End to End Project | Scenario Based
Course Content
M7 – Airflow Essentials
-
Self Learning : Airflow Foundations
-
Project Session 27 – Airflow session 1.mp4
01:48:52 -
Project Session 28 – Airflow session 2.mp4
39:00 -
Project Session 29 – Airflow session 3.mp4
37:24
Kafka Essentials
-
Streaming Basics
23:23 -
Kafka session 1
48:04 -
Kafka session 2
39:55 -
kafka session 3
15:00 -
Kafka session 4
19:09 -
Kafka session 5
01:04:18
MongoDB Essentials
-
Introduction to NoSQL | Use Cases | Types
59:00 -
MongoDB Essential Elements | CRUD Operations
37:15 -
Indexing in MongoDB
01:27:02 -
Data Modeling in MongoDB
01:20:59 -
Sharding in MongoDB
33:53
AWS for Data Engineering
-
Big Data With AWS || Demo Session
36:43 -
AWS IAM User, Group and Policies
01:26:35 -
Big Data With AWS | IAM Roles
01:23:47 -
AWS Cloud Infrastructure
31:39 -
S3 Session 1
01:40:25 -
S3 Session 2
01:00:49 -
S3 Session 3
01:09:45 -
AWS glue | Crawlers And Jobs
01:23:58 -
BDA | Glue Scenarios | Glue Workflows
01:24:01 -
Glue Scenarios
01:15:03 -
Glue Project : CDC
01:46:21 -
EMR basics EC2 introduction
23:36 -
EMR_basics_IAM_Role
24:26 -
Starting EMR cluster | Connecting to EMR Cluster
58:25 -
AWS EMR | Starting and Deploying Spark Application on EMR
58:04 -
Cluster Mode Deployment | Accessing Web UIs | Steps Introduction
53:48 -
Deploying Spark Application in Cluster Mode on EMR
11:15 -
Deploying Spark Application using Steps on AWS EMR
01:02:03 -
Athena Basics
01:23:48 -
Athena on Command Line
59:43 -
Using Athena through python code
41:27 -
Redshift And Data Warehousing Introduction
41:34 -
Redshift Clusters | Snapshots | S3 Copy
59:16 -
Creating redshift cluster and making it publicly accessible
13:58 -
Redshift connect using sql workbench and python script
38:55 -
dist keys and sort keys in redshift
24:30 -
DIST Keys hands on
30:38 -
Redshift Federated Queries
40:58 -
What is Streaming Data | Streaming Data Terminologies
23:24 -
Kinesis Data Streams | Kinesis Architecture and terminologies
40:05 -
Streaming Data using Console Producer | Python Producer and Python Consumer
39:48
Code Walkthrough
-
End to End Spark App Overview
27:05 -
Understanding Code Flow | Bronze Layer Flow
01:05:42 -
Understanding Code Flow | Silver Layer Flow
01:00:22 -
Understanding Code Flow | Gold Layer Flow
21:15
Development Phase
-
Modularizing your code
01:41:49 -
logging, Validations, Environments, Scripts
01:17:18 -
Validations
49:13
Deployment Phase
-
Deployment Stage 1
01:48:28 -
Deploymemnt Stage 1 Internals
01:22:32 -
Deployment Stage Final
02:09:41
Project Installations and setups
-
Project | Kafka Installation
18:04 -
RDS MySQL Setup
37:01 -
Nifi Installation
00:00
Project Management – Operations
-
session 1 – project management session
01:33:10 -
session 2 – project management session
01:36:17 -
session 3 – project management session
02:04:17 -
session 4 – project management session
01:55:18 -
session 5 – project management session
02:05:02 -
session 6 – project management session
45:29 -
session 7 – project management session
01:42:29 -
session 8 – project management session
01:37:34
Deployment With AWS
-
Project Session 1 – Project Flow and Project Architecture Overview
01:06:00 -
Project Session 2 – Ingestion Tool Choices
32:31 -
Project Session 3 – ETL Tool Choices
18:04 -
Project Session 4 – Incremental Ingestion | CDC | Deltas
44:55 -
Project Session 5 – Ingestion Code with Incremental Logic
44:01 -
Project Session 6 – Glue Job Demo
46:30 -
Project Session 7 – Nifi Basic Ingestion
34:50 -
Project Session 8 – Nifi Concepts
54:34 -
Project Session 9 – FTP to S3 Using Nifi (Basic)
01:28:02 -
Project Session 10 – FTP to S3 Ingestion (Industry Level)
49:38 -
Project Session 11 – Dev Phase Warm Up
30:33 -
Project Session 12 – Dev Phase Setup
17:37 -
Project Session 13 – Spark Application Development Key Strategies
44:11 -
Project Session 14 – Importance and Foundations of Logging
30:50 -
Project Session 15 – Step Up Logging
54:43 -
Project Session 16 – Integrating Logging and Validations
32:19 -
Project Session 17 – Testing Session 1
01:03:41 -
Project Session 18 – Testing Session 2
39:56 -
Project Session 19 : Mocking
49:00 -
Project Session 20 – Write Modes.mp4
32:58 -
Project Session 21 – bronze to silver upsert.mp4
38:15 -
Project Session 22 – Gold Logic.mp4
31:04 -
Project Session 23 – Gold layer hands on.mp4
36:49 -
Project Session 24 – Glue scenarios.mp4
42:01 -
Project Session 25 – Glue to s3 scenarios.mp4
01:00:54 -
Project Session 26 – Nifi to s3 scenarios.mp4
54:18 -
Project Session 27 – Airflow session 1.mp4
01:48:52 -
Project Session 28 – Airflow session 2.mp4
39:00 -
Project Session 29 – Airflow session 3.mp4
37:24
Deployment With Databrics
-
project session 1 purpose architecture and tech stack.mp4
01:04:07 -
project session 2 rdbms ingestion cdc setup.mp4
59:23 -
project session 3 industry grade cdc pipeline for rdbms.mp4
01:27:37 -
project session 4 rdbms to delta scenarios – 1.mp4
01:02:56 -
project session 5 rdbms to delts scenarios – 2.mp4
47:46 -
project session 6 streaming data processing template and basic terminolohies.mp4
01:04:08 -
project session 7 structured streaming fundamentals for auto loader.mp4
01:08:52 -
project session 8 auto loader demo 1.mp4
01:01:13 -
project session 9 general data qulity checks.mp4
57:10
Student Ratings & Reviews
5.0
Total 2 Ratings
5
2 Ratings
4
0 Rating
3
0 Rating
2
0 Rating
1
0 Rating
best
Hello Sandip sir, I just wanted to say how genuinely impressed I am with your teaching. One random day I was searching for an Airflow tutorial on YouTube, and honestly, there are barely any good ones that explain things the way they’re actually used in the real world. Then I found your video and from that moment I became a fan of your teaching style.
I went through all of your YouTube tutorials, and I even purchased your end-to-end project pipeline with the medallion architecture. The way you explain concepts is just amazing. It feels real, practical, and so easy to connect with. Truly appreciate your work.
I went through all of your YouTube tutorials, and I even purchased your end-to-end project pipeline with the medallion architecture. The way you explain concepts is just amazing. It feels real, practical, and so easy to connect with. Truly appreciate your work.
