Build Big Data Pipelines w/ Hadoop, Flume, Pig, MongoDB
Learn how to combine Hadoop, MongoDB, Pig, Sqoop and Flume to Architect and Build Big Data Pipelines and Data lakes.
Learn how to stitch individual Big Data Technologies together to solve business problems
Build and deploy Big Data Pipelines
How do you build end-to-end Big Data Pipelines using multiple Big Data Technologies? You have seen courses and books that teach you individual technologies, but how do you combine and apply them to solve your business problems? This course teach you exact that !
Building Big Data Solutions require you to acquire data from multiple sources, transport them, process them and store them in Big Data repositories. You have to do that with scalability and reliability. Big Data Technologies like Hadoop, Sqoop, Pig, Flume etc. solve individual problems, but building an end-to-end solution requires stitching them together. This course teaches you how to do that. You solve complete business problems by building end-to-end pipelines in this course.
Introduction
1
About the Mentor
2
Your Learning Process - Learn, Train, Practice, Apply
3
Goals for this Course
4
Setting up CDH Quickstart VM
5
Resource Bundle - Code Examples and Practice Exercises
Building Big Data Pipelines
1
Unique Challenges with Big Data Pipelines
2
Platform vs Application for Big Data
3
Big Data Pipelines - Recommended Strategy
4
Introduction to Use Cases
Apache Sqoop - an Introduction
1
What is Apache Sqoop?
2
Sqoop Command Line Overview
3
Simple Import Command
4
Setting up a job and Executing
5
@Practice() : Sqoop Practice Exercise
Apache Flume : An introduction
1
What is Apache Flume?
2
Flume Agents
3
Installing and Running Flume
4
Chaining Flume Agents
5
Use Case 1 : Netcat to Console
6
Use Case 2 : Spool Directory to Destination Directory
7
Use Case 3 : Chaining with Directory, Avro and HDFS
8
@Practice() : Flume Practice Exercise
Apache Pig - An introduction
1
What is Apache Pig?
2
Pig Latin Basics
3
Pig Operations
4
Data Engineering with Pig
5
Pig Data Flows
6
Data Engineering with Pig
7
@Practice() : Pig Practice Exercises
Mongo DB - An introduction
1
What is Mongo DB?
2
Mongo DB Organization
3
Installation
4
Inserting Data
5
Querying Data
6
Updating and Deleting Data
7
@Practice() : Mongo DB Practice Exercise
Pipeline Use Case 1 : RDBMS to Big Data
1
Use Case 1 - Problem and Solution Overview
2
Use Case Solution - Building the Pipeline
Pipeline Use Case 2 : Web Logs to Mongo DB
1
Use Case 2 - Problem and Solution Overview
2
Use Case Solution - Building the Pipeline
Pipeline Use Case 3 : Multiple Sources to Monthly Summary
1
Use Case 3 - Problem and Solution Overview
2
Use Case Solution - Building the Pipeline
APPLY Project : Building multi-source pipelines
1
Apply Project - Problem Statement
2
Apply Project : Solution Review
Conclusion
1
Closing Remarks
2
BONUS Lecture : Other courses you should check out
You can view and review the lecture materials indefinitely, like an on-demand channel.
Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don't have an internet connection, some instructors also let their students download course lectures. That's up to the instructor though, so make sure you get on their good side!
30-Day Money-Back Guarantee
Includes
3 hours on-demand video
8 articles
Full lifetime access
Access on mobile and TV
Certificate of Completion