3.45 out of 5
3.45
27 reviews on Udemy

Data Quality & Profiling with ETL Pentaho DI & DataCleaner

Explore how to improve the data quality by profiling, cleaning and automating the DQM process with ETL & Cleansing Tools
Instructor:
Rajkumar V
214 students enrolled
English [Auto-generated]
Understand the key terms in Data Quality
Understand the basics of Pentaho Data Integration and Datacleaner tool
Understand how to install and use the tools for creating a Data profiling and cleansing jobs
Understand how to Integrate Datacleaner tool with PDI
Understand how to automate the key phases of Data Quality Management framework
Several hands on sessions to understand how to implement data quality jobs, data profiling jobs and how to schedule these jobs

Learn the key terminologies, basic concepts, implementation techniques that you will need to build fully functional data quality implementations with the popular ETL tool – Pentaho Data Integration and Data Quality tool – DataCleaner. Learn by comprehensive hands-on sessions to improve the data quality by profiling, cleansing and automating the DQM process with tools.

The concepts learned in this course can be applied in other ETL or Data Quality assurance tool as well. 

Take care of the data to take care of the business
Learning the fundamentals and understanding the implementation of Data Quality tasks is something very imperative for any business to keep up in this competitive world. Ensuring the data quality will keep your business or projects from losses.

There are plenty of opportunities in data domain, and being able to learn and appreciate the importance of data quality will give you a confidence to tackle the challenges that you encounter while handling data of any volume and format.

Content and Overview

Through this 8 session course, 43 lectures and 103 minutes of content, you will learn the key terms in Data Quality, basics of Pentaho Data Integration and Datacleaner tool, how to install the tools, how to automate the key data quality and data profiling tasks with series of demo sessions.

You can test the knowledge gained through the sessions by attending quizzes and every use case mentioned in the course are explained with demo sessions thereby enabling you to practice the newly learned skills. 

Downloadable Resources

You can also download the files (available in the last session of this course) used for demo sessions in the course to practice at your end.

Learners who completes this course will have the knowledge and confidence to implement fully functional automated data quality solutions in the projects.

What does this Course cover?

1
Introduction to Target Audience and Key Takeaways

A brief introduction about the Target Audience and what one could Benefit from this course.

2
Why to bother about Data Quality?

This lecture focuses on explaining the Importance of Data Quality in any business domain and the Consequences when not ensuring data quality.

3
Course Outline

This lecture focuses on to list what other lectures in the entire course will cover in a brief manner.

Building the Foundation

1
Key Terminologies and Concepts in Data Quality

At the end of this lecture, students will be able to List and Define some of the Key Terms in Data Quality domain.

Tools Introduction And Installation

1
Why to use a tool for DQA Tasks?

This lecture focuses on to 

  1. Explain the Advantages of using a tool for Data Quality tasks and 
  2. Why Pentaho Data Integration and DataCleaner tools are applied in this area
2
Introduction to Pentaho Data Integration or Kettle

As part of this lecture, students will get to know about the Key building blocks of PDI

3
Introduction to DataCleaner

As part of this lecture, students will get to know about the Key building blocks of DataCleaner

4
Installation of PDI

This lecture focuses on how to Install PDI community edition and Verify the installation

5
Installation of DataCleaner in Stand-alone mode

This lecture focuses on how to Install DataCleaner community edition and Verify the installation

6
Installation of DataCleaner as Plug-in with PDI

This lecture focuses on how to Install and Integrate DataCleaner with PDI

Working with DataCleaner

1
Section Outline

This lecture focuses on to List the topics covered as part of this section, that includes various features in DataCleaner tool and how to utilize or implement those features with a series of hands-on demos.

2
Connecting a Datastore with DataCleaner

This demo session on DataCleaner focuses on to 

  • List available options to connect various data sources
  • How to connect the data sources
  • How to List the configured data sources
  • Edit and Remove the saved datastores
3
Designing a Job with DataCleaner

This demo session focuses on How to design a job with DataCleaner tool

4
Previewing or Executing a DataCleaner Job

This demo session focuses on How to Preview or Execute a DataCleaner job.

5
Preserving Cleansed data with DataCleaner

This demo session focuses on How to store the cleansed data with DataCleaner tool for immediate or later use.

6
Command line options

This demo session focuses on various Command line options available with DataCleaner tool.

7
Executing a DataCleaner Job in Command line

This demo session focuses on How to execute the job in Command line.

8
Scheduling a DataCleaner Job

This demo session focuses on 

  1. How to schedule a DataCleaner job and 
  2. The idea presented shall be used for Scheduling the job in various operating systems

9
Parameterizable DataCleaner Jobs

This demo session focuses on 

  1. How to design a parameters enabled  job and
  2. How to execute the parameter enabled job in command line
10
Configuration in DataCleaner

This demo session focuses on to

  1. List the various configuration files
  2. Configure using conf.xml file
11
Logging in DataCleaner

After this lecture, students will get to know 

  1. The Default Log location
  2. The Default Log Config file location 
  3. How to change the logging path to a custom location
12
Logging in PDI

This lecture focuses on 

  1. Various logging options in PDI
  2. How to Configure and Use various logging features in PDI

Integrating DataCleaner with PDI

1
Section Outline
2
Creating a Job with PDI

As part of this demo session, students will learn How to design a job in PDI using Spoon GUI and save it.

Additional resources related to various PDI job entries are added with this lecture.

3
Creating a Transformation with PDI

As part of this demo session, students will learn How to design a Transformation in PDI using Spoon GUI and save it.

Additional resources related to various PDI Transformation steps and instructions to configure database connectivity settings are added with this lecture.

4
Executing a DataCleaner Job in PDI

As part of this demo session, students will learn How to execute a job in PDI using Spoon GUI.

5
Scheduling a PDI Job

As part of this demo session, students will learn 

  1. How to schedule a PDI job using Kitchen Utility and Crontab
  2. The idea explained can be used to Extend the scheduling of PDI job in other Operating systems with various schedulers. 

Walk Through on Demo Use Cases

1
Section Outline
2
Data Quality Dimensions - Use Cases - Part 1
3
Data Quality Dimensions - Use Cases - Part 2
4
Data Quality Dimensions - Use Cases - Part 3
5
Data Profiling and Other Use Cases

Demo

1
Demo on Completeness
2
Demo on Conformity
3
Demo on Referential Integrity
4
Demo on Validity
5
Demo on Accuracy
6
Demo on Duplicate check - Single field
7
Demo on various Data Profiling Tasks
8
Demo on Duplicate check - Multiple fields and Adhoc Profiling
9
Demo on Integrating DataCleaner Job with PDI

This demo session focuses on explaining

  1. How to integrate DataCleaner job with PDI
  2. The use case involved for integrating a DataCleaner job with PDI
  3. How to generalize the job design to store the various data quality issues observed in a database table and later use this stored information as part of data quality solution automation tasks.
10
Tips on Automating Data Quality Solution

As part of this demo session, students will Takeaway tips to automate data quality solution by

  1. Executing a data quality control job using the tools and 
  2. Log the observations in a table
  3. How to use the persisted information for further notification and 
  4. Reports generation process

What Next?

1
Ways to move forward

Thank You Note and Resources in form of 

  1. Reference links
  2. Helpful links to seek support, 
  3. Source code files and other relevant files used in various demos of this course are attached
2
PDI Command Line

Pentaho Data Integration Command Line Utilities

3
DataCleaner Command Line

DataCleaner Command Line Options

4
Parameterizable DataCleaner Jobs
You can view and review the lecture materials indefinitely, like an on-demand channel.
Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don't have an internet connection, some instructors also let their students download course lectures. That's up to the instructor though, so make sure you get on their good side!
3.5
3.5 out of 5
27 Ratings

Detailed Rating

Stars 5
4
Stars 4
9
Stars 3
9
Stars 2
3
Stars 1
3
418de47e15b6a4d6dea73813d88a4742
30-Day Money-Back Guarantee

Includes

2 hours on-demand video
Full lifetime access
Access on mobile and TV
Certificate of Completion