Data Quality & Profiling with ETL Pentaho DI & DataCleaner
Learn the key terminologies, basic concepts, implementation techniques that you will need to build fully functional data quality implementations with the popular ETL tool – Pentaho Data Integration and Data Quality tool – DataCleaner. Learn by comprehensive hands-on sessions to improve the data quality by profiling, cleansing and automating the DQM process with tools.
The concepts learned in this course can be applied in other ETL or Data Quality assurance tool as well.
Take care of the data to take care of the business
Learning the fundamentals and understanding the implementation of Data Quality tasks is something very imperative for any business to keep up in this competitive world. Ensuring the data quality will keep your business or projects from losses.
There are plenty of opportunities in data domain, and being able to learn and appreciate the importance of data quality will give you a confidence to tackle the challenges that you encounter while handling data of any volume and format.
Content and Overview
Through this 8 session course, 43 lectures and 103 minutes of content, you will learn the key terms in Data Quality, basics of Pentaho Data Integration and Datacleaner tool, how to install the tools, how to automate the key data quality and data profiling tasks with series of demo sessions.
You can test the knowledge gained through the sessions by attending quizzes and every use case mentioned in the course are explained with demo sessions thereby enabling you to practice the newly learned skills.
You can also download the files (available in the last session of this course) used for demo sessions in the course to practice at your end.
Learners who completes this course will have the knowledge and confidence to implement fully functional automated data quality solutions in the projects.
What does this Course cover?
A brief introduction about the Target Audience and what one could Benefit from this course.
This lecture focuses on explaining the Importance of Data Quality in any business domain and the Consequences when not ensuring data quality.
This lecture focuses on to list what other lectures in the entire course will cover in a brief manner.
Building the Foundation
At the end of this lecture, students will be able to List and Define some of the Key Terms in Data Quality domain.
Tools Introduction And Installation
This lecture focuses on to
- Explain the Advantages of using a tool for Data Quality tasks and
- Why Pentaho Data Integration and DataCleaner tools are applied in this area
As part of this lecture, students will get to know about the Key building blocks of PDI
As part of this lecture, students will get to know about the Key building blocks of DataCleaner
This lecture focuses on how to Install PDI community edition and Verify the installation
This lecture focuses on how to Install DataCleaner community edition and Verify the installation
This lecture focuses on how to Install and Integrate DataCleaner with PDI
Working with DataCleaner
This lecture focuses on to List the topics covered as part of this section, that includes various features in DataCleaner tool and how to utilize or implement those features with a series of hands-on demos.
This demo session on DataCleaner focuses on to
- List available options to connect various data sources
- How to connect the data sources
- How to List the configured data sources
- Edit and Remove the saved datastores
This demo session focuses on How to design a job with DataCleaner tool
This demo session focuses on How to Preview or Execute a DataCleaner job.
This demo session focuses on How to store the cleansed data with DataCleaner tool for immediate or later use.
This demo session focuses on various Command line options available with DataCleaner tool.
This demo session focuses on How to execute the job in Command line.
This demo session focuses on
- How to schedule a DataCleaner job and
- The idea presented shall be used for Scheduling the job in various operating systems
This demo session focuses on
- How to design a parameters enabled job and
- How to execute the parameter enabled job in command line
This demo session focuses on to
- List the various configuration files
- Configure using conf.xml file
After this lecture, students will get to know
- The Default Log location
- The Default Log Config file location
- How to change the logging path to a custom location
This lecture focuses on
- Various logging options in PDI
- How to Configure and Use various logging features in PDI
Integrating DataCleaner with PDI
As part of this demo session, students will learn How to design a job in PDI using Spoon GUI and save it.
Additional resources related to various PDI job entries are added with this lecture.
As part of this demo session, students will learn How to design a Transformation in PDI using Spoon GUI and save it.
Additional resources related to various PDI Transformation steps and instructions to configure database connectivity settings are added with this lecture.
As part of this demo session, students will learn How to execute a job in PDI using Spoon GUI.
As part of this demo session, students will learn
- How to schedule a PDI job using Kitchen Utility and Crontab.
- The idea explained can be used to Extend the scheduling of PDI job in other Operating systems with various schedulers.
Walk Through on Demo Use Cases
This demo session focuses on explaining
- How to integrate DataCleaner job with PDI
- The use case involved for integrating a DataCleaner job with PDI
- How to generalize the job design to store the various data quality issues observed in a database table and later use this stored information as part of data quality solution automation tasks.
As part of this demo session, students will Takeaway tips to automate data quality solution by
- Executing a data quality control job using the tools and
- Log the observations in a table
- How to use the persisted information for further notification and
- Reports generation process
Thank You Note and Resources in form of
- Reference links,
- Helpful links to seek support,
- Source code files and other relevant files used in various demos of this course are attached
Pentaho Data Integration Command Line Utilities
DataCleaner Command Line Options