Creating Data Quality

Creating Data Quality

This section provides users with comprehensive guidance and support on navigating and utilizing the Infoveave Data Quality creation features effectively.

Creating Data Quality Manually

1. The process begins by clicking the New Data quality button.
2. This action opens a dialog box where you can select a specific data quality type (e.g., Data Quality using AI or Data Quality) by using the toggle button before proceeding to the next step.
3. After selecting the data quality type select the connection and table to be used for the data quality checks and click on Next.
4. On the Setup tab enter the name and description of the DQ.
5. Drag a column from the columns tab to the designer.
6. Click on Add rule button to add the rules and configure them.
7. On the rule setup screen, you can select the rules for data quality. These rules include checks such as accuracy, completeness, validity, etc.
8. After adding the rules, click on Save. Once saved, configure each rule and click on Validate button.
9. After all the rules are configured and validated click on Validate rules icon to view the validation results.
10. Once the validation is complete, the results are displayed in a table that shows the success percentage and other details about the rule configurations. If needed, you can tweak the rules and validate again.
11. After validating the rules, click on Save to save the configuration.
12. The final step is to Execute Data Quality, which runs the configured data quality checks.
13. The execution of the data quality check displays the results in a table, showing various details such as success rates for different rules. The user can monitor the progress and ensure that the data quality checks are successfully completed.
14. You can also schedule your Data Quality execution under Schedules tab. After adding the schedule click on Save.

Creating Data Quality Using AI

1. The process begins by clicking the New Data Quality button.
2. In the resultant dialog box, enable the toggle AI enabled.
3. After selecting the data quality type select the connection and table to be used for the data quality checks and click on Next.
4. An AI generated description is auto-filled in the resulting pop-up. You can also choose to share Catalogue information with AI by checking the displayed check-box. Click on Generate Data Quality Rules button.
5. AI generates the data quality rules for the selected dataset, ensuring that the data meets specified standards and is ready for further analysis.
   • The relevant Column name for each rule is specified, ensuring that the validation is applied accurately.
   • Each rule is accompanied by a Citation and Rule Group, explaining its purpose, such as ensuring uniqueness, non-null values, or the correct format for columns like IDs or dates.
   • Each rule is assigned a unique Rule name and a Rule type. Once reviewed, the rules can be applied to the data by clicking on Generate Data Quality, which enforces the specified checks across the dataset.
     
6. DQ name and description is auto generated by AI.
7. Click on Save button. After saving you can Validate each rule.
8. After each rule is validated click on Validate rules icon to view the validation results.
9. Once the validations are completed, the results are displayed in a green table that shows the success percentage and other details about the rule configurations. If needed, you can tweak the rules and validate again.
10. After validating the rules, you can click Save to save the configuration.
11. The final step is to Execute Data Quality, which runs the configured data quality checks.
12. The execution of the data quality check displays the results in a table, showing various details such as success rates for different rules. The user can monitor the progress and ensure that the data quality checks are successfully completed.
13. You can also schedule your Data Quality execution under Schedules tab. After adding the schedule click on Save.