Fetch Data Quality errors
Description
The Fetch Data Quality Errors activity retrieves records from a dataset that failed specific validation rules during a previously executed Data Quality run in Infoveave. This activity plays a crucial role in data cleansing, monitoring, auditing, and reporting workflows by enabling users to focus only on records that did not meet predefined data quality standards.
You begin by selecting an existing Data Quality configuration (Data Quality) that was already executed. Then, you specify one or more Data Quality Rules (Data Quality Rule) from that configuration. The activity will filter and return only those rows from the validated dataset that failed the selected rules.
This output allows data engineers, analysts, or business users to identify bad data that violates conditions such as missing values (Not Null Check), incorrect formats (Pattern Matching, Email Validation), duplicates (Column Uniqueness), and more. The returned data includes not only the invalid rows but also the specific rule each row failed, which helps in pinpointing the exact issue.
This activity is especially useful in data pipelines where maintaining data trustworthiness is vital before passing data downstream. Failed records can be:
- Reviewed by business users to understand the nature of the errors.
- Exported or reported to relevant stakeholders or teams responsible for data input.
- Corrected manually or sent into a remediation workflow.
- Used for rule refinement, helping to enhance rule definitions or thresholds based on recurring error trends.
Example Use Case: If a Sales dataset contains records with missing
Phone Numberformats, executing this activity with those rule filters will extract only the rows with issues. This data can be reviewed for correction before proceeding with customer communications or analytics.
Ultimately, this activity acts as a diagnostic tool for highlighting low-quality data and enables organizations to improve data hygiene continuously. It ensures data quality compliance by offering transparency into exactly which records are problematic and why — leading to better data governance, operational efficiency, and trust in analytics.
Input
Not Applicable
Output
| Output Type | Structure |
|---|---|
| Data | List of rows with failed validations |
Configuration Fields
| Field Name | Required | Description |
|---|---|---|
| Data Quality | Yes | Select Data quality from the list |
| Data Quality Rule | Yes | Select the Rule From the List |
Sample Input
Not Applicable
Sample Configuration
| Field | Value |
|---|---|
| Data quality | Sales |
| Data quality rule | NullCheck |
Sample Output
| CustomerId | Phone | Rule | |
|---|---|---|---|
| 123 | (empty) | 12345 | |
| 124 | [email protected] | Null Check |