Description
The Execute Data Quality activity runs a selected Data Quality Profile, which includes a set of data quality validation rules applied to a dataset. These rules help ensure that the dataset complies with defined standards, such as completeness, consistency, and accuracy.
Each rule targets specific columns and performs checks like null constraints, pattern matching, range validation, etc. The result is a structured output that summarizes rule-level performance and data health.
Use Case:
Use this activity to validate raw or processed data before it enters downstream workflows like reporting, transformation, or export, ensuring early error detection.
Not Applicable
Output
| Type | Description |
|---|
| Data | List of rule-level execution results for each column |
Configuration Fields
| Field | Required | Description |
|---|
| Data Quality | Yes | The predefined Data Quality profile to execute. Selected via DataQualityId. |
Sample Configuration
| Field | Value |
|---|
Data Quality | Customer Data Profile |
Sample Output
| Column Name | Rule Group | Rule Name | Rule Type | Success(%) | Total Rows | Rule Description |
|---|
| age | Range | Age Between 18–60 | Validation | 97.5 | 1000 | Validates if age falls between 18 and 60 |
| email | Pattern | Valid Email Format | Validation | 99.2 | 1000 | Checks if email matches standard pattern |
| id | Uniqueness | Unique Identifier | Uniqueness | 100.0 | 1000 | Ensures values are unique per row |
Supported Rule Types
| Rule Name | Description |
|---|
| Value Range Validation | Ensures numbers fall within a specified minimum and maximum range. |
| Formula Integrity Check | Validates formulas and computed field integrity within the dataset. |
| Not Null Check | Verifies that required fields are not left empty. |
| Consistent Casing | Ensures text casing (e.g., UPPER, lower, Title Case) is consistent. |
| Custom SQL Validation | Runs user-defined SQL queries for specialized checks. |
| Recent Data Validation | Checks if date values fall within a recent or defined date period. |
| Distinct Record Check | Ensures records are unique across all specified columns. |
| Column Uniqueness | Verifies that a single column’s values are unique. |
| Pattern Matching | Validates values against a regular expression pattern. |
| String Length Validation | Ensures text values meet length constraints (e.g., min/max length). |
| Reference Data Validation | Validates values against a reference dataset or lookup list. |
| Predefined Value Check | Confirms that values exist within a defined static list. |
| String Value Matching | Checks for exact matches with expected string values. |
| SQL Pattern Matching | Validates values using SQL LIKE patterns (e.g., %abc%, abc_). |
| Monotonic Sequence Check | Ensures values are increasing or decreasing consistently. |
Notes
- This activity does not require input data directly but operates based on a previously defined data quality configuration.
- If no rules are defined in the selected profile, the execution will still complete successfully but return no results.
- Supports both exact-match and pattern-based validations, including date formats, SQL, and regex.
- All validations are performed in bulk and summarized in the output table.
