Skip to content

Add RegexFormatAdherence and DateimeFormatAdherence to DataValidity property #902

Description

@sarahmish

Problem Description

Once we have created the RegexFormatAdherence (#897) and DateimeFormatAdherence (#898) metrics, we should update the DataValidity property to include these two metrics in its calculation.

With this change:

  • Datetime columns will be checked twice for Data Validity: once for min/max boundaries and another time for datetime format.
  • ID columns might also be checked twice for Data Validity: once for primary key uniqueness (if it's a primary key) and another time for regex format.

Expected behavior

Update DataValidty property to include regex adherence and datetime adherence.

  • These checks should only happen if the regex_format and datetime_format are supplied in the metadata for those columns. We should update the progress bar to account for these extra checks that will be done.
  • The details for this property should include the new metric.

Score Calculation

This property should add the following metrics to calculate the score:

  • RegexFormatAdherence

    • input sdtypes: ID
  • DateimeFormatAdherence

    • input sdtypes: datetime
>>> report.get_details(property_name='Data Validity', table_name='users')

Table        Column        Metric                   Score
users        user_id       KeyUniqueness            1.0   
users        user_id       RegexFormatAdherence     1.0
users        age           BoundaryAdherence        1.0     
users        dob           BoundaryAdherence        1.0            
users        dob           DatetimeFormatAdherence  1.0
...

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions