Error % not calculated correctly for ColumnBased checks

[samratmitra-0812]

Example

CheckType: minNumRows
minValue: 1800
actualValue: 1144

In the above example error % should be (1800 - 1144) * 100/1800 = 36.44%
However, it is calculated as (1 * 100)/1144 = 0.09%

Similar issue exists for ColumnMaxCheck.

[phpisciuneri]

Yes, I agree that this is a bug.

Basically, the error % as it is implemented now, only makes sense in terms of row based checks. It is counting the number of rows that contain an exception for a check and dividing by the total number of rows.

Row based checks support the optional threshold parameter so that a user can configure some % of errors that they are willing to tolerate. For instance, it may be OK that 1% of the values in a column are null.

However, for column aggregate checks (like minNumRows) we are essentially comparing one value (the result of an aggregation) to a parameter or parameters specified in the check. I see three options for how we can proceed in these cases:

1. Relative error (as you have outlined above).
2. Binary pass/fail. If check passes 0% error, else 100% error.
3. Make no attempt to quantify error for column based checks.

I think I am in favor of the third option. The column based checks are binary in that they are either pass/fail. We already report pass/fail so it seems redundant to specify 0 or 100% for the error. Relative error does have a sensible definition in this context, but the interpretation would be different from that of the row based checks and I think that would be a source of confusion.

I am happy to hear other opinions/options.

[samratmitra-0812]

Hi @phpisciuneri , @colindean

I (and other TMs in the team) am actually in favour of Option 1. In fact, after discussing with the team, I have almost implemented a fix based on that. Post this fix, the message for ColumnBased checks will contain the expected value (specified in yaml), the actual value (as calculated by the validator), and the relative error %. I will forward you a sample email report so that you can take a look.

Also, the person looking at the report will most likely know what the check is supposed to do. And since we are providing enough information in the reports, I am not sure if there will be much confusion.

And Option 1 may also enable us to implement a tolerance threshold feature for ColumnBased checks as well.

Let me know your thoughts.

[phpisciuneri]

@samratmitra-0812 I will take a look at the PR.

Please remember that this is an open-source repository. If other users have opinions they need to come to the issues here and comment accordingly so we can sort these things out in the proper way.

[colindean]

Fixed by #43 a while ago