Suggest an Expectation or an Expectation Package!

[kyleaton]

Hello!

This category is a place for people to begin to suggest Expectations and Expectation Packages for Great Expectations! Please feel free to create a post based on Expectations or Package ideas!

Some Relevant links:

• Expectation Gallery to see if your Expectation already exists.
• How to create Custom Expectations
• Contributing a Package
• How to contribute a Custom Expectation to Great Expectations

[anikolova-vmware]

Hello Team,

Can you implement a data duplication check on an entire table or a table subset?

• records in pre-selected columns are unique (no duplicated rows)

[jedisom]

Couldn't this be accomplished with multiple expect_column_proportion_of_unique_values_to_be_between expectations?

[ActivateConsulting]

Foreign key expecation in sql server

[heather-dice]

1. Column row count to be between
2. Column null count to be between
3. Column not null count to be between

[METTUNR29]

Identify if the columns has SSN pattern or/ phone number pattern or/ EIN pattern or/ Shipping tracking number. each is a spserate need though.

[jedisom]

Couldn't this be completed with a expect_column_values_to_match_regex expectation that already exists?

[jedisom]

It would be great to have a rolling year, or date, window. For example valid values are between this year plus X and this year minus Y. Same thing could be done for months, days, etc.

[ashankergit]

This would be great to have. I would like to check if the sources have latest data for yesterday, and this would be perfect

[kujaska]

I need this, too:
Anomaly Detection on Great Expectations
#7161

[rogalski]

expect_multicolumn_values_to_be_in_set.
Looks like simple generalization over expect_column_pair_values_to_be_in_set, I was quite surprised it's not available according to docs.

[pcontrerasnun]

An expectation for categorical columns to check proportion of each distinct value. For example, given column A that has distinct values 1, 2, 3. An expectation to check column has X% aprox of value 1, Y% aprox of value 2 and Z% aprox of value 3

[Athangar-tec]

It would be good to have an expectation that compares two table. This could be for both delta and full mode.

[PalBence]

It would be great if there was a expect_column_pair_values_to_not_be_equal function. It would be basically the 'not' version of the expect_column_pair_values_to_be_equal function. Or is there already a way to achieve this outcome? Thanks for the answers.

[webcoderz]

Risk of data disclosure

[gerileka]

I would be great to have something like:

• Check the standard deviation of a column for time series tables. If the last received value is X standard deviation from a given window raise alarm
• I join the foreign key from @ActivateConsulting but also in the API level

[ashankergit]

I would like to have an expectation that checks if one column is not null and the other is null. We currently have if both are null or not null

[mollysrour]

Expectation that checks for regex mappings but for a list column instead of a string column

[BPilippG]

For me and my Team it would be great to have some sort of expect_column_values_exist_between that allows testing if a column has at least one value within a specified interval.
The existing expect_column_distinct_values_to_contain_set is similar but does not always do the job because

• it requires specifying a discrete set instead of an interval, and
• it uses the computationally expensive metric column.value_counts, i.e. reads the complete batch.

A possible use case is to regularly test large tables in an automated pipeline whether new records are present.

Thanks & Greetings

[adaboina21]

Hi ,

Can we have expectation where we compare values between two columns ?

Example :
Doc No","Account","Dist Type","Debit","Credit","Salesperson ID","Comm Amount"
"BF03248518","12101-900-000","Accts Receivable",3886.82,0,"",""

here we can compare Credit and Debit Amount values

Thanks,

[ria3110]

Are there any expectations available to match source data (csv file) and destination data (data in table)

[cwdjankoski]

Is there a way to check for anomalies in counts of events per day - but grouped by another categorical columns i.e.

+------------+------------+-------------+
|    date    | event_name | event_count |
+------------+------------+-------------+
| 2024-06-21 | event_A    |         100 |
| 2024-06-21 | event_B    |        1000 |
| 2024-06-20 | event_A    |         110 |
| 2024-06-20 | event_B    |         990 |
+------------+------------+-------------+

Is there a way to check and fail expectation if e.g. only the count for (event_A) is outlier on certain date ?
I tried looking through the new/legacy gallery and contributed expectations but could not find anything.
Thank you in advance for any comments.