Repository navigation
add support for Row Level of granularity DQ reports #225
Description
Activity
I would also welcome this functionality to be exposed.
+1 Accessing the scala function from Pyspark fails when assertions are specified using Lambda functions.
Thank you for raising this — you're correct that
rowLevelResultsAsDataFramefrom Deequ's Scala API is not currently exposed in PyDeequ. This is a valid feature gap.The challenge noted by @0xbadidea is also real: our lambda-based assertions use Python callback objects (via
ScalaFunction1), which don't serialize cleanly when Deequ tries to evaluate them at the row level.We'll track this as an enhancement request. A contribution adding a
rowLevelResultsAsDataFrameclassmethod toVerificationResultinpydeequ/verification.pywould be welcome — the implementation would call through to the JVM method similarly to howcheckResultsAsDataFrameworks today. PRs are appreciated if anyone wants to take a crack at it.This has also been flagged for our maintainer team to review.
Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.
rowLevelResultsAsDataFramehas since been exposed in PyDeequ via #262 (merged). It is available asVerificationResult.rowLevelResultsAsDataFrame(spark_session, verificationResult, data)(pydeequ/verification.py), with a usage snippet in the README. That should cover the row-level DQ report use case described here, so I believe this can be closed. (Duplicate of #223 / #224 — all three request the same method.)This is correct —
rowLevelResultsAsDataFrameis now available in PyDeequ. You can use it as follows:from pydeequ.checks import * from pydeequ.verification import * check = Check(spark, CheckLevel.Warning, "Review Check") checkResult = VerificationSuite(spark) \ .onData(df) \ .addCheck( check.isComplete("c") \ .isContainedIn("a", ["foo", "bar", "baz"])) \ .run() # Get row-level results rowLevelResult_df = VerificationResult.rowLevelResultsAsDataFrame(spark, checkResult, df) rowLevelResult_df.show()
This returns the original DataFrame with additional Boolean columns (named after the check description) indicating which rows passed or failed. Note that only checks with row-level-capable constraints (e.g.,
isComplete,isContainedIn,hasPattern,isUnique) will produce output columns — aggregate-only checks likehasSizeare skipped.This issue can be closed as resolved.
Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 67928783) — may not be fully accurate. Reply if this doesn't help.
Hello folks, it seems u forgot to expose in Python PyDeequ, the following Deequ Scala method, which in turn makes it impossible to use PyDeequ to produce row level DQ report - only column/dataset SUMMARY level DQ reports are possible at the moment
def rowLevelResultsAsDataFrame( sparkSession: SparkSession, verificationResult: VerificationResult, data: DataFrame): DataFrame = {
why is that, any plans to rectify and when?