Enable Custom Analyzer in python by implementing pydeequ.analyzers._AnalyzerObject

Is your feature request related to a problem? Please describe.
Enabling and showing the example of how to extend pydeequ.analyzers._AnalyzerObject to define custom Analyzer in python.

Describe the solution you'd like
Be able to implement:

class MyCustomAnalyzer(_AnalyzerObject):
    """Get the maximum of a numeric column."""

    def __init__(self, column, my_property: str = None):
        """
        :param str column: column to find the maximum.
        :param str my_property: custom property
        """
        self.column = column
        self.my_property = my_property

    @property
    def _analyzer_jvm(self, foo: AnalyzerInput) -> AnalyzerOutput:
       # my custom transformation that transforms well defined AnalyzerInput into AnalyzerOutput
        bar: AnalyzerOutput = ...
        return bar

and then run it in VerificationSuite, ex:

results = (VerificationSuite(spark)
            .onData(df)
            .useRepository(repository)
            .saveOrAppendResult(ResultKey(spark, ResultKey.current_milli_time(), {'tag': 'my-tag'}))
            .addAnomalyCheck(OnlineNormalStrategy(
                        lowerDeviationFactor=0.01,
                        upperDeviationFactor=0.01,
                        ignoreStartPercentage=0.1,
                        ignoreAnomalies=False,
            ), MyCustomAnalyzer("column_name", my_property="yeey!")) 
            .run())

Describe alternatives you've considered
When calculating Anomalies - every time I have a custom metrics (to focus attention - lets say Sum() / CountDistinct()) I build temporary table that has one row, ex:

|        value_unique_name          |
-----------------------------------
| <value of Sum() / CountDistinct() |

and than run Anomaly over pydeequ.analyzers.Sum (or Mean, ie. transformation that gives identity). Its best if those custom metrics have seperate pydeequ metrics repository to the source table.

Additional context
In anybody hacked it in a better way than described in Describe alternatives you've considered let us know in the comments!

Thanks for this feature request! This is a valid enhancement — enabling users to define custom analyzers in Python that can be seamlessly used with VerificationSuite, AnalysisRunner, and anomaly detection would be a great addition to PyDeequ.

The challenge here is that _analyzer_jvm currently needs to return a JVM-side Deequ Analyzer object (a Scala class), so a pure-Python custom analyzer would require either:

A bridge mechanism (e.g., implementing a Scala Analyzer interface via Py4J callbacks), or
A composable analyzer pattern that lets users combine existing JVM-side analyzers (like your Sum / CountDistinct example) without needing a temporary table workaround.

I'm marking this as an enhancement for the team to investigate. Your workaround of pre-computing the metric into a single-row DataFrame and running an identity analyzer over it is currently the most practical approach, but we agree it shouldn't be necessary. We'll look into what's feasible given the Py4J/JVM boundary constraints.

This has also been flagged for our maintainer team to review.

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.

Navigation Menu

Repository navigation

Enable Custom Analyzer in python by implementing pydeequ.analyzers._AnalyzerObject #186

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Enable Custom Analyzer in python by implementing pydeequ.analyzers._AnalyzerObject #186

Description

Activity

github-actions commented on May 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions