Repository navigation
Is PyDeequ 2.0 still being actively developed? State of PyDeequ in general #270
Description
Activity
Thanks for raising this — it's a fair and important question for teams planning migrations.
To be transparent: PyDeequ development has slowed significantly. The core maintainers have had limited bandwidth, and while the 2.0.0b1 beta with Spark Connect support was a step in the right direction, there is currently no published timeline for a stable 2.0 release on PyPI or official Spark 4.x support.
The project is not formally abandoned, but it would be accurate to describe it as in maintenance mode rather than active feature development at this time. Community contributions are welcome and will be reviewed.
Regarding your specific questions:
-
Active development vs. maintenance mode — Closer to maintenance mode currently. Bug fixes and community PRs are still accepted, but there's no dedicated team driving new features on a regular cadence.
-
Spark 4.1 / PyPI 2.0 plan — There is no concrete roadmap for Spark 4.1 support at this time. The
_jvmbridge dependency is indeed the core blocker for Spark Connect compatibility, which the 2.0 beta began addressing. If your migration timeline is near-term, I'd recommend evaluating alternatives in parallel (e.g., native Deequ on Scala/Spark, or other Python data quality frameworks) rather than blocking on a PyDeequ 2.0 GA release.
If you or your team are interested in contributing toward Spark 4.x support, we'd welcome the collaboration. Otherwise, I understand if you need to hedge your bets given the uncertainty.
Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.
-
- changed the title
[-]Is PyDeequ 2.0 still actively developed? State of PyDeequ in general[/-][+]Is PyDeequ 2.0 still being actively developed? State of PyDeequ in general[/+]on Jun 12, 2026 This issue is tagging a maintainer (@chenliu0831) directly for visibility on the project's roadmap and development status. This requires an official response from the PyDeequ maintainers regarding Spark 4.x support plans and the 2.0 release timeline — I cannot speak authoritatively on behalf of the project's direction.
This has also been flagged for our maintainer team to review.
Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 67928783) — may not be fully accurate. Reply if this doesn't help.
I'm still pushing a few updates on Spark Connect support recently and working with rest of the maintaining team in AWS to make sure they are comfortable. Deequ & its Python binding (PyDeequ) will still be maintained. As far as I know, Spark connect and Spark 4+ support is desired by the internal owning team towards the second half of this year.
Reacted by Bill PrattDo we need to apply it to the 2.0 branch as well? What's the release cycle for these types of changes for 1.X?
It would depend on how long we need to keep Py4J bridge approach (could be forever). The version strategy is yet to be aligned with the maintaining team. If you feel the need to cherry pick some changes to V2 branch, shoot me a PR.
I've reached the limit of what I can assist with on this issue. The maintainer team has been notified and will take over.
Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: n/a) — may not be fully accurate. Reply if this doesn't help.
@chenliu0831 I'll look to open a PR. Is the branch v2_rewrite the right one open a PR to? Also, how can we get a new release of 1.X with my change since the latest version on pypi is 1.5 released April 2025?
I've reached the limit of what I can assist with on this issue. The maintainer team has been notified and will take over.
Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: n/a) — may not be fully accurate. Reply if this doesn't help.
Hi @chenliu0831 checking back in 2.0 progress. Now that Spark 4 is becoming more broadly available, do you have a timeline when Pydeequ 2.0 will be ready? Anything I can help with?
Reacted by Josh DeVriesI've reached the limit of what I can assist with on this issue. The maintainer team has been notified and will take over.
Generated by AI (model: us.anthropic.claude-sonnet-4-6, prompt: n/a) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadowI realize that Pydeequ 2.0 is targeting to move away from the Py4J-based JVM bridge to a modern client-server architecture using Spark Connect and not necessarily to target my request above for Spark 4 support. It appears deequ has some Spark 4 versions available. I wonder if a separate PR to add spark 4 support is required.
@billpratt Your PR for Spark 4.1 was merged in.
We’re currently running PyDeequ on Spark 3.5 and planning a future migration to Spark 4.1 / Delta Lake 4.1 / Python 3.13. Before we commit further to PyDeequ, we’d like to understand the project’s current state.
The 2.0.0b1 beta was announced in January 2025 with Spark Connect support — great direction, since _jvm bridge calls aren’t supported under Spark Connect which is the default in Spark 4.x. However, the last PyPI release was v1.5 in April 2025 and there’s been no follow-up since.
Two questions:
Is the project in active development or maintenance mode?
Is there a plan to support Spark 4.1 and get 2.0 published to PyPI?
This will directly inform whether we continue building on PyDeequ or evaluate alternatives ahead of our migration. Thanks for any visibility.