Repository navigation
FileSystemMetricsRepository file to be a parquet (/ delta) #185
Description
Activity
- changed the title
[-]FileRepository file to be a parquet (/ delta)[/-][+]FileSystemMetricsRepository file to be a parquet (/ delta)[/+]on Jan 6, 2024 I agree Parquet would be a much better storage format and that Deequ metrics should be properly data modeled.
Could we have a separate feature request of supporting Delta/Iceberg which uses parquet file format in storage layer? It would good to spec out the use-case and benefits in that one.
Would also be good to know why .json format was a chosen? And is there any way we can benefit from this choice?
I think human-readable might be the biggest benefit, but the context of the design might have been lost in history.
Thanks for the detailed feature request! I agree that Parquet would be a much better storage format for the metrics repository, especially at scale with thousands of anomaly validations. The JSON format does become a bottleneck for analytical queries in a Spark environment.
This is a valid enhancement that would require implementing a new
ParquetMetricsRepository(and potentially a Delta-based one as a follow-up). I'm escalating this for development consideration. As noted in the thread, a separate issue for Delta/Iceberg support would help scope that work independently.For now, as a workaround, you could load the JSON metrics repository into a DataFrame and write it out as Parquet/Delta yourself for analytical queries, though a native implementation would obviously be cleaner.
This has also been flagged for our maintainer team to review.
Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.
Is your feature request related to a problem? Please describe.
We run thousands of anomalies validations on hundreds of data sources. We often want to explain why they fail. To do so - first step is to look at the underlying metrics - persisted in PyDeeQu managed json. Processing json in spark environment is not efficient (both programmatically and not).
Describe the solution you'd like
One way to provide explanations for failed anomalies is to expose json repository file underlying FileSystemMetricsRepository to analytical user. Since PyDeeQu is a PySpark framework the best and natural choice is to store repository data as .parquet/delta files instead of json. This could also cover:
Describe alternatives you've considered
Consider DELTA format - probably way batter choice but requires more development to enable incremental processing and rollbacks.
Additional context
Would also be good to know why .json format was a chosen? And is there any way we can benefit from this choice?