Example for persisting metrics from metricsRepository to file on S3 and reloading the metrics file in Glue to perform anomaly detection #144

Activity

1. chenliu0831

   added

   enhancementNew feature or requestNew feature or request

   documentationImprovements or additions to documentationImprovements or additions to documentation

   on Aug 3, 2023
2. github-actions commented on May 18, 2026

   

   github-actionsbot

   on May 18, 2026 – with GitHub Actions

   Contributor

   More actions

   Thanks for the feature request! This is a common ask for AWS Glue users. Here's how you can persist metrics to S3 and reload them for anomaly detection:

   from pydeequ.repository import FileSystemMetricsRepository, ResultKey
   from pydeequ.analyzers import AnalysisRunner, Size, Completeness, Mean
   from pydeequ.anomaly_detection import RelativeRateOfChangeStrategy
   from pydeequ.verification import VerificationSuite
   import pydeequ
   
   # Use an S3 path directly for the FileSystemMetricsRepository
   metrics_s3_path = "s3://your-bucket/path/to/metrics.json"
   repository = FileSystemMetricsRepository(spark, metrics_s3_path)
   
   # Job Run 1: Compute and persist metrics
   resultKey = ResultKey(spark, ResultKey.current_milli_time(), {"source": "my_table", "run": "daily"})
   
   analysisResult = AnalysisRunner(spark) \
       .onData(df) \
       .addAnalyzer(Size()) \
       .addAnalyzer(Completeness("col_a")) \
       .useRepository(repository) \
       .saveOrAppendResult(resultKey) \
       .run()
   
   # Job Run 2 (subsequent run): Reload and run anomaly detection
   repository = FileSystemMetricsRepository(spark, metrics_s3_path)
   
   verificationResult = VerificationSuite(spark) \
       .onData(new_df) \
       .useRepository(repository) \
       .addAnomalyCheck(
           RelativeRateOfChangeStrategy(maxRateDecrease=0.5, maxRateIncrease=2.0),
           Size()
       ) \
       .saveOrAppendResult(resultKey) \
       .run()

   Key points:

   • FileSystemMetricsRepository works with S3 paths natively (Spark's Hadoop FS handles s3://).
   • The metrics are stored as JSON — Deequ's repository format is JSON-based, not Parquet. If you need Parquet, you'd need to export the metrics DataFrame separately via AnalyzerContext.successMetricsAsDataFrame() and write it yourself, but that won't be reloadable as a MetricsRepository.
   • Each run appends to the same JSON file, building history over time.

   For Glue specifically, ensure your Glue job role has read/write access to the S3 path. This approach avoids recomputing metrics over all historical partitions.

   We'd welcome a community contribution adding an S3-specific tutorial to the tutorials/ directory if anyone wants to take this on!

   ---

   Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.