Example for running on AWS (Glue, EMR, SageMaker, etc) #140

Activity

1. chenliu0831

   added

   good first issueGood for newcomersGood for newcomers

   on Jul 8, 2023
2. candicexu918 commented on Jul 15, 2023

   

   candicexu918

   on Jul 15, 2023

   More actions

   The new released pydeequ can be installed successfully in Glue but somehow failed to be imported with error "RuntimeError: SPARK_VERSION environment variable is required. Supported values are: dict_keys(['3.3', '3.2', '3.1', '3.0', '2.4'])", which results from the get_spark_version function in the config file.

   In Glue when I ran the below code it returned null -

   SPARK_VERSION = os.environ.get('SPARK_VERSION')
   print(SPARK_VERSION)
   

   But when I ran print(spark.version[:3]) it did return '3.1' as expected.

   Is this something can be enhanced in the package or should I explore other ways to use pydeequ in Glue?
3. chenliu0831 commented on Jul 18, 2023

   

   chenliu0831

   on Jul 18, 2023

   Author

   More actions

   @candicexu918 This is a workaround #138 (comment). Glue may have a unique setup of how you could pass env var to Python process
4. github-actions commented on May 18, 2026

   

   github-actionsbot

   on May 18, 2026 – with GitHub Actions

   Contributor

   More actions

   This is a known pain point. The SPARK_VERSION environment variable is required because pydeequ/configs.py reads it at import time via os.environ["SPARK_VERSION"], and AWS Glue doesn't set this variable by default.

   Workaround: Set the environment variable before importing pydeequ:

   import os
   os.environ["SPARK_VERSION"] = "3.3"  # or whatever your Glue Spark version is
   
   import pydeequ

   Alternatively, if you already have a SparkSession available, you can detect it dynamically:

   import os
   from pyspark.sql import SparkSession
   spark = SparkSession.builder.getOrCreate()
   os.environ["SPARK_VERSION"] = spark.version
   
   import pydeequ

   The root issue is that configs.py resolves the Spark version at module load time and only checks the environment variable — it doesn't attempt to detect the version from an active SparkSession. An enhancement to auto-detect from the runtime Spark version (falling back to the env var) would be a welcome PR. For now, the workaround above is the recommended approach for Glue, EMR, and SageMaker environments.

   ---

   Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.
5. github-actions

   added

   documentationImprovements or additions to documentationImprovements or additions to documentation

   enhancementNew feature or requestNew feature or request

   on May 18, 2026