AWS Glue Job runs indefinitely after exception occurs when adding Checks to the VerificationSuite #156

Activity

1. chenliu0831

   added

   bugSomething isn't workingSomething isn't working

   researching

   on Sep 21, 2023
2. mattwaismann commented on Oct 2, 2023

   

   mattwaismann

   on Oct 2, 2023

   Author

   More actions

   I experienced this issue again when I missed a positional argument when incorrectly calling check.hasCompleteness("col_name")

   Expected behavior: Glue job stops with Exception: TypeError: hasCompleteness() missing 1 required positional argument: 'assertion'

   Actual behavior: Glue job continued to run indefinitely
3. github-actions commented on May 18, 2026

   

   github-actionsbot

   on May 18, 2026 – with GitHub Actions

   Contributor

   More actions

   Thanks for reporting this. This is a legitimate bug — when an exception occurs during Check construction (e.g., a typo like hasCompletenss or a missing argument), the Python callback server that PyDeequ starts for passing lambda functions to the JVM gets into a hung state, preventing the Spark/Glue job from terminating.

   The root cause is in the PythonCallback class in scala_utils.py — once the callback server is started, if an unhandled Python exception occurs before run() completes, there's no cleanup mechanism to shut down the callback server, leaving the JVM waiting indefinitely for a response.

   We'll look into adding proper error handling/cleanup around the callback server lifecycle to ensure exceptions propagate correctly and the job terminates. In the meantime, as a workaround, you can wrap your verification suite in a try/except and explicitly call:

   try:
       # your verification code
   except Exception as e:
       spark.sparkContext._gateway.shutdown_callback_server()
       spark.stop()
       raise e

   This ensures the callback server and Spark session are properly shut down even when an error occurs.

   This has also been flagged for our maintainer team to review.

   ---

   Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.
4. nikolauspschuetz

   added a commit that references this issue on Jun 27, 2026

   fix: avoid py4j callback-server default-port (25334) collision

   ...

   4ad5b3b
5. nikolauspschuetz

   added a commit that references this issue on Aug 31, 2026

   fix: avoid py4j callback-server default-port (25334) collision

   ...

   ea0022a
6. sudsali

   added a commit that references this issue on Sep 14, 2026

   fix: bind py4j callback server to a dynamic port to avoid 25334 colli…

   ...

   1262146