pdb can't be used if hydra main decorator used

[ferrine]

🚀 Feature Request: PDB support

Motivation

I got used to debug pipelines with pdb, but it looks to be impossible with hydra decorator

Example

Consider a script

# test.py
from omegaconf import DictConfig
import hydra


@hydra.main()
def main(cfg: DictConfig):
    1/0


if __name__ == '__main__':
    main()

and you want to use pdb to fix the error. For deep learning it might be inspecting weights for nans, grad norms, stack levels.
Without any interaction, program finishes with error code 1

$ python test.py
Traceback (most recent call last):
  File "test.py", line 7, in main
    1/0
ZeroDivisionError: division by zero

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

with HYDRA_FULL_ERROR=1

$ HYDRA_FULL_ERROR=1 python test.py
Traceback (most recent call last):
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 203, in run_and_report
    return func()
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 353, in <lambda>
    lambda: hydra.run(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 105, in run
    return run_job(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/core/utils.py", line 123, in run_job
    ret.return_value = task_function(task_cfg)
  File "test.py", line 7, in main
    1/0
ZeroDivisionError: division by zero

with python debugger pdb which is possible to catch any unexpected exception

$ HYDRA_FULL_ERROR=1 python -m pdb -c continue test.py
Traceback (most recent call last):
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 203, in run_and_report
    return func()
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 353, in <lambda>
    lambda: hydra.run(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 105, in run
    return run_job(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/core/utils.py", line 123, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/ferres/ntl/cfan/experiments/hydra/test.py", line 7, in main
    1/0
ZeroDivisionError: division by zero
The program exited via sys.exit(). Exit status: 1
> /home/ferres/ntl/cfan/experiments/hydra/test.py(1)<module>()
-> from omegaconf import DictConfig
(Pdb) ll
  1  -> from omegaconf import DictConfig
  2     import hydra
  3     
  4     
  5     @hydra.main()
  6     def main(cfg: DictConfig):
  7         1/0
  8     
  9     
 10     if __name__ == '__main__':
 11         main()
(Pdb) 

but expected to be

 HYDRA_FULL_ERROR=1 python -m pdb -c continue test.py                  
Traceback (most recent call last):
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 203, in run_and_report
    return func()
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 353, in <lambda>
    lambda: hydra.run(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 105, in run
    return run_job(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/core/utils.py", line 123, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/ferres/ntl/cfan/experiments/hydra/test.py", line 7, in main
    1/0
ZeroDivisionError: division by zero
Traceback (most recent call last):
  File "/usr/lib/python3.8/pdb.py", line 1704, in main
    pdb._runscript(mainpyfile)
  File "/usr/lib/python3.8/pdb.py", line 1573, in _runscript
    self.run(statement)
  File "/usr/lib/python3.8/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/ferres/ntl/cfan/experiments/hydra/test.py", line 1, in <module>
    from omegaconf import DictConfig
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/main.py", line 32, in decorated_main
    _run_hydra(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 352, in _run_hydra
    run_and_report(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 276, in run_and_report
    raise ex
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 203, in run_and_report
    return func()
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 353, in <lambda>
    lambda: hydra.run(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 105, in run
    return run_job(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/core/utils.py", line 123, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/ferres/ntl/cfan/experiments/hydra/test.py", line 7, in main
    1/0
ZeroDivisionError: division by zero
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/ferres/ntl/cfan/experiments/hydra/test.py(7)main()
-> 1/0
(Pdb) ll
  5     @hydra.main()
  6     def main(cfg: DictConfig):
  7  ->     1/0

Pitch

I see some possible solutions
A simple solution is to add a flag HYDRA_DEBUG to env variables as done with HYDRA_FULL_TRACE. Here Exception handling is too broad and after a clean up you can still decide between sys.exit(1) and raise ex (changing this locally solved my problems)

Describe alternatives you've considered
I considered manually hacking source code of /hydra/_internal/utils.py (1 line change)

Are you willing to open a pull request? (See CONTRIBUTING) <-- broken link in template
Yes, why not

[omry]

1. Please set a title to your issue.
2. Please explain what the problem is before suggesting solutions.

[ferrine]

@omry done

[omry]

Thanks.

[omry]

#930 is changing the behavior of Hydra to raise the exception if HYDRA_FULL_ERROR is true instead of printing it and exiting.