Question: expected behavior when running on the same hydra.run.dir

[yuvalkirstain]

Hi,

If I change the config with:

hydra:
  run:
    dir: ./outputs/my_experiment

and then I run more than once the experiment (either one after the other or concurrently), what will happen inside ./outputs/my_experiment?

I would expect it to end with an exception because this dir should be unique, but want to make sure.

[Jasha10]

The current behavior is not to trigger an exception. On the second run:

• outputs/my_experiment/my_app.log will be appended to
• the files in outputs/my_experiment/.hydra will be overwritten

If you want to check whether outputs/my_experiment exists (and raise an exception if it does exist), you can do this using a Callback. The callback should implement the methods on_run_start and the on_multirun_start (assuming that you use --multirun mode). Here is an example implementation:

# my_app.py
import os.path
from typing import Any
from omegaconf import DictConfig
import hydra
from hydra.experimental.callback import Callback

class MyCallback(Callback):
    def on_run_start(self, config: DictConfig, **kwargs: Any) -> None:
        if os.path.exists(config.hydra.run.dir):
            raise BaseException("Output dir already exists!")

    def on_multirun_start(self, config: DictConfig, **kwargs: Any) -> None:
        if os.path.exists(config.hydra.sweep.dir):
            raise BaseException("Output dir already exists!")

@hydra.main(config_path=".", config_name="config")
def my_app(cfg: DictConfig) -> None:
    print("app ran")

if __name__ == "__main__":
    my_app()

# config.yaml
hydra:
  callbacks:
    my_callback:
      _target_: my_app.MyCallback
  run:
    dir: ./outputs/my_experiment
  sweep:
    dir: ./outputs/my_multirun_experiment

Note that this implementation raises a BaseException (instead of an Exception subclass). Hydra's current implementation of callbacks uses a try/except block to catch subclasses of Exception and issue a UserWarning instead.

[yuvalkirstain]

Thanks for the quick and detailed response ❤️