- Implement
_convert_="object"
option forinstantiate
, enabling conversion of non-_target_
structured configs to instances of the backing dataclass / attr class. (#1719) - Enable layering of the
@hydra.main
decorator on top of other decorators produced usingfunctools.wraps
. (#2303) - Allow for non-leading dashes in override keys (#2363)
- support specifying an absolute path with
--config-path
(#2368) - Support python3.11 (#2443)
- Fix an issue where Hydra's exception-handling logic could raise an
AssertionError
(#2342)
hydra.runtime.choices
is now updated correctly during multi-run (#1882)hydra.verbose=True
now works with multirun. (#1897)- Fix a resolution error occurring when a nested class is passed as a
_target_
keyword argument toinstantiate
(#1914) - It is now possible to pass other callable objects (besides functions) to
hydra.main
. (#2042)
- Add support to Hydra's instantiation API for creation of
functools.partial
instances via a_partial_
keyword. (#1283) - Support defining basic sweeping in input config. (#1376)
- Improve error message with more context when an omegaconf exception occurs during the config merge step. (#1697)
- Add
--experimental-rerun
command-line option to reproduce pickled single runs (#1805) - Add experimental Callback for pickling job info. (#2092)
- Implement tab completions for appending to the defaults list (+group=option) and deleting from the defaults list (~group). (#1841)
- Enable the use of the pipe symbol
|
in unquoted strings when parsing command-line overrides. (#1850) - Support for Python 3.10 (#1856)
- Improve clarity of error messages when
hydra.utils.instantiate
encounters a_target_
that cannot be located (#1863) - The
instantiate
API now acceptsListConfig
/list
-type config as top-level input. (#1950) - Improve error messages raised in case of instantiation failure. (#2099)
- Add callback for logging JobReturn. (#2100)
- Support disable changing working directory at runtime. (#910)
- Support setting hydra.mode through config. (#394)
- The antlr version requirement is updated from 4.8 to 4.9, to align better with current antlr versions
- If user code raises an exception when called by
instantiate
, raise anInstantiateError
exception instead of an instance of the same exception class that was raised by the user code. (#1911) - Remove support for deprecated arg
config_loader
to Plugin.setup, and update signature ofrun_job
to requirehydra_context
. (#1953)
The remaining changes are protected by the new version_base support, which allows one to either configure Hydra to support older setups / config, or configure Hydra to use the following more modern defaults:
- Remove deprecated "old optional" defaults list syntax (#1952)
- Remove support for the legacy hydra override syntax (see deprecation notice). (#2056)
- Remove support for old hydra.experimental.{compose,initialize} interface
- Remove support for
_name_
and_group_
from package header (see deprecation notice) - Remove support for legacy default list interpolation format (see deprecation notice)
- Remove support for TargetConf class
- Remove support for strict flag from compose API (see deprecation notice)
- Remove support for ".yml" extensions, requiring ".yaml" instead.
- Default to not changing the working directory at runtime. Use
hydra.job.chdir=True
to reinstate old behavior. - Default to not adding any directory to the config path. (see
config_path
options)
- Allow range() in override grammar to have only one argument (the stop value), e.g. range(3) (#1664)
- Add support for a HYDRA_DEPRECATION_WARNINGS_AS_ERRORS envorinment variable. When it is set, using deprecated features causes an exception instead of a warning. (#1689)
- To make migration from Hydra 1.0 to 1.1 easier, Hydra will now issue a warning if the primary config defines config values and Defaults List if the Defaults List does not specify
_self_
(#1755)
- Re-introduced a deprecated strict flag in the Compose API (#1694)
- Relax handling of legacy hydra overrides in the defaults list to make migration from Hydra 1.0 to 1.1 easier (#1748)
- Allow Defaults List interpolation in nested configs and document that the interpolation keys must be absolute (#1668)
- instantiate API now supports
target
as a parameter name (#1669) - Fix
--cfg=job --resolve
and--help --resolve
so that the ${hydra:...} resolver now prints properly (#1681) - Fix unexpected changes to defaults list in some cases when appending an experiment (+experiment=test) (#1706)
- Fixed composition where top level of a config composed from a config group is a list (#1724)
- Fix bug where overriding hydra config groups more than once without using the override keyword resulted in an error instead of deprecation warnings (#1730)
- Fix failure when sanitizing stack traces resulting from job exceptions (#1739)
This is the biggest Hydra release yet.
- OmegaConf 2.1 supports relative interpolations, nested interpolations, more powerful resolvers and better compatibility with plain Python dict and list.
- Recursive defaults list: Every config can now have a Defaults List.
- Recursive instantiation: When instantiating objects using the
instantiate
API, nested defined objects are instantiated automatically.
There have also been many bug fixes and performance improvements, as well of as some breaking changes - both in Hydra and in OmegaConf. Please check the API changes and deprecation sections in these release notes and in the OmegaConf release notes.
- Upgrade to OmegaConf 2.1. Please check the release notes for it (#1426)
- Support for Python 3.9. (#1062)
- Improve performance of config composition in a benchmark by 64% (#1328)
- Allow
@
,$
and?
symbols in unquoted values in overrides (#1074, #1437, #1597) - Add a --resolve flag that can be used with the --cfg and --help flags to resolve interpolations before the config is printed. (#1585,#1482)
- It is now possible to disable Hydra's logging configuration (#1130)
- Support for Zsh tab completion (#347)
- Support for force-add of config values via
++key=value
(force-add overrides if the value exists and adds it otherwise) (#1049) - New experimental Callback API (#1402)
- Support for Defaults List in any config (Recursive relative defaults) (#1170)
- Composition order of a config with a Defaults List can be specified with the
_self_
keyword. (#326) - Support for configuring the config search path from the primary config (#274)
- Add support for selecting multiple configs from the same Config Group (#499)
- Final choices of defaults list are retained in the dictionary hydra.runtime.choices (#956)
- Support for recursive instantiation with
hydra.utils.instantiate()
(#566) - Instantiate now fully supports positional arguments (#1432)
- Support for converting parameters to primitive containers during instantiation (#1015)
- It is now possible to override the
_target_
of instantiation with a Python type or a string (#1017)
- Default composition order change, see this for details (#1217)
- Remove deprecated strict flag from
@hydra.main
and theCompose API
(#1010) - Passing a config name as config_path to
@hydra.main
is now an error (#1010) - Promote
hydra.experimental.{compose,initialize,initialize_config_dir,initialize_config_module}
out from thehydra.experimental
module to the hydra module. (#1030) - The override grammar now requires that, in quoted strings, any sequence of \ preceding a quote (either an escaped quote, or the closing quote) must be escaped (#1600)
- Support partial failures in MULTIRUN mode (#1377)
- Add
HydraContext
torun_job
and Launcher/Sweeper'ssetup
(#1498)
- Instantiate is now recursive by default, use
_recursive_=False
to disable recursive instantiation. (#566) - Deprecated TargetConf, you should no longer be extending or annotating with it (#1010)
- Remove ObjectConf and deprecated params support that was deprecated in Hydra 1.0 (#1010)
- HydraContext required in run_job, Launcher and Sweeper's setup methods, see issue for details (#1498)
- ConfigSourcePlugins needs to be modified to support recursive defaults, see link for details (#1080)
- Hydra no longer erroneously changes the USER environment variable in pytest unit tests once installed (#1059)
- Fix a bug where Structured Config has a field like
list : Optional[List[int]] = None
(#1117) - No longer modifies exception stack trace when running under a debugger (#1237)
hydra.job.{id,num}
are now properly passed to jobs in multirun (#1270)- Add support for
%f
directive (microseconds) to the${now:PATTERN}
resolver (#1287) - Fixed a bug where tab completion did not work if the Defaults List had a missing (???) item. (#1381)
- Fix
+
overrides to properly insert new values into Structured Configs. (#1515) - Fix edge cases where using the command line to set a key to a value containing a string ending with a backslash could crash (#1600)
--cfg=hydra
now works for read-only config (#1461)- Change hydra.job_logging and hydra.hydra_logging to be non-optional (#1656)
- Configs with unicode characters are now working on Windows (#1659)
- Add Optuna Sweeper plugin
- New Defaults List page (#1170)
- New Extending Configs pattern (#1170)
- Major updates to the Packages page (#1170)
- New Configuring Experiments pattern (#1170)
- Documentation updates for configuring search path (#1449)
- Documentation for
hydra.callbacks
(#1526) - Document how to configure Hydra plugins (#1582)
This release contains a few small changes to make Hydra 1.0 a bit more compatible with OmegaConf 2.1 and Hydra 1.1 to enable easier migration. There is no risk in using it if you are using 1.0.6 now as it just makes the migration a bit smoother in some scenarios:
- If you are upgrading OmegaConf to 2.1 before upgrading Hydra to 1.1.
- It enables some configs to be compatible with both versions in some rare cases related to the change in the default composition order in Hydra 1.1.
- Enables Hydra 1.0 to work with OmegaConf 2.1 (Manual installation of OmegaConf 2.1 is required) (#1634)
- Hydra 1.0 ignores
_self_
in Defaults List for partial future compatibility with Hydra 1.1 (#1712)
- Fix interpolation in Hydra config node (#1335)
- Remove
Plugins
fromSingleton.get_state()
(#1330)
- Support Python 3.9 . (#1062)
- Deprecate support for renaming packages via the command line. (#1140)
- Fixed hydra.job.id and hydra.job.num not getting passed to jobs in multirun (#1270)
- Support
additional_parameters
as an optional param in the Submitit launcher plugin. - Add Optuna Sweeper plugin
- Limit OmegaConf dependency to 2.0 versions (#1253)
- Hydra no longer errorenously changes the USER environment variable in pytest unit tests once installed (#1059)
- Allow @ symbol in unquoted values in overrides (#1074)
- Fix a bug where Structured Config has a field like 'list : Optional[List[int]] = None` (#1117)
- Switch Ax Sweeper plugin to use new Hydra command line syntax
- Switch Nevergrad Sweeper plugin to use new Hydra command line syntax
- Revert the use of "_" to separate the path element in keys for the Ax sweeper plugin.
- Fixed interaction between interpolation and instantiate (#1001)
- Fix float parsing for values where the exponent starts with 0 (e.g 1e-05) (#999)
This is a small update improving support for config dataclasses generated by configen
- Overridden parameters annotated as Any in the config are not type checked by instantiate() (#982)
- Add ability to escape special characters in unquoted command line values (#954)
- Add support for merging dicts and lists specified in the commend line (#939)
- Fix appending dict values with new fields to config with + (#946)
- Fix for Override Grammar integer parsing: Integers are now compatible with Python integers ('+' prefix, correct handling of '_') (#954)
- Upgrade to OmegaConf 2.0.2 (#755)
Hydra 1.0 is out! See blog post for high level details.
Major new features in Hydra 1.0
- Config type safety via Structured Configs
- More powerful command line
- New plugins enabling remote launching and hyper parameter optimization
- Improved error reporting
- Reduce nesting levels with config packages
These release notes are covering the changes since 1.0.0rc4.
To see the complete release notes for 1.0 take a look at the release notes of:
- Add support for casting interval to ints and floats (#915)
- Fix to flush logging instead of shutting it down at job end (#833)
- compose() no longer initialized logging subsystem (#833)
- Fix overriding of hydra.job.env_set from the command line (#854)
- Fix tab completion to not output errors in some cases when using --multirun (#856)
- Fix OmegaConf custom resolvers not being propagated to Sweepers properly (#861)
- Interpolation accepts any char between ${ and } (#891)
- Update the signature of hydra.utils.instantiate() allow input configs of Any type (#896)
- Fix instantiate to work correctly when parameters are interpolations into a parent node (#904)
- Apps will now raise underlying exception if env HYDRA_FULL_ERROR=1 (#926)
- Fix integer/float choice from spec in the Nevergrad plugin
- New page in Structured Configs tutorial: Static schema with many configs (#918)
- New page in Structured Configs tutorial: Dynamic schema with many configs (#918)
Be aware that cfg.pretty()
is now deprecated and you should use OmegaConf.to_yaml(cfg)
instead.
- Upgrade to OmegaConf 2.0.1rc12 (#889)
- Fix bug with bash autocompletion for the users who has
.
in their PATH environment variable (#868) - Fix instantiate to return None when None config is passed (#887)
- Extended Override grammar supports range(),glob(),type casts,shuffle, sort and more. (#752)
- Add support for specifying an additional config dir in the command line (#874)
- Allow overriding of the
defaults
andhydra
nodes in primary Structured Configs that does not mentioneddefaults
andhydra
. (#877)
- ObjectConf is deprecated for TargetConf. See upgrade page. (#882)
- All plugins updated to use TargetConf. Be sure to update all your Hydra plugins (#882)
- Instantiated objects no longer have a params node in the conf. keys like hydra.launcher.params.foo are changed to hydra.launcher.foo. Update your configs and overrides if you were overriding such parameters. (#882)
- Fixed parsing of two nested lists with whitespace between them: [[a], [b]] (#836)
- New Extended command line syntax page (#752)
- Introducing a DSL for the command line override syntax (#797)
- Sanitize exception messages by default, showing only interesting stack frames (#646)
- Add a new --info flag to show Hydra debug information without running the user function. (#662)
- Add --package flag, can be used with --cfg to print a specific config package (#674)
- Improve the Compose API initialization methods (#702)
- Switch internal package resource loader from pkg_resources to importlb_resources (#719)
- Allow interpolation in the Hydra config node to access the application config (#771)
- Hydra provides pytest fixtures (hydra_sweep_runner, hydra_task_runner, hydra_restore_singletons) (#687)
- Added contexts to help initialize Hydra in a localized context (Mostly for Unit Tests and Jupyter Notebooks) (#687)
- Prints error when command line override looks like a sweep but --multirun is not specified (#712)
- Frozen Structured Configs are respected during composition command line overrides (#724)
- Hydra argparse flags are now consistently using dash-style. (#661)
- Restructure the configuration of the Submitit launcher (#682)
- Deprecated cls in favor of target in hydra.utils.{instantiate()/call{}} (#721)
- Add hydra/py.typed to pip package to enable type checking in downstream projects (#657)
- Fix initialization of Hydra in a module imported from a Jupyter Notebook (#695)
- Fix handling of main module override via HYDRA_MAIN_MODULE (#717)
- Fix interpolations in overrides to evaluate lazily (#725)
- Fixed handling of multirun when target config node is introduced by composition (#726)
- Sweepers are now validating config composition before launching (#775)
- Save
.hydra/*.yaml
before task execution (#678)
- Add Redis Queue Launcher plugin (Jan-Matthis)
- Upgrade ax-platform used by Ax sweeper plugin to 0.1.13 or newer
- Add a unit testing example in the Hydra example application (#687)
- New Terminology page (#795)
- New Advanced/Hydra in Jupyter Notebooks page (#795)
- New Advanced/Hydra in Unit Tests page (#795)
- New Advanced/Override Grammar/Basic page (#795)
Hydra 1.0 is a major release introducing many new features and breaking some compatibility.
- Upgrade to OmegaConf 2.0 (Release notes) (#630)
- Optional config type safety via Structured Configs (#629)
- Improve command line and config composition error reporting (#349)
- Hydra config can now be accessed through interpolation using ${hydra:key}, for example ${hydra:job.name} (#325)
- Support for setting environment variable of running job (#7)
- Changes command line processing (requiring + and ~ prefixes for appending and removing items) (#598)
- Introducing
@package
header for config files (#586) - Add command line override flags for
config_path
andconfig_name
(#386) - hydra.main() now take an optional cfg object to passthrough to the function (#575)
- Add hydra.experimental.{initialize_with_file, initialize_with_module} (#574)
- Support for disabling the creation of the
.hydra
subdirectory by overriding "hydra.output_subdir" to "null" (#324) - Add
hydra.utils.call()
to call methods and functions as well as instantiate objects. Search module paths more generically. (#498) - Add support for overriding package from command line and defaults list (#235)
- Config source is now abstracted, allowing additional config sources to be used (#257)
- New ConfigSource plugin API allowing configs to be provided by external plugins (#367)
- Add isort to ensure imports are always sorted (#340)
- Codebase is now passing mypy --strict type checking (#342)
- Improve performance of plugin discovery and instantiation (#489)
- Modules whose name starts with "_" are skipped during plugin discovery (#494)
- Add Ax Sweeper plugin (Shagun Sodhani)
- Add Nevergrad Sweeper plugin (Jérémy Rapin)
- Add Joblib Launcher plugin (Jan-Matthis)
- Add Submitit Launcher plugin to launch jobs to SLURM clusters
- Add Fish shell Tab Completion plugin (Binsheng Liu) (#549)
- Drop support Python 2.7 and 3.5 (#313)
- hydra.main() now takes an additional optional config_name and composite-style config_path is deprecated (#395)
- Launcher API launch method now takes an additional initial_job_idx indicating the id of the first job in the batch (#284)
- Singleton metaclass is now exposed at hydra.core.Singleton (#371)
- Moved HydraConfig from hydra.plugins.common.utils to hydra.core (#371)
- Move several formerly internal APIs to hydra/core to ensure plugins does not need to use internal APIs (#371)
- Plugin import now requires explicit name (from hydra.plugins.launcher import Launcher) (#371)
- Object Config "class" field is deprecated in favor of "cls" and will be removed in a future version. (#389)
- Experimental compose API config_file changed to config_name (#395)
- User plugins should be modified to not import twice during plugin discovery. see issue for details. (#482)
- Change hydra.core.plugins.Plugins class to a Singleton. access should be changed to the pattern Plugins.instance().foo() (#489)
- Plugins should now include test fixtures (sweep_runner, task_runner) via a standardized conftest.py (#521)
- Switch Python 3 native namespace packages for plugins (See task for details) (#534)
- Packaged configuration directories now requires an
__init__.py
at their top level (#536) - Appending config groups to the defaults list via the command line now requires a + prefix (#598)
- Removing an item from the defaults list by assigning null (db=null) is deprecated, use ~db instead (#598)
- Installed Hydra applications no longer need have an additional
entry()
function on the stack (#92)
- Fix a bug causing sys.exit() error code to not be propagated (#351)
- Shutdown logging subsystem aftter job finishes to ensure log files are flushed and closed (#378)
- Fix a bug with utils.instantiate() failing if params contains interpolated values. (#388)
- Allow hydra.utils.instantiate() to accept non primitive objects for passthrough by name (#400)
- Fix to work when an Hydra app is executed in Jupyter notebook using the %run command (#481)
- Plugins are no longer imported twice during plugin discovery (#482)
- to_absolute_dir(path) now converts relative path to be relative to os.cwd() when used outside of Hydra (#496)
- Working examples are provided for all Hydra plugins in plugins/examples (#253)
- The basic tutorial was rewritten to reflect many changes (#602)
- Add a new tutorial covering Structured Configs (#628)
- Pin Hydra 0.11 to OmegaConf 1.4 to avoid breaking compatibility when OmegaConf 2.0 is released (#334)
- Document a simple Ray example (#317)
- Change website from cli.dev to hydra.cc (#314)
- Fixes --cfg command line flag not working (#305)
- Add hydra.experimental API for composing configuration on demand (hydra.experimental.compose) (#219)
- Add hydra.utils.get_original_cwd() to access original working directory and hydra.utils.to_absolute_path() to convert a path to absolute path (#251)
- Change hydra logging format pattern, example: "[2019-10-22 16:13:10,769][HYDRA] Installed Hydra Plugins" (#254)
- Change --cfg to require config type (one of 'job', 'hydra' or 'all') (#270)
- Upgrade to OmegaConf 1.4.0, see full change log here (#280)
- Experimental support for Jupyter notebooks via compose API (#281)
- Allow configuring override_dirname via hydra.job.config.override_dirname to exclude specific keys or change separator characters (#95)
- Fix a bug that caused out of order composition when mixing config-groups with non-config-group in the defaults block (#261)
- Allow '=' to be used in the value of an override, eg. foo=bar=10 (key foo, value bar=10). (#266)
- Allow composing config files with dot in their name (foo.bar.yaml) (#271)
- hydra-colorlog plugin adds colored log output.
- Document utils.get_original_cwd() and utils.to_absolute_path("foo") (#251)
- Change the default sweep subdir from ${hydra.job.num}_${hydra.job.id} to ${hydra.job.num} (#150)
- App help now contains config groups and generated config App help can now be customized Hydra help is now available via --hydra-help (#1)
- Simplify install and uninstall commands for tab completion (#200)
- hydra.runtime.cwd now contains the original working directory the app was launched from (#244)
- Fix an error with tab completion that occurred when a TAB was pressed after two spaces (#203)
- Fix a bug when sweeping over groups that are used in an interpolation in defaults (#206)
- Fix tab completion for cases where app name looks like foo.par (#213)
- Describe news fragment types in more details in development/contributing (#150)
- Examples dir now mirrors the structure of the website docs (#209)
- Old demos directory was removed and a new tutorial directory as added (#167)
- Make strict mode the default when a config file is specified directly (#150)
- Replace --verbose with a standard override (hydra.verbose) (#161)
- Add IntegrationTestSuite to test_utils for testing Launcher plugins (#168)
- Add support for specifying which config to print in -c, options are job (default), hydra or all. (#176)
- Hydra is now hosted on https://cli.dev! (#75)
- Move all Hydra configuration files to a subfolder (.hydra by default) under the output folder (#77)
- Fix a bug in tab completion of missing mandatory values. (#145)
- Fix a bug with multirun overriding free config groups (not in defaults) when strict mode is enabled (#181)
- Fix a bug that prevented sweeping over an unspecified ('???') default group (#187)
- Updated the primary tutorial on the web site and added a brand new tutorial directory (#58)
- Documented debugging methods in website (#6)
- Move FAIRTask, Submititit and FAIR-cluster-defaults plugins to fairinternal/hydra-fair-plugins repository (#138)
- Remove Fairtask and Submitit example configs from demos as they are no longer needed (#146)
- Created hydra-core pip package, Hydra can now installed with 'pip install hydra-core' (#143)
- Finalized submitit support (#18)
- Add default launcher config for Fairtask and Submitit launcher plugins
- hydra.utils.instantiate() can now take an additional optional kwargs dictionary to pass to constructor of the instantiated object (#142)
- Initial support for bash completion, see documentation for details (#8)
- Fixed Town Crier to generate correct links to issues in generated news file. (#122)
- Fixed bug with overriding hydra defaults from user config file. (#123)
- Fixed Singletons not getting serialized to remote jobs. (#128)
- Fixed a bug that prevented using submitit in strict mode (#129)
- Fixed example submitit config to set the job name (#133)
- Changed Hydra core defaults to be more appropriate for open source use. To get the FAIR cluster defaults install the fair_cluster plugin (#103)
- Changed output directories default to group by day (output/DAY/TIME instead of output/DAY_TIME) (#121)
- Added the ability to prevent a default from being merged it by assigning null to it (hydra/launcher=null) (#106)
- Implemented Plugin discovery mechanism, this can be used to find all installed plugins that implement a specific interface (#119)
- Implemented an API to for manipulating the config search path (#120)
- Fixed config loading to properly support the use case in demos/8_specializing_config (#104)
- Fixed error message when the user config contains defaults that is a mapping and not a list (#107)
- Fixed config loading order to allow properly overriding Hydra config groups from user config (#115)
- New plugin: fair_cluster Change Hydra defaults to be appropriate for the FAIR cluster
- Initial search path and plugins documentation (#103)
- Hydra config groups were moved to the hydra/namespace (#101)
- Removed support for .hydra directory, Hydra can be configured directly from the job config. (#91)
- Config loading rewrite fixed #88 (#88)
- Move non-public APIs into hydra._internal:
- Moved non-API code into hydra._interanl to flag it as private.
- Moved plugin interfaces into hydra.plugins.
- Moved code meant to be used by plugins to hydra.plugins.common. (#78)
- Integrated towncrier to bring you this news! (#45)
- Hydra is now compatible with Windows (#63)
- Hydra apps can now be packaged and installed along with their configuration files. (#87)
- It is now possible to use ${override_dirname} in the output directory of a local job (#31)
- Override_dirname separator changed from : to =, for example: foo/a:10,b:10 => foo/a=10,b=10 (#63)
- Fixed automatic detection of missing copyright headers (#72)
- fixed a bug that caused an empty config to be returned if specified config file did not have a .yaml extension. (#80)
- Multi change diff:
- Logging config search path in verbose logging to assist debugging of config load issues
- Saving hydra.yaml into the job dir to assist debugging hydra issues
- Fixed a bug caused by fairtask logging change
- Improved integration-tests debuggability by switching hydra to debug logging in them
- Added selective plugin testing to nox using env, for example PLUGINS=fairtask would only test fairtask. (#87)
- Improved the contributing docs (#45)
- Documented Hydra app packaging under Deployment/Application packaging (#87)