Ax 1.2.2

NOTE: this will be the last Ax release before SQLAlchemy becomes a required dependency

Deprecations

• Add deprecation warning to AxClient (#4749)
• Add deprecation warning to 'optimize' loop API (#4697)
• Deprecate Trial.runner (#4460)
• Deprecate TensorboardMetric's percentile in favor of quantile (#4676)
• Deprecate default_data_type argument to Experiment (#4698)

New Features

• Efficient leave-one-out cross-validation for Gaussian processes (#4631)
• Add patience parameter to PercentileEarlyStoppingStrategy (#4595)
• Log-scale support for ChoiceParameter (#4591)
• Support ChoiceParameter in Log transform (#4592)
• Add robust trial status polling to Orchestrator (#4756)
• Expose validation for TL experiments and fetching of candidate TL sources through AxService (#4615)
• Add PreferenceOptimizationConfig with storage layer support (#4638)
• Add PLBO transform and metric ordering validation (#4633)
• Add expect_relativized_outcomes flag to PreferenceOptimizationConfig (#4632)
• Add kendall tau rank correlation diagnostic (#4617)
• Vectorize SearchSpace membership check for performance (#4762)

Analyses

• New UtilityProgression Analysis for tracking optimization progress over time (#4535)
• New Best Trials Analysis for identifying top-performing trials (#4545)
• New Early Stopping Healthcheck analysis (#4569)
• New Predictable Metrics Healthcheck analysis (#4598)
• New Baseline Improvement Healthcheck analysis (#4673)
• New Complexity Rating Healthcheck for assessing optimization difficulty (#4556)
• Analysis to visualize experiment generation strategy (#4759)
• Add Pareto frontier display on MOO objective scatter plots (#4708)
• Add Progression Plots for MapMetric experiments to ResultsAnalysis (#4705)
• Add SEM display option to ContourPlot (#4690)
• Add markers to ProgressionPlot line charts (#4693)
• GraphvizAnalysisCard and HierarchicalSearchSpaceGraph visualization (#4616)
• IndividualConstraintsFeasibilityAnalysis replaces ConstraintsFeasibilityAnalysis (#4527)

Bug Fixes

• Fix tied trial bug in PercentileESS: Use rank() for n_best_trial protection (#4587)
• Fix StandardizeY not updating weights in ScalarizedObjective (#4619)
• Fix StratifiedStandardizeY behavior with ScalarizedObjective & ScalarizedOutcomeConstraint (#4621)
• Fix floating point precision issue in step_size validation (#4604)
• Fix progression normalization logic in early-stopping strategies (#4525)
• Fix dependent parameter handling in Log transform (#4679)
• Drop NaN values in MAP_KEY column before align_partial_results (#4634)
• Filter failed trials from plots (#4725)
• Allow single progression early stopping checks when patience > 0 (#4635)
• Update SOBOL transition criterion to exclude ABANDONED and FAILED trials (#4776)

Other

• Speed up MapDataReplayMetric (#4654)
• Fast MapData.df implementation (#4487)
• Validate patience <= min_progression in PercentileEarlyStoppingStrategy (#4639)
• Enforce smoothing in [0, 1) for TensorBoardMetric (#4661)
• Enforce sort_values=True for numeric ordered ChoiceParameter (#4597)
• Add error if PowerTransformY is used with ScalarizedObjective (#4622)
• Support ScalarizedObjective in get_best_parameters with model predictions (#4594)
• Rename model_kwargs -> generator_kwargs (#4668)
• Rename model_gen_kwargs -> generator_gen_kwargs (#4667)
• Rename model_cv_kwargs -> cv_kwargs (#4669)

Ax 1.2.1

Bug fixes

• Improved error messaging for client.compute_analyses when certain analyses are
  not yet available (#4441)
• Fix tooltip mismatch bug in ArmEffectsPlot (#4479)

Other changes

• Bumped pinned botorch version to 0.16.1 (#4570)
• Removed deprecated robust optimization functionality (#4493)
• Allow HierarchicalSearchSpace to be constructed with multiple root nodes (#4560)

Ax 1.2.0

New features

• DerivedParameterConfig allows users to specify parameters which are not tuned,
  instead taking the value of some expression of other tunable parameters (#4454)
• New argument simplify_parameter_changes in client.configure_generation_strategy
  (defaulted to False) which when True informs Ax to change as few parameters as
  possible relative to the status quo without degrading optimization performance. Has
  a near-zero effect on generation time (#4409)
• Default to qLogNParEgo acquisition function for multi-objective optimization in
  multi-objective optimization when the number of objectives is > 4, leading to
  improved walltime performance (#4347)

Bug fixes

• Fix issue during candidate generation involving MapMetrics providing progressions
  at different scales i.e. one progression goes up to 10^9 and the other goes up to
  10^6 by normalizing to [0, 1] (#4458)

Other changes

• Improve visual clarity in ArmEffectsPlot by removing certain elements including
  red "infeasibility" halos and optional cumulative best line (#4397, #4398)
• Instructions on citing Ax included in README.md and ax.dev (#4317, #4357)
• New "Using external methods for candidate generation in Ax" tutorial on website (#4298)

Ax 1.1.2

Bug fixes

• Fixed rendering issue in ArmEffectsPlot when the number of arms displayed is greater than 20 (#4273)

Other changes

• Enabled Winsorization transform by default, improving surrogate performance in the presence of outliers (#4277)

Ax 1.1.1

Bug fixes

• Correctly filter out observations from Abandoned trials/arms during candidate
  generation (#4155)
• Handle scalarized objectives in ResultsAnalysis (#4193)
• Fix bug in polytope sampler when generating more than one candidate in a batch (#4244)

Other changes

• Transition from setup.py to pyproject.toml for builds, modernizing Ax's build
  configuration and bringing it in compliance with PEP 518 and PEP 621 (#4100)
• Add py.typed file, which allows typecheckers like Pyre, mypy, etc. to see Ax's types
  and avoid a TypeStub error when importing Ax (#4139)
• Improve legibility of ArmEffectPlot by modifying legend and x-axis labels (#4220,
  #4243)
• Address logspew in OneHotEncoder transform (#4232)

Ax 1.1.0

New Features

• New option for the method parameter in client.configure_generation_strategy:
  quality -- allows uers to indicate they would like Ax to generate the highest
  quality candidates it is able to at the expense of slower runtime (#4042)
• New logic for deciding which analyses to produce by default in
  client.compute_analyses (#4013)
• New parameters in client.summarize allow users to filter their summary by trial
  index and/or trial status (#4012, #4118)

Bug Fixes

• Allow client.summarize to be called without a GenerationStrategy being set
  (i.e. before client.configure_generation_strategy or client.get_next_trails
  has been called.) (#3801)
• Fixed incorrect grouping in TopSurfacesAnalysis (#4095)
• Fixed ContourPlot failing to compute in certain search spaces with parameter
  constraints (#4124)
• Misc. plotting fixes and improvements

Other changes

• Bumped pinned botorch version to 0.15.1
• Performance improvements in SensitivityAnalysis (#3891)
• Improved optimization performance in constrained optimization settings (#3585)
• Augmented logging in Client, early stopping module (#4044, #4108)

Ax 1.0.0

We released Ax in 2019 along with Botorch, and both libraries have evolved significantly since their initial release. Today, we are excited to release Ax version 1.0.0.

Along with typical new features, improvements, and bug fixes this release introduces a new public API (found in ax.api) which allows users to access all fully supported Ax features via a singular entry point. Classes and functions from Ax entry points (“Loop API, “Service API”, and “Developer API”) remain present in the library, but are considered deprecated and may be changed backwards incompatibly (or removed) in subsequent major or minor releases of Ax.

This release also comes with a full rewrite of our documentation which can be found at ax.dev

Highlights

• New unified API can be found in ax.api module. The classes and functions here constitute the Ax public API
  • The Client class manages state throughout the optimization process
  • Various Config classes allow the user to configure their search space, the optimization’s goals, etc.
• Complete documentation overhaul
  • An entirely rewritten https://ax.dev/ introduces users to important concepts, demonstrates usage on synthetic and real-world optimization problems, and thoroughly documents the API

Breaking changes

Now that we have defined a singular Ax API, users should make efforts to use this as an entry point into the library. This will give users assurance that the method signatures they use will remain stable between major versions of Ax, and allow our team to provide better support via Github Issues.

This will be straightforward for those previously using the “Service API”, which closely resembles the current API – users of the “Loop API” should look towards the Quickstart Guide (https://ax.dev/docs/next/tutorials/quickstart/) for an example which demonstrates using the new API for the simple optimizations the Loop API previously served, and “Developer API” users should also use the full documentation to understand how the new API can serve their needs.

Maintenance Release, New Website, Improved Performance in Mixed Search Spaces

Compatibility

• Require BoTorch 0.13.0

Highlights

• AX website has been upgraded to utilize Docusaurus v3, with the API reference being hosted by ReadTheDocs. The tutorials now expose an option to open with Colab, for easy access to a runtime with modifiable tutorials. The old versions of the website can be found at archive.ax.dev.
• Updated the default transforms and optimizers used in MBM-based models, to offer improved optimization performance in mixed discrete search spaces.
• Implemented a preview version of Ax 1.0 API (ax/preview). The new API is available for use & testing, but remains subject to change until the 1.0 release.
• Introduced new Analysis cards (ax/analysis) for analyzing experiment progress, results and model quality; which will replace various plotting and reporting utilities with a unified API.
• Deprecated a number of legacy models, including GPEI, in favor of the modular BoTorch model (MBM) API.
• Refactored Ax benchmark APIs to simplify problem setup and to support additional use cases such as noisy & asynchronous benchmarking.
• Added EBAshr model, which is used for repeating arms in online experiments as well as regression analysis in the health checks (#3276).
• New advanced capabilities:
  • MBM now supports per-metric model specification and model selection.
  • Support for new transition criteria and input constructors in GenerationNodes.

V0.4.3 - Maintenance Release

Key things:

• Update to BoTorch 0.12.0 – which included some major updates to underlying modeling, please see BoTorch release for detailed description
• Fix for a sql storage bug related to an import trying to import a deleted file

Compatibility

• BoTorch 0.12.0 -- which included some major updates to underlying modeling, please see BoTorch release for detailed description
• pin numpy to < 2.0 until tensorboard cuts a release (#2641)
• Make SQLAlchemy==1.4.17 a required dependency (#2690)

New Features

• AnalysisCard refactor (#2589)
• New parallel coordinates plot (#2590)
• Add an optimal_value to mixed-integer problems (#2596)
• Add AutoTransitionAfterGenCriterion to storage (#2614)
• Storage support for RBFKernel & LogNormalPrior (#2616)
• Add storage support for SobolQMCNormalSampler (#2622)
• AuxiliaryExperiment (#2632)
• Add auxiliary_experiments to Experiment (#2634)
• Create Scheduler.generate_candidates() function (#2640)
• AnalysisCard load/save methods (#2645)
• Healthcheck analysis class (#2646)
• add auxiliary experiments to SQAExperiment (#2658)
• Add Scheduler.compute_analyses method (#2660)
• Change time limit for unit tests to 60 seconds and introduce a decorator for long running tests (#2661)
• Add utility methods to Analysis (#2663)
• Introduce ParamBasedTestProblem for benchmarking (#2675)
• Run ModelLauncher with a BenchmarkRunner and BenchmarkMetric (#2681)
• Add init_position argument to UniformGenerator (#2686)
• Run Torchvision problems with Benchmark[Problem/Runner/Metric]; consolidate PyTorchCNN problems (#2688)
• allow more candidate trials than max_trials in Scheduler (#2689)
• create utility for trying to load GS (#2694)
• Implement Data/MapData.clone and use it in Experiment.clone_with (#2703)
• Compute ground-truth optimization trace on BenchmarkProblem (#2704)
• Add evaluate_oracle method to BenchmarkRunner (#2705)
• Merge create_single_objective_problem_from_botorch and create_multi_objective_problem_from_botorch; support constrained MOO (#2722)
• Add an option to override ModelSpec.model_key (#2726)
• Update test_generation_strategy to utilize MBM models (#2734)
• Refactor computation of optimization trace (#2747)
• Update the remaining models to use new default covar & likelihood modules (#2742)
• Add qLogNParEGO to ACQUISITION_FUNCTION_REGISTRY (#2749)
• Transform batch to new sq (#2755)

Bug Fixes

• Disable long test timeouts when using a debugger (#2745)
• Require all problems to have an optimal_value; simplify Problem inheritance (#2601)
• Update the default SingleTaskGP prior (#2610)
• Remove reference_point from multi-objective benchmark problems (#2619)
• More precise type annotations for MultiObjectiveOptimizationConfig (#2620)
• Change blob type to str (#2624)
• Revert D60472381 (#2627)
• Check for Nans and Infs in TensorboardMetric (#2628)
• Add warning to SerializationMixin, to prevent usage that will result in circular deps (#2629)
• Use Standardize by default for SingleTaskGP (#2630)
• Modify get_data to error out on nan/inf (#2633)
• fix bug in UnitX when transforming new search space (#2639)
• Add _maybe_clear_multiplexer method to deal with memory leaks (#2642)
• Fix Surrogate.best_out_of_sample_point (#2652)
• Fix clear_multiplexer (#2656)
• Fix Pyre errors in sqa_classes (#2659)
• Relax type hints on compute() (#2662)
• Fix type error in validate_sphinx script (#2665)
• Make sure random seed persists beyond storage (#2671)
• Log an exception rather than erroring out when data is found for metrics that are not attached to the experiment (#2680)
• fix input args for target-aware model in TLMB (#2685)
• Stop calling float() on single element Series (#2691)
• Serialize data classes based on their fields only (#2697)
• [Website] Use mathjax svg renderer (#2707)
• Take out error about HSS not being supported with batch trials (#2709)
• Make sure is_ax_equal returns a boolean (#2714)
• Remove internal type shed for plotly (#2715)
• Bring back ax/benchmark/init.py (#2719)
• BUG FIX: Only construct surrogate once in SurrogateRunner (#2720)
• Mock out downloading data from internet in torchvision unit test (#2725)
• robustify PL test (#2729)
• Require unique model_key in GenerationNode, clean up model selection errors (#2730)
• Add longtest decorator to pairwise model tests (#2731)
• Ensure new arm names do not match a different name on the experiment (#2732)
• Ignore fixed parameters for search space dimensionality computation (#2765)
• In ‘best_point’ don’t require GeneratorRun to have best_arm_predictions to predict from model (#2767)
• Fix Scheduler test_run_trials_and_yield_results_with_early_stopper that was timing out the CI (#2770)
• Make Acquisition.optimize work with discrete optimizer regardless of whether raw_samples is in optimizer_options (#2772)

Other Changes

• Document setup_import_mocks (#2736)
• Rename maximum_hypervolume to optimal_value in MOO Benchmark Problems (#2598)
• Make all benchmark problems take an observe_noise_stds argument (#2600)
• Move lazy construction of a surrogate from problem to runner (#2603)
• Update BestModelSelector to return a ModelSpec rather than the index (#2609)
• Add STATUSES_EXPECTING_DATA to trials consts (#2621)
• Drop critertion from AutoTransitionAfterGenCriterion name (#2625)
• Expanded testing get_pbo_experiment testing stub (#2631)
• Use reduced state for axolotl (#2637)
• AnalysisCard encoder/decoder refactor (#2643)
• SQAAnalysisCard refactor (#2644)
• codemod pep585_imports (#2648)
• Add Sphinx doc to Healthcheck analysis (#2649)
• Pass botorch_model_class to Surrogate._set_formatted_inputs (#2653)
• [Move model input constructor helper functions off of Surrogate, to standalone functions](htt...

V0.4.2 - Maintenance Release

Key things:

• Update to BoTorch 0.12.0 – which included some major updates to underlying modeling, please see BoTorch release for detailed description
• Fix for a sql storage bug related to an import trying to import a deleted file

Compatibility

• BoTorch 0.12.0 -- which included some major updates to underlying modeling, please see BoTorch release for detailed description
• pin numpy to < 2.0 until tensorboard cuts a release (#2641)
• Make SQLAlchemy==1.4.17 a required dependency (#2690)

New Features

• AnalysisCard refactor (#2589)
• New parallel coordinates plot (#2590)
• Add an optimal_value to mixed-integer problems (#2596)
• Add AutoTransitionAfterGenCriterion to storage (#2614)
• Storage support for RBFKernel & LogNormalPrior (#2616)
• Add storage support for SobolQMCNormalSampler (#2622)
• AuxiliaryExperiment (#2632)
• Add auxiliary_experiments to Experiment (#2634)
• Create Scheduler.generate_candidates() function (#2640)
• AnalysisCard load/save methods (#2645)
• Healthcheck analysis class (#2646)
• add auxiliary experiments to SQAExperiment (#2658)
• Add Scheduler.compute_analyses method (#2660)
• Change time limit for unit tests to 60 seconds and introduce a decorator for long running tests (#2661)
• Add utility methods to Analysis (#2663)
• Introduce ParamBasedTestProblem for benchmarking (#2675)
• Run ModelLauncher with a BenchmarkRunner and BenchmarkMetric (#2681)
• Add init_position argument to UniformGenerator (#2686)
• Run Torchvision problems with Benchmark[Problem/Runner/Metric]; consolidate PyTorchCNN problems (#2688)
• allow more candidate trials than max_trials in Scheduler (#2689)
• create utility for trying to load GS (#2694)
• Implement Data/MapData.clone and use it in Experiment.clone_with (#2703)
• Compute ground-truth optimization trace on BenchmarkProblem (#2704)
• Add evaluate_oracle method to BenchmarkRunner (#2705)
• Merge create_single_objective_problem_from_botorch and create_multi_objective_problem_from_botorch; support constrained MOO (#2722)
• Add an option to override ModelSpec.model_key (#2726)
• Update test_generation_strategy to utilize MBM models (#2734)
• Refactor computation of optimization trace (#2747)
• Update the remaining models to use new default covar & likelihood modules (#2742)
• Add qLogNParEGO to ACQUISITION_FUNCTION_REGISTRY (#2749)
• Transform batch to new sq (#2755)

Bug Fixes

• Disable long test timeouts when using a debugger (#2745)
• Require all problems to have an optimal_value; simplify Problem inheritance (#2601)
• Update the default SingleTaskGP prior (#2610)
• Remove reference_point from multi-objective benchmark problems (#2619)
• More precise type annotations for MultiObjectiveOptimizationConfig (#2620)
• Change blob type to str (#2624)
• Revert D60472381 (#2627)
• Check for Nans and Infs in TensorboardMetric (#2628)
• Add warning to SerializationMixin, to prevent usage that will result in circular deps (#2629)
• Use Standardize by default for SingleTaskGP (#2630)
• Modify get_data to error out on nan/inf (#2633)
• fix bug in UnitX when transforming new search space (#2639)
• Add _maybe_clear_multiplexer method to deal with memory leaks (#2642)
• Fix Surrogate.best_out_of_sample_point (#2652)
• Fix clear_multiplexer (#2656)
• Fix Pyre errors in sqa_classes (#2659)
• Relax type hints on compute() (#2662)
• Fix type error in validate_sphinx script (#2665)
• Make sure random seed persists beyond storage (#2671)
• Log an exception rather than erroring out when data is found for metrics that are not attached to the experiment (#2680)
• fix input args for target-aware model in TLMB (#2685)
• Stop calling float() on single element Series (#2691)
• Serialize data classes based on their fields only (#2697)
• [Website] Use mathjax svg renderer (#2707)
• Take out error about HSS not being supported with batch trials (#2709)
• Make sure is_ax_equal returns a boolean (#2714)
• Remove internal type shed for plotly (#2715)
• Bring back ax/benchmark/init.py (#2719)
• BUG FIX: Only construct surrogate once in SurrogateRunner (#2720)
• Mock out downloading data from internet in torchvision unit test (#2725)
• robustify PL test (#2729)
• Require unique model_key in GenerationNode, clean up model selection errors (#2730)
• Add longtest decorator to pairwise model tests (#2731)
• Ensure new arm names do not match a different name on the experiment (#2732)
• Ignore fixed parameters for search space dimensionality computation (#2765)
• In ‘best_point’ don’t require GeneratorRun to have best_arm_predictions to predict from model (#2767)
• Fix Scheduler test_run_trials_and_yield_results_with_early_stopper that was timing out the CI (#2770)
• Make Acquisition.optimize work with discrete optimizer regardless of whether raw_samples is in optimizer_options (#2772)

Other Changes

• Document setup_import_mocks (#2736)
• Rename maximum_hypervolume to optimal_value in MOO Benchmark Problems (#2598)
• Make all benchmark problems take an observe_noise_stds argument (#2600)
• Move lazy construction of a surrogate from problem to runner (#2603)
• Update BestModelSelector to return a ModelSpec rather than the index (#2609)
• Add STATUSES_EXPECTING_DATA to trials consts (#2621)
• Drop critertion from AutoTransitionAfterGenCriterion name (#2625)
• Expanded testing get_pbo_experiment testing stub (#2631)
• Use reduced state for axolotl (#2637)
• AnalysisCard encoder/decoder refactor (#2643)
• SQAAnalysisCard refactor (#2644)
• codemod pep585_imports (#2648)
• Add Sphinx doc to Healthcheck analysis (#2649)
• Pass botorch_model_class to Surrogate._set_formatted_inputs (#2653)
• [Move model input constructor helper functions off of Surrogate, to standalone functions](h...