Add check_safe parameter to Early-Stopping Strategies
#4579
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Nov 25, 2025
Summary: This diff adds a `check_safe` boolean parameter to `BaseEarlyStoppingStrategy` and all its child classes to control whether the `_is_harmful` safety check is applied when making early stopping decisions. When `check_safe=False` (default), the safety check is bypassed and early stopping decisions from `_should_stop_trials_early` are applied directly. When `check_safe=True`, the `_is_harmful` check gates early stopping to prevent potentially harmful stopping decisions. The parameter is added to: - `BaseEarlyStoppingStrategy.__init__` - `ModelBasedEarlyStoppingStrategy.__init__` - `PercentileEarlyStoppingStrategy.__init__` - `ThresholdEarlyStoppingStrategy.__init__` All child classes default to `check_safe=False` to maintain backward compatibility while allowing opt in to safety checks as needed. Differential Revision: D87492602
ltiao
force-pushed
the
export-D87492602
branch
from
017f62f to
dba8eea
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4579 +/- ##
==========================================
- Coverage 96.50% 96.49% -0.02%
==========================================
Files 557 557
Lines 57359 57387 +28
==========================================
+ Hits 55356 55377 +21
- Misses 2003 2010 +7 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Nov 26, 2025
Summary: This diff adds a `check_safe` boolean parameter to `BaseEarlyStoppingStrategy` and all its child classes to control whether the `_is_harmful` safety check is applied when making early stopping decisions. When `check_safe=False` (default), the safety check is bypassed and early stopping decisions from `_should_stop_trials_early` are applied directly. When `check_safe=True`, the `_is_harmful` check gates early stopping to prevent potentially harmful stopping decisions. The parameter is added to: - `BaseEarlyStoppingStrategy.__init__` - `ModelBasedEarlyStoppingStrategy.__init__` - `PercentileEarlyStoppingStrategy.__init__` - `ThresholdEarlyStoppingStrategy.__init__` All child classes default to `check_safe=False` to maintain backward compatibility while allowing opt in to safety checks as needed. Differential Revision: D87492602
… Naming (facebook#4578) Summary: **Context:** This diff refactors the early stopping strategies codebase to eliminate code duplication and improve code clarity through better naming conventions. - Eliminated ~50 lines of duplicate data preparation logic - Established single source of truth for data preparation in base class - Improved code maintainability and consistency - No functional changes - purely refactoring **Changes:** 1. **Moved `_prepare_aligned_data()` to base class** (`BaseEarlyStoppingStrategy`) * Previously duplicated in `PercentileEarlyStoppingStrategy` and `MultiObjectiveEarlyStoppingStrategy` (and also in future concrete implements of `_is_harmful`). Now a reusable helper method available to all strategies 2. **Renamed method for clarity** * `_check_validity_and_get_data()` → `_lookup_and_validate_data()` (emphasizes "lookup" terminology consistent with Ax codebase conventions; more accurately describes the method's purpose) 3. **Improved parameter naming across all strategies** * `df`, `df_raw` → `wide_df`, `long_df` * Clearly distinguishes between wide and long format dataframes * Updated in all early stopping strategy implementations and tests 4. **Updated documentation** Reviewed By: saitcakmak Differential Revision: D87573286
Summary: This diff adds a `check_safe` boolean parameter to `BaseEarlyStoppingStrategy` and all its child classes to control whether the `_is_harmful` safety check is applied when making early stopping decisions. When `check_safe=False` (default), the safety check is bypassed and early stopping decisions from `_should_stop_trials_early` are applied directly. When `check_safe=True`, the `_is_harmful` check gates early stopping to prevent potentially harmful stopping decisions. The parameter is added to: - `BaseEarlyStoppingStrategy.__init__` - `ModelBasedEarlyStoppingStrategy.__init__` - `PercentileEarlyStoppingStrategy.__init__` - `ThresholdEarlyStoppingStrategy.__init__` All child classes default to `check_safe=False` to maintain backward compatibility while allowing opt in to safety checks as needed. Reviewed By: saitcakmak Differential Revision: D87492602
ltiao
force-pushed
the
export-D87492602
branch
from
dba8eea to
0696738
Compare
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Nov 26, 2025
Summary: This diff adds a `check_safe` boolean parameter to `BaseEarlyStoppingStrategy` and all its child classes to control whether the `_is_harmful` safety check is applied when making early stopping decisions. When `check_safe=False` (default), the safety check is bypassed and early stopping decisions from `_should_stop_trials_early` are applied directly. When `check_safe=True`, the `_is_harmful` check gates early stopping to prevent potentially harmful stopping decisions. The parameter is added to: - `BaseEarlyStoppingStrategy.__init__` - `ModelBasedEarlyStoppingStrategy.__init__` - `PercentileEarlyStoppingStrategy.__init__` - `ThresholdEarlyStoppingStrategy.__init__` All child classes default to `check_safe=False` to maintain backward compatibility while allowing opt in to safety checks as needed. Reviewed By: saitcakmak Differential Revision: D87492602
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Nov 26, 2025
Summary: This diff adds a `check_safe` boolean parameter to `BaseEarlyStoppingStrategy` and all its child classes to control whether the `_is_harmful` safety check is applied when making early stopping decisions. When `check_safe=False` (default), the safety check is bypassed and early stopping decisions from `_should_stop_trials_early` are applied directly. When `check_safe=True`, the `_is_harmful` check gates early stopping to prevent potentially harmful stopping decisions. The parameter is added to: - `BaseEarlyStoppingStrategy.__init__` - `ModelBasedEarlyStoppingStrategy.__init__` - `PercentileEarlyStoppingStrategy.__init__` - `ThresholdEarlyStoppingStrategy.__init__` All child classes default to `check_safe=False` to maintain backward compatibility while allowing opt in to safety checks as needed. Reviewed By: saitcakmak Differential Revision: D87492602
|
This pull request has been merged in c66160c. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This diff adds a
check_safeboolean parameter toBaseEarlyStoppingStrategyand all its child classes to control whether the_is_harmfulsafety check is applied when making early stopping decisions.When
check_safe=False(default), the safety check is bypassed and early stopping decisions from_should_stop_trials_earlyare applied directly. Whencheck_safe=True, the_is_harmfulcheck gates early stopping to prevent potentially harmful stopping decisions.The parameter is added to:
BaseEarlyStoppingStrategy.__init__ModelBasedEarlyStoppingStrategy.__init__PercentileEarlyStoppingStrategy.__init__ThresholdEarlyStoppingStrategy.__init__All child classes default to
check_safe=Falseto maintain backward compatibility while allowing opt in to safety checks as needed.Differential Revision: D87492602