Skip to content

Conversation

@fkiraly
Copy link
Collaborator

@fkiraly fkiraly commented Jul 8, 2025

Fixes #141 by the following approach:

  • adds a tag property:higher_or_lower_is_better to experiments to signifiy whether this is minmization or maximization
  • ensures the score method is always "higher is better"
  • moves the extension locus to a new method pair evaluate / _evaluate, which has the same orientation as the new tag property:higher_or_lower_is_better. The _score method no longer exists.

The SklearnCvExperiment also gets internal functionality to detect the sign from the metric passed. Since metrics in sklearn are not tagged properly, there is some clunky detection logic to infer this non-existent tag.

Further changes:

  • current experiments are adapted with correct tags property:higher_or_lower_is_better
  • current _score methods are changed to _evaluate
  • clarifications in extension templates
  • clarifications in docstrings

Decision to move to maximization, see discussion in #141.

@fkiraly fkiraly added the enhancement New feature or request label Jul 8, 2025
# if "deterministic", two calls of score must result in the same value
#
"property:higher_or_lower_is_better": "lower",
# valid values: "higher", "lower", "mixed"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is mixed? It is not handled by the cost method.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in theory there could be an experiment that is neither "lower is better" nor "higher is better" - an example would be a metric such as "empirical coverage of 90% nominal prediction intervals", where the best value is 90% (not 0% or 100%)

Copy link
Collaborator

@SimonBlanke SimonBlanke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update readme.md (and docstrings?). They still speak of "higher is better" or maximization in several places.

@fkiraly
Copy link
Collaborator Author

fkiraly commented Jul 9, 2025

Please update readme.md (and docstrings?). They still speak of "higher is better" or maximization in several places.

Can you point towards these?

Stupid question, also: what is the current consensus in gfo and the adapter methods? Maximize, or minimize?
(we should be consistent with that imo)

@SimonBlanke
Copy link
Collaborator

Can you point towards these?

Just search for higher/maximize in the readme. I take care of the api docs.

Stupid question, also: what is the current consensus in gfo and the adapter methods? Maximize, or minimize?

gfo maximizes the objective function in the current api. But I aim for support for minimize/maximize as well.

@fkiraly
Copy link
Collaborator Author

fkiraly commented Jul 21, 2025

ouch - I will then flip the sign as an action, is that what I should do? So it is consistently maximize?

What is your preferred action?

  • leave at minimize and change the docstrings
  • flip to maximize and leave docstrings

?

That way it would be consistent with the score terminology!

@SimonBlanke
Copy link
Collaborator

@fkiraly Let's flip it to maximize and leave the docstrings. I'll go through the code again, but we can probably merge after that.

@fkiraly
Copy link
Collaborator Author

fkiraly commented Jul 27, 2025

ok, that is also more consistent with score terminology then.

@fkiraly fkiraly requested a review from SimonBlanke July 27, 2025 15:47
@fkiraly fkiraly marked this pull request as ready for review July 27, 2025 15:47
@fkiraly
Copy link
Collaborator Author

fkiraly commented Jul 27, 2025

I think this is ready now - I have made the following changes:

  • score is now always the method to be maximized, i.e., in all experiments it has positive orientation (towards maximization).
  • the extension contract focuses now on the method pair evaluate / _evaluate. The orientation of this is expected to be the same as the tag property:higher_or_lower_is_better.
  • score is evaluate with sign flipped based on the tag property:higher_or_lower_is_better.

I think this is an elegant way to minimize the extension boilerplate for the optimizers, which now are assumed to always maximize.

@SimonBlanke SimonBlanke merged commit 18f8a0f into hyperactive-project:master Jul 28, 2025
76 of 78 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ENH] encoding minimization vs maximization

2 participants