Support multi-objective optimization

[anencore94]

/kind feature

Describe the solution you'd like

• katib currently use only single-objective optimization, (even though katib could display additional metrics, the optimizing process focused on single objective function)

• However, as you might know well, optuna support multi-objective optimization

  • which means, for example, we can optimize the hyperparameters to maximize accuracy and minimize time-cost simultaneously.

• In such case, I know we can define a new objective function which combines two objectives, such as maximize accuracy / time-cost or something else more complicated.

• However, I wonder is there any plan to support such feature natively in katib.

[andreyvelich]

Thank you for creating this @anencore94.
I think that feature will require some API modification since type, goal and objectiveMetricName can be unique: https://github.com/kubeflow/katib/blob/master/pkg/apis/controller/common/v1beta1/common_types.go#L94-L103.

@anencore94 @gaocegege @johnugeorge Any ideas how we can implement this ?

@c-bata Does Goptuna support Multi-object optimization ?

[c-bata]

@c-bata Does Goptuna support Multi-object optimization ?

No. I'm not planning to implement multi-objective support in Goptuna because it requires some breaking changes.

As @anencore94 said, I think Optuna is a good choice if Katib will support multi-objective optimizations.
Optuna supports a lot of multi-objective optimization algorithms like NSGA-II, multi-objective TPE, and the algorithms provided by BoTorch. And it is not difficult to implement Optuna based suggestion service since its internal design is almost similar with Goptuna.

[g-votte]

Hi, I'm one of the maintainers of Optuna. Thank you for referring to Optuna as a possible backend of the multi-objective API.

As @c-bata mentions, Optuna supports several algorithms for multi-objective optimization, with almost the same interface as single objective optimization.

I think we can go forward by (1) implementing the Optuna suggestion service, and (2) fixing the interface of Katib's multi-objective functionality.

For (1), I created the PR #1613 to add the suggestion service. I appreciate your taking a look and providing any feedback.

[anencore94]

Can we close this issue by #1613 ?

[andreyvelich]

Can we close this issue by #1613 ?

This issue still is progress. If we want to support multi-objective optimisation, we have to modify our Katib Experiment APIs.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.