Save Suggestion state after deployment is deleted

[andreyvelich]

/kind feature

Katib should have functionality to save Suggestion state somewhere besides Suggestion pod.
Some users would like to resume Experiments, but they don't want to have always running Suggestion deployment. For example we can use PV.

We can use ResumeExperiment flag from here: #1061 to specify resuming experiment mechanism.

/cc @johnugeorge @gaocegege @hougangliu @richardsliu

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.89

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[andreyvelich]

/priority p0

[andreyvelich]

@johnugeorge @gaocegege What do you think about saving Suggestion states using object serialisation, e.g pickle in Python: https://docs.python.org/3/library/pickle.html.
I believe Fairing also uses pickle to serialise training code.

In that case, if ResumeExperiment: Always, after each GetSuggestion call, Suggestion can create pickle file with the current object state. Each suggestion has <AlgorithmNameService> class. For example:

katib/pkg/suggestion/v1alpha3/hyperopt/service.py

Line 15 in 58e0764

class HyperoptService(api_pb2_grpc.SuggestionServicer, HealthServicer):

.

Then, we can save this pickle file in PV and use it to resume experiment with appropriate state for the Suggestion service class

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/katib	0.78

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[gaocegege]

The approach LGTM if the suggestion is implemented in Python. But I prefer a general solution for different languages.