Bump Python version to 3.9

[tenzen-y]

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

Python 3.6 is approaching EOL(2021-12-23) although all suggestions and earlystopping services are used it.
So, bump Python version to 3.9.

https://devguide.python.org/#status-of-python-branches

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

/assign

[tenzen-y]

Also, I think it needs to upgrade tensorflow since tensorflow 1.11.0 is not distributed binary for Python 3.7.

katib/cmd/metricscollector/v1beta1/tfevent-metricscollector/Dockerfile

Line 1 in 16e0574

FROM tensorflow/tensorflow:1.11.0

Ref: https://pypi.org/project/tensorflow/1.11.0/#files

[gaocegege]

SGTM. We need to upgrade suggestion, early stopping and metricscollector.

[tenzen-y]

I'm wondering if I should upgrade to v1.15.5, which is the last release of Tensorflow v1 or upgrade to the latest release, v2.7.0.
WDYT about the TensorFlow version? @kubeflow/wg-automl-leads

[andreyvelich]

Thank you for creating this @tenzen-y.

Should we jump to Python 3.9 if it is possible, do we have any concern about it ?

I'm wondering if I should upgrade to v1.15.5, which is the last release of Tensorflow v1 or upgrade to the latest release, v2.7.0.

I think TF >= 2.0 requires some code changes, especially in ENAS Suggestion.
If we can handle it, sure we can use the latest TF release.

[tenzen-y]

Thanks for the suggestion! @andreyvelich

Should we jump to Python 3.9 if it is possible, do we have any concern about it ?

I also think we can jump to Python 3.9.

I think TF >= 2.0 requires some code changes, especially in ENAS Suggestion.
If we can handle it, sure we can use the latest TF release.

I see, I would like to update to Tensorflow >= 2.0 and also modify ENAS.

[gaocegege]

Does TF 2.0 metrics collector work well with TF 1.x events?

[tenzen-y]

I believe that the output format of metrics has not changed between TF v1 and v2, as shown in the following document.

• v1:

events.out.tfevents.1613185647.kokoro-gcp-ubuntu-prod-2039871251.28609.1446.v2

• v2:

events.out.tfevents.1613185646.kokoro-gcp-ubuntu-prod-2039871251.28609.5.v2

https://www.tensorflow.org/tensorboard/migrate#in_tf_2x

WDYT? @gaocegege @andreyvelich

[andreyvelich]

Sounds good @tenzen-y.
Once we modify TF Event metrics collector, we need to be sure that it is working with TF 2.0

[tenzen-y]

Once we modify TF Event metrics collector, we need to be sure that it is working with TF 2.0

SGTM

I think it might be better to create a training code using TF 2.0 based on mnist_with_summaries.py, to add it as the e2e test separate from tfjob.

WDYT? @andreyvelich @gaocegege

[andreyvelich]

Sounds good to me, we can maintain another image under trial-images with TF with summaries example.
What do others think @gaocegege @johnugeorge ?

[tenzen-y]

we can maintain another image under trial-images with TF with summaries example.

Sounds good, thanks!