Migrate build repo code from using Python 3.7 to 3.9 version #3351

peterzhuamazon · 2023-03-28T18:15:58Z

Migrate build repo code from using Python 3.7 to 3.9 version.

Python3.7 support will ends on 2023/06/23.
https://endoflife.date/python

Note: 3.9 Ends in 2 years and 5 months (05 Oct 2025)

We need to migrate the code to Python 3.9 for the highest stability possible.

Acceptance Criteria:

All the build code should be using 3.9
Moving from 3.7 to 3.9 in the build repo, updating the pipfile and lock file.
Test all the existing workflows with the new python version.
Update all the CI images to use Python 3.9.
Testing the compilation of the cpython with centos docker image.

Thanks.

The text was updated successfully, but these errors were encountered:

peterzhuamazon · 2023-03-29T19:30:42Z

The version 3.11 is currently feature adding version.
It is not stable enough for the production use.
@bbarani I would recommend using version 3.10 if we really want to go that high, but in my personal experience the most stable one is 3.9 version after 3.8.

Thanks.

rishabh6788 · 2023-06-12T18:23:50Z

[TRIAGE] This is pre-req for #1563.
We also need to upgrade pip from v21 to v23 to accommodate python upgrade.
Tagging @peterzhuamazon.

peterzhuamazon · 2023-06-20T17:52:20Z

It might benefit to change the default emulation buildx to cross compilation:

https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

PRs:

[Python 3.9 Upgrade] Docker CI Images update 3.7 to 3.9 (Part 1) #3634
[Python 3.9 Upgrade] Change docker build jenkins workflow to use the v2 docker-builder images #3641
[Python 3.9 Upgrade] Upgrade docker buildx 0.9.1 and fix Python compilation issues #3643
[Python 3.9 Upgrade] Replace Python 3.9.17 to 3.9.7 to resolve compilation inconsistency on CentOS7 #3650
[Python 3.9 Upgrade] OpenSearch-Build Repo Python Migration to 3.9 Version #3658
[Python 3.9 Upgrade] Add the test docker images upgrades to Python 3.9 (Part 2) #3680
[Python 3.9 Upgrade] Update macos and windows packer agent with Python 3.9 opensearch-ci#311
[Python 3.9 Upgrade] Resolve python 3.9 and cmake installation on Windows opensearch-ci#312
[Python 3.9 Upgrade] Lock Windows Python to 3.9.7 and cmake to 3.21.3 (Part 2) opensearch-ci#313
[Python 3.9 Upgrade] Restore python39 back to latest version due to old json not supported on scoop for 3.9.7 anymore opensearch-ci#314
[Python 3.9 Upgrade] Update Runner Ami IDs for Python 3.9 Upgrade opensearch-ci#316
[Python 3.9 Upgrade] [NodeJS Upgrade] Add AL2023 agent and switch perftest and benchmarktest from AL2 to AL2023 with Python 3.9 opensearch-ci#317
[Python 3.9 Upgrade] [NodeJS Upgrade] Switch the internal ami name from al2 to al2023 opensearch-ci#319
[Python 3.9 Upgrade] [NodeJS Upgrade] Replace ami id for al2023 with agents in #319 opensearch-ci#320
[Python 3.9 Upgrade] [NodeJS Upgrade] Update perftest and benchmark test runner to AL2023 #3711
[Python 3.9 Upgrade] [NodeJS Upgrade] Update developer guide for Python 3.9 Upgrade and Node Version Upgrade #3714
[Python 3.9 Upgrade] Update macos agent to Python 3.9 export and add M58xlarge gradle check runner opensearch-ci#323
Switch validation workflow for docker to run on AL2023 and Gradle Check to new M58xlarge runners #3810
Switch single host from AL2 to AL2023 opensearch-ci#332
Add AL2023 Docker Host Runner and Make M58xlarge Gradle Check Host opensearch-ci#330
Update promote artifact jenkins workflow to use Python3.9 image #3804

peterzhuamazon · 2023-06-22T19:51:28Z

Able to get the pip packages freeze version and didnt change behaviors on all the isort/flake8/mypy/pytest results.

% pipenv run isort --check .
Skipped 24 files

% pipenv run flake8 .

% pipenv run mypy .
Success: no issues found in 375 source files

Able to build/assemble artifacts for OS and OSD.

% curl https://localhost:9200/ -ku admin:admin
{
  "name" : "dev-dsk-zhujiaxi-2a-c8d595b4.us-west-2.amazon.com",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "gxjxiKG0Qbu3mdKhHAQqMw",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.8.0",
    "build_type" : "tar",
    "build_hash" : "db90a415ff2fd428b4f7b3f800a51dc229287cb4",
    "build_date" : "2023-06-22T17:07:14.480104Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

{"overall":{"since":"2023-06-22T18:46:13.818Z","state":"green","title":"Green","nickname":"Looking good","icon":"success","uiColor":"secondary"}

Able to run integTest on OS and OSD:

2023-06-22 19:37:52 INFO     | geospatial           | with-security        | PASS  |
2023-06-22 19:37:52 INFO     | geospatial           | without-security     | PASS  |

2023-06-22 20:29:31 INFO     | alertingDashboards   | with-security        | PASS  |
2023-06-22 20:29:31 INFO     | alertingDashboards   | without-security     | PASS  |

peterzhuamazon · 2023-06-22T19:52:33Z

% pipenv run python --version
Python 3.9.7

Installing dependencies from Pipfile.lock (9f5fbd)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

% pipenv run pip freeze
atomicwrites==1.4.1
attrs==23.1.0
Cerberus==1.3.4
certifi==2023.5.7
cfgv==3.3.1
chardet==5.1.0
charset-normalizer==2.1.1
click==8.1.3
coverage==4.5.4
DataProperty==0.55.1
decorator==5.1.1
distlib==0.3.6
distro==1.8.0
exceptiongroup==1.1.1
filelock==3.12.2
flake8==3.9.2
identify==2.5.24
idna==3.4
importlib-metadata==4.12.0
iniconfig==2.0.0
isort==5.12.0
jproperties==2.1.1
maison==1.4.0
mbstrdecoder==1.1.3
mccabe==0.6.1
mypy==0.971
mypy-extensions==1.0.0
nodeenv==1.8.0
packaging==23.1
pathspec==0.11.1
pathvalidate==2.5.2
platformdirs==3.7.0
pluggy==1.2.0
pre-commit==2.15.0
psutil==5.9.5
py==1.11.0
pycodestyle==2.7.0
pydantic==1.10.9
pyflakes==2.3.1
pytablewriter==0.64.2
pytest==7.1.3
pytest-cov==2.10.1
python-dateutil==2.8.2
pytz==2023.3
PyYAML==5.4.1
requests==2.28.1
retry==0.9.2
ruamel.yaml==0.17.32
ruamel.yaml.clib==0.2.7
ruyaml==0.91.0
six==1.16.0
sortedcontainers==2.4.0
tabledata==1.3.1
tcolorpy==0.1.3
toml==0.10.2
tomli==2.0.1
typed-ast==1.5.4
typepy==1.3.1
types-PyYAML==5.4.12
types-requests==2.31.0.1
types-urllib3==1.26.25.13
typing_extensions==4.6.3
urllib3==1.26.16
validators==0.20.0
virtualenv==20.23.1
yamlfix==1.0.1
yamllint==1.27.1
zipp==3.8.1

peterzhuamazon · 2023-06-29T20:46:25Z

Windows Python3.9 Build and Assemble Runs now:

$ python3
Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.release()
'10'
>>> import os
>>> os.name
'nt'
>>>

$ curl https://localhost:9200 -ku admin:admin
{
  "name" : "EC2AMAZ-D5JNNTJ",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "OCLI5GC0SKCA5CRnarNnTw",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.8.0",
    "build_type" : "zip",
    "build_hash" : "db90a415ff2fd428b4f7b3f800a51dc229287cb4",
    "build_date" : "2023-06-29T20:03:07.832804700Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

$ curl localhost:5601/api/status -ku admin:admin
{"name":"EC2AMAZ-D5JNNTJ","uuid":"d2d0dc8a-ae48-444e-9a3b-9d79351bce0c","version":{"number":"2.8.0","build_hash":"8bd48f16ad37a5dfa805234223e
4d5bffa926abe","build_number":1,"build_snapshot":false},"status":{"overall":{"since":"2023-06-29T22:14:23.028Z","state":"green","title":"Gree
n","nickname":"Looking good","icon":"success","uiColor":"secondary"},"statuses":[{"id":"core:opensearch@2.8.0","message":"OpenSearch is avail
able","since":"2023-06-29T22:14:23.028Z","state":"green","icon":"success","uiColor":"secondary"}......

peterzhuamazon · 2023-06-30T20:53:55Z

Also just fyi once we go to 3.9 all client release that uses this image release-centos7-clients-v3 needs to be upgraded to release-centos7-clients-v4.

peterzhuamazon · 2023-07-05T23:30:39Z

We are starting the Python 3.9 Upgrade now, as Jenkins instance will be unstable from now on.
Start time: 2023/07/05 4:29 PM PST.
ETA: 3hrs

An email to the team has been sent out last week.
A msg to the public slack channel was also sent.

Thanks.

peterzhuamazon · 2023-07-06T00:47:22Z

Test deployment staging:

+ ./build.sh manifests/2.9.0/opensearch-2.9.0.yml -d tar --component OpenSearch -p linux -a x64

Installing dependencies in . ...

Warning: the environment variable LANG is not set!

We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.

Creating a virtualenv for this project...

Pipfile: /var/jenkins/workspace/Python39-test-distribution-build-opensearch-zhujiaxi@2/Pipfile

Using /usr/local/bin/python3.9 (3.9.7) to create virtualenv...

created virtual environment CPython3.9.7.final.0-64 in 483ms

  creator CPython3Posix(dest=/usr/share/opensearch/.local/share/virtualenvs/Python39-test-distribution-build-opensearc-QLML_HWD, clear=False, no_vcs_ignore=False, global=False)

  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/usr/share/opensearch/.local/share/virtualenv)

    added seed packages: pip==23.1.2, setuptools==67.8.0, wheel==0.40.0

  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

peterzhuamazon · 2023-07-06T02:58:28Z

Verified all build and assemble on all platforms are running well.

peterzhuamazon · 2023-07-06T03:10:39Z

All docker actions are running ok with no issues.

peterzhuamazon · 2023-07-06T03:10:58Z

Will send to production and check gradle check.

gaiksaya · 2023-07-06T06:09:03Z

@peterzhuamazon We need to update developer guide too to use python 3.9 https://github.com/opensearch-project/opensearch-build/blob/main/DEVELOPER_GUIDE.md#python-37

peterzhuamazon · 2023-07-06T18:13:06Z

Jenkins deploy success on production host and the migration deployment is complete.

gaiksaya · 2023-07-07T18:21:08Z

Adding meta task to update all 1-click release process: #3712

bbarani · 2023-07-21T23:54:58Z

@peterzhuamazon Can we close this issue now?

peterzhuamazon · 2023-07-25T18:43:09Z

Yes @bbarani we have just completed all the changes now, thanks.

peterzhuamazon added the enhancement New Enhancement label Mar 28, 2023

github-actions bot added the untriaged Issues that have not yet been triaged label Mar 28, 2023

Divyaasm removed the untriaged Issues that have not yet been triaged label Mar 28, 2023

Divyaasm changed the title ~~Migrate build repo code from using Python 3.7 to 3.8 version~~ Migrate build repo code from using Python 3.7 to 3.11 version Mar 28, 2023

prudhvigodithi changed the title ~~Migrate build repo code from using Python 3.7 to 3.11 version~~ Migrate build repo code from using Python 3.7 to 3.9 version May 1, 2023

peterzhuamazon mentioned this issue May 16, 2023

[Proposal] Support AL2023/NodeJS18 for OpenSearch/Dashboards Releases #1563

Closed

peterzhuamazon self-assigned this Jun 14, 2023

peterzhuamazon added release v2.9.0 python Pull requests that update Python code upgrade dependencies Pull requests that update a dependency file Build Libraries & Interfaces python upgrade Python Upgrade labels Jun 14, 2023

peterzhuamazon mentioned this issue Jun 14, 2023

[Python 3.9 Upgrade] Docker CI Images update 3.7 to 3.9 (Part 1) #3634

Merged

peterzhuamazon pinned this issue Jun 19, 2023

gaiksaya removed the release label Jun 19, 2023

peterzhuamazon mentioned this issue Jun 19, 2023

[Python 3.9 Upgrade] Change docker build jenkins workflow to use the v2 docker-builder images #3641

Merged

This was referenced Jun 20, 2023

[Python 3.9 Upgrade] Upgrade docker buildx 0.9.1 and fix Python compilation issues #3643

Merged

[Python 3.9 Upgrade] Replace Python 3.9.17 to 3.9.7 to resolve compilation inconsistency on CentOS7 #3650

Merged

peterzhuamazon mentioned this issue Jul 3, 2023

[Python 3.9 Upgrade] Update Runner Ami IDs for Python 3.9 Upgrade opensearch-project/opensearch-ci#316

Merged

gaiksaya mentioned this issue Jul 6, 2023

Change default docker image to support python 3.9 opensearch-project/opensearch-build-libraries#239

Merged

gaiksaya unpinned this issue Jul 25, 2023

peterzhuamazon closed this as completed Jul 25, 2023

peterzhuamazon mentioned this issue Jan 29, 2024

[Deprecation] Properly deprecate CentOS7 as CI build image/Supported OS and switch to Almalinux8 #4379

Open

peterzhuamazon mentioned this issue Sep 4, 2024

[Upgrade] Migrate build repo code from using Python 3.9 to 3.12 version #4994

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate build repo code from using Python 3.7 to 3.9 version #3351

Migrate build repo code from using Python 3.7 to 3.9 version #3351

peterzhuamazon commented Mar 28, 2023 •

edited by prudhvigodithi

Loading

peterzhuamazon commented Mar 29, 2023

rishabh6788 commented Jun 12, 2023

peterzhuamazon commented Jun 20, 2023 •

edited

Loading

peterzhuamazon commented Jun 22, 2023 •

edited

Loading

peterzhuamazon commented Jun 22, 2023

peterzhuamazon commented Jun 29, 2023 •

edited

Loading

peterzhuamazon commented Jun 30, 2023

peterzhuamazon commented Jul 5, 2023

peterzhuamazon commented Jul 6, 2023

peterzhuamazon commented Jul 6, 2023

peterzhuamazon commented Jul 6, 2023

peterzhuamazon commented Jul 6, 2023

gaiksaya commented Jul 6, 2023

peterzhuamazon commented Jul 6, 2023

gaiksaya commented Jul 7, 2023

bbarani commented Jul 21, 2023

peterzhuamazon commented Jul 25, 2023

Migrate build repo code from using Python 3.7 to 3.9 version #3351

Migrate build repo code from using Python 3.7 to 3.9 version #3351

Comments

peterzhuamazon commented Mar 28, 2023 • edited by prudhvigodithi Loading

Acceptance Criteria:

peterzhuamazon commented Mar 29, 2023

rishabh6788 commented Jun 12, 2023

peterzhuamazon commented Jun 20, 2023 • edited Loading

peterzhuamazon commented Jun 22, 2023 • edited Loading

peterzhuamazon commented Jun 22, 2023

peterzhuamazon commented Jun 29, 2023 • edited Loading

peterzhuamazon commented Jun 30, 2023

peterzhuamazon commented Jul 5, 2023

peterzhuamazon commented Jul 6, 2023

peterzhuamazon commented Jul 6, 2023

peterzhuamazon commented Jul 6, 2023

peterzhuamazon commented Jul 6, 2023

gaiksaya commented Jul 6, 2023

peterzhuamazon commented Jul 6, 2023

gaiksaya commented Jul 7, 2023

bbarani commented Jul 21, 2023

peterzhuamazon commented Jul 25, 2023

peterzhuamazon commented Mar 28, 2023 •

edited by prudhvigodithi

Loading

peterzhuamazon commented Jun 20, 2023 •

edited

Loading

peterzhuamazon commented Jun 22, 2023 •

edited

Loading

peterzhuamazon commented Jun 29, 2023 •

edited

Loading