Skip to content

Commit

Permalink
Use minimum versions for safety run on dependencies for install
Browse files Browse the repository at this point in the history 
Details:

* Changed safety run for install dependencies to use the exact minimum versions
  of the dependent packages, by moving them into a separate
  minimum-constraints-install.txt file that is included by the existing
  minimum-constraints.txt file. (issue #939)

Signed-off-by: Andreas Maier <maiera@de.ibm.com>
  • Loading branch information
@andy-maier
andy-maier committed Mar 20, 2024
1 parent 9faaa8e commit 06edfb5
Show file tree
Hide file tree
Showing 6 changed files with 205 additions and 110 deletions.
4 changes: 3 additions & 1 deletion .safety-policy-all.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ security:
42559:
reason: Fixed Pip version requires Python>=3.6 and is used there; Pip is not shipped with this package
42923:
reason: Fixed Ansible version is sated to be 0 (invalid issue)
reason: Fixed Ansible version is stated to be 0
42926:
reason: Fixed Ansible version is 3.0 but we need to support Ansible 2.9
43975:
Expand Down Expand Up @@ -124,6 +124,8 @@ security:
reason: Fixed ansible-core version 2.15.8 requires Python>=3.9 and is used on Python>=3.12
64227:
reason: Fixed Jinja2 version 3.1.3 requires Python>=3.7 and is used on Python>=3.8
65278:
reason: Fixed cryptography version 42.0.0 requires Python>=3.7 and is used there

# Continue with exit code 0 when vulnerabilities are found.
continue-on-vulnerability-error: False
80 changes: 79 additions & 1 deletion .safety-policy-install.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,88 @@ security:
# reason: {text} # optional: Reason for ignoring it. Will be reported in the Safety reports
# expires: {date} # optional: Date when this ignore will expire
ignore-vulnerabilities:
38932:
reason: Fixed cryptography version requires Python>=3.6 and is used there
39252:
reason: Fixed cryptography version 3.3 requires Python=2.7 or Python>=3.6 and is used there
39525:
reason: Fixed Jinja2 version is not used by Ansible sanity test
39606:
reason: Fixed cryptography version requires Python==2.7 or Python>=3.6 and is used there
39611:
reason: PyYAML full_load method or FullLoader is not used
40291:
reason: Fixed Pip version requires Python>=3.6 and is used there
42559:
reason: Fixed Pip version requires Python>=3.6 and is used there; Pip is not shipped with this package
42923:
reason: Fixed Ansible version is stated to be 0
43975:
reason: Fixed Urllib3 versions are excluded by requests
51499:
reason: Fixed Wheel version requires Python>=3.7 and is used there; Risk is on Pypi side
52495:
reason: Fixed Setuptools version requires Python>=3.7 and is used there; Risk is on Pypi side
53048:
reason: Fixed cryptography version requires Python>=3.6 and is used there
53298:
reason: Fixed cryptography version 39.0.1 requires Python>=3.7 (fails on 3.6) and is used there
53299:
reason: Fixed cryptography version 39.0.1 requires Python>=3.7 (fails on 3.6) and is used there
53301:
reason: Fixed cryptography version 39.0.1 requires Python>=3.7 (fails on 3.6) and is used there
53302:
reason: Fixed cryptography version 39.0.1 requires Python>=3.7 (fails on 3.6) and is used there
53303:
reason: Fixed cryptography version 39.0.1 requires Python>=3.7 (fails on 3.6) and is used there
53304:
reason: Fixed cryptography version 39.0.1 requires Python>=3.7 (fails on 3.6) and is used there
53305:
reason: Fixed cryptography version 39.0.1 requires Python>=3.7 (fails on 3.6) and is used there
53306:
reason: Fixed cryptography version 39.0.1 requires Python>=3.7 (fails on 3.6) and is used there
53307:
reason: Fixed cryptography version 39.0.1 requires Python>=3.7 (fails on 3.6) and is used there
54219:
reason: Fixed ansible version is 2.10.0, but need to support ansible 2.9
54229:
reason: Not yet fixed in any ansible version, but need to support ansible 2.9
54230:
reason: Not yet fixed in any ansible version, but need to support ansible 2.9
54564:
reason: Fixed ansible version is 7.0.0, but need to support ansible 2.9
58755:
reason: Fixed requests version 2.31.0 requires Python>=3.7 and is used there
59062:
reason: Fixed cryptography version 41.0.0 requires Python>=3.7 and is used there
59473:
reason: Fixed cryptography version 41.0.2 requires Python>=3.7 and is used there
59956:
reason: Fixed certifi version 2023.07.22 requires Python>=3.6 and is used there
60223:
reason: Fixed cryptography version 41.0.3 requires Python>=3.7 and is used there
60224:
reason: Fixed cryptography version 41.0.3 requires Python>=3.7 and is used there
60225:
reason: Fixed cryptography version 41.0.3 requires Python>=3.7 and is used there
61601:
reason: Fixed urllib3 version 1.26.17 requires Python>=3.6 and is used there
61893:
reason: Fixed urllib3 version 1.26.18 requires Python>=3.6 and is used there
62044:
reason: Fixed pip version 23.3 requires Python>=3.7 and is used there
62451:
reason: Fixed cryptography version 41.0.4 requires Python>=3.7 and is used there
62452:
reason: Fixed cryptography version 41.0.5 requires Python>=3.7 and is used there
62556:
reason: Fixed cryptography version 41.0.6 requires Python>=3.7 and is used there
63066:
reason: Fixed ansible-core version 2.15.8 requires Python>=3.9 and is used on Python>=3.12
64227:
reason: Fixed Jinja2 version 3.1.3 requires Python>=3.7 and is used on Python>=3.8
65278:
reason: Fixed cryptography version 42.0.0 requires Python>=3.7 and is used there

# Continue with exit code 0 when vulnerabilities are found.
continue-on-vulnerability-error: False
continue-on-vulnerability-error: False
12 changes: 6 additions & 6 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,7 @@ help:
@echo ' check - Run flake8'
@echo ' sanity - Run Ansible sanity tests (includes pep8, pylint, validate-modules)'
@echo ' ansible_lint - Run ansible-lint on distribution archive (and built it)'
@echo ' safety - Run safety on sources'
@echo ' safety - Run safety for install and all'
@echo ' check_reqs - Perform missing dependency checks'
@echo ' docs - Build the documentation for all enabled (docs/source/conf.py) versions in: $(doc_build_dir) using remote repo'
@echo ' docslocal - Build the documentation from local repo contents in: $(doc_build_local_dir)'
Expand Down Expand Up @@ -355,7 +355,7 @@ endif
@echo '$@ done.'

.PHONY: check_reqs
check_reqs: _check_version $(done_dir)/develop_$(pymn)_$(PACKAGE_LEVEL).done minimum-constraints.txt requirements.txt
check_reqs: _check_version $(done_dir)/develop_$(pymn)_$(PACKAGE_LEVEL).done minimum-constraints.txt minimum-constraints-install.txt requirements.txt
ifeq ($(python_major_version),2)
@echo "Makefile: Warning: Skipping the checking of missing dependencies on Python 2.x" >&2
else
Expand All @@ -364,7 +364,7 @@ ifeq ($(PACKAGE_LEVEL),ansible)
else
@echo "Makefile: Checking missing dependencies of this package"
pip-missing-reqs $(src_py_dir) --requirements-file=requirements.txt
pip-missing-reqs $(src_py_dir) --requirements-file=minimum-constraints.txt
pip-missing-reqs $(src_py_dir) --requirements-file=minimum-constraints-install.txt
@echo "Makefile: Done checking missing dependencies of this package"
@echo "Makefile: Checking missing dependencies of some development packages"
@rc=0; for pkg in $(check_reqs_packages); do dir=$$($(PYTHON_CMD) -c "import $${pkg} as m,os; dm=os.path.dirname(m.__file__); d=dm if not dm.endswith('site-packages') else m.__file__; print(d)"); cmd="pip-missing-reqs $${dir} --requirements-file=minimum-constraints.txt"; echo $${cmd}; $${cmd}; rc=$$(expr $${rc} + $${?}); done; exit $${rc}
Expand Down Expand Up @@ -471,7 +471,7 @@ $(done_dir)/install_pip_$(pymn)_$(PACKAGE_LEVEL).done: Makefile
$(PYTHON_CMD) -m pip install $(pip_level_opts) pip setuptools wheel
echo "done" >$@

$(done_dir)/safety_all_$(pymn)_$(PACKAGE_LEVEL).done: $(done_dir)/develop_$(pymn)_$(PACKAGE_LEVEL).done Makefile $(safety_all_policy_file) minimum-constraints.txt
$(done_dir)/safety_all_$(pymn)_$(PACKAGE_LEVEL).done: $(done_dir)/develop_$(pymn)_$(PACKAGE_LEVEL).done Makefile $(safety_all_policy_file) minimum-constraints.txt minimum-constraints-install.txt
ifeq ($(python_major_version),2)
@echo "Makefile: Warning: Skipping Safety for all packages on Python $(python_version)" >&2
else
Expand All @@ -486,7 +486,7 @@ else
endif
endif

$(done_dir)/safety_install_$(pymn)_$(PACKAGE_LEVEL).done: $(done_dir)/develop_$(pymn)_$(PACKAGE_LEVEL).done Makefile $(safety_install_policy_file) requirements.txt
$(done_dir)/safety_install_$(pymn)_$(PACKAGE_LEVEL).done: $(done_dir)/develop_$(pymn)_$(PACKAGE_LEVEL).done Makefile $(safety_install_policy_file) minimum-constraints-install.txt
ifeq ($(python_major_version),2)
@echo "Makefile: Warning: Skipping Safety for install packages on Python $(python_version)" >&2
else
Expand All @@ -495,7 +495,7 @@ ifeq ($(python_m_n_version),3.5)
else
@echo "Makefile: Running Safety for install packages"
-$(call RM_FUNC,$@)
safety check --policy-file $(safety_install_policy_file) -r requirements.txt --full-report
safety check --policy-file $(safety_install_policy_file) -r minimum-constraints-install.txt --full-report
echo "done" >$@
@echo "Makefile: Done running Safety for install packages"
endif
Expand Down
5 changes: 5 additions & 0 deletions docs/source/release_notes.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,11 @@ Availability: `AutomationHub`_, `Galaxy`_, `GitHub`_
* Added a new make target 'end2end_show' to show the HMCs defined for end2end
tests. (issue #888)

* Changed safety run for install dependencies to use the exact minimum versions
of the dependent packages, by moving them into a separate
minimum-constraints-install.txt file that is included by the existing
minimum-constraints.txt file. (issue #939)

**Cleanup:**

* Increased versions of GitHub Actions plugins to increase node.js runtime
Expand Down
106 changes: 106 additions & 0 deletions minimum-constraints-install.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Pip constraints file for install dependencies.
#
# This file specifies constraints that define the minimum versions of any
# direct and indirect dependent packages for installation and development of
# the project.
# They are used when setting up test environments with the minimum versions of
# any dependent packages.


# Dependencies for installation with Pip (must be installed in a separate pip call)

# For the base packages, we use the versions from Ubuntu 18.04 as a general
# minimum, and then increase it to the first version that introduced support
# for a particular Python version:
# pip 10.0.0 introduced the --exclude-editable option.
# pip 18.0 is needed on pypy3 (py36) to support constraints like cffi!=1.11.3,>=1.8.
# Pip 20.2 introduced a new resolver whose backtracking had issues that were resolved only in 21.2.2.
# Pip 21.0 removed support for Python<=3.5
# pip>=21.0 is needed for the cryptography package on Windows on GitHub Actions.
pip==19.3.1; python_version <= '3.5'
pip==21.3.1; python_version == '3.6'
pip==23.3; python_version >= '3.7'

# setuptools 51.0.0 removed support for py35
# setuptools 59.7.0 removed support for py36
# setuptools 68.1.0 removed support for py37
# setuptools 65.5.1 fixes safety issue 52495. However, using that version causes
# the ansible sanity test for Ansible 2.9 to fail in the pylint step due to
# this Python warning being displayed on stderr:
# _distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils
# See Ansible issue 71602 which describes that setuptools 50.0.0 introduced
# that issue and that it was supposedly fixed in setuptools 50.1.0.
# Therefore, we stay on an older version of setuptools for Ansible 2.9
# (i.e. Python 2.7)
# The warning reoccurred since then under some (unknown) circumstances.
# setuptools>=66.1.0 is required on Python 3.12.
setuptools==39.0.1; python_version == '2.7'
setuptools==50.3.2; python_version == '3.5'
setuptools==59.6.0; python_version >= '3.6' and python_version <= '3.11'
setuptools==66.1.0; python_version >= '3.12'

wheel==0.30.0; python_version <= '3.6'
wheel==0.38.1; python_version >= '3.7'


# Direct dependencies for install (must be consistent with requirements.txt)

# Keep consistent with table for 'minimum' packages in development.rst and with requirements.txt
ansible==2.9.27; python_version == '2.7' # sanity test is not run here
ansible==2.9.27; python_version >= '3.5' and python_version <= '3.7'
ansible==2.9.27; python_version == '3.8'
ansible==4.1.0; python_version == '3.9'
ansible==5.0.1; python_version == '3.10'
ansible==7.0.0; python_version == '3.11'
ansible==9.0.1; python_version >= '3.12'

# ansible-core is pulled in by ansible>=4.0, and is needed for pip-check-reqs checks.
ansible-core==2.11.12; python_version == '3.9'
ansible-core==2.12.10; python_version == '3.10'
ansible-core==2.14.2; python_version == '3.11'
ansible-core==2.16.0; python_version >= '3.12'

# other ansible or ansible-core dependencies
resolvelib==0.5.3; python_version == '3.9'
resolvelib==0.5.3; python_version == '3.10'
resolvelib==0.5.3; python_version >= '3.11'
distlib==0.3.6

requests==2.25.0; python_version <= '3.6'
requests==2.31.0; python_version >= '3.7'

pytz==2016.10; python_version <= '3.9'
pytz==2019.1; python_version >= '3.10'

zhmcclient==1.13.3


# Indirect dependencies for install (must be consistent with requirements.txt)

# Jinja2 is used by ansible/ansible-core and by ansible-doc-extractor.
Jinja2==2.10.1; python_version <= '3.7'
Jinja2==3.1.3; python_version >= '3.8'

# MarkupSafe is used by Jinja2
MarkupSafe==1.1.0; python_version <= '3.7'
MarkupSafe==2.0.0; python_version >= '3.8'

cryptography==3.3.2; python_version == '2.7'
cryptography==3.0; python_version == '3.5'
cryptography==3.4.7; python_version == '3.6'
cryptography==42.0.2; python_version >= '3.7'

importlib-metadata==0.12; python_version <= '3.7'
importlib-metadata==4.8.3; python_version >= '3.8'

packaging==20.5; python_version <= '3.5'
packaging==21.3; python_version >= '3.6'
PyYAML==5.3.1; python_version <= '3.5'
PyYAML==6.0.1; python_version >= '3.6'

python-dateutil==2.8.2
jsonschema==3.0.1; python_version <= '3.6'
jsonschema==4.10.0; python_version >= '3.7'
urllib3==1.26.18; python_version == '2.7'
urllib3==1.26.9; python_version == '3.5'
urllib3==1.26.18; python_version >= '3.6'
108 changes: 6 additions & 102 deletions minimum-constraints.txt
View file
Original file line number Diff line number Diff line change
@@ -1,108 +1,12 @@
# Pip constraints file.
# Pip constraints file for install and development.
#
# This file specifies constraints that define the minimum versions of any
# direct and indirect dependent packages for installation and development of
# the project.
# They are used when setting up test environments with the minimum versions of
# any dependent packages.
# This constraints file specifies constraints that match the minimum versions
# specified in the requirements files for install and development. The reason
# for this approach is that in the CI systems, we want to be able to test with
# the minimum package versions in order to catch any incorrect minimum versions.

# Dependencies for installation with Pip (must be installed in a separate pip call)

# For the base packages, we use the versions from Ubuntu 18.04 as a general
# minimum, and then increase it to the first version that introduced support
# for a particular Python version:
# pip 10.0.0 introduced the --exclude-editable option.
# pip 18.0 is needed on pypy3 (py36) to support constraints like cffi!=1.11.3,>=1.8.
# Pip 20.2 introduced a new resolver whose backtracking had issues that were resolved only in 21.2.2.
# Pip 21.0 removed support for Python<=3.5
# pip>=21.0 is needed for the cryptography package on Windows on GitHub Actions.
pip==19.3.1; python_version <= '3.5'
pip==21.3.1; python_version == '3.6'
pip==23.3; python_version >= '3.7'

# setuptools 51.0.0 removed support for py35
# setuptools 59.7.0 removed support for py36
# setuptools 68.1.0 removed support for py37
# setuptools 65.5.1 fixes safety issue 52495. However, using that version causes
# the ansible sanity test for Ansible 2.9 to fail in the pylint step due to
# this Python warning being displayed on stderr:
# _distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils
# See Ansible issue 71602 which describes that setuptools 50.0.0 introduced
# that issue and that it was supposedly fixed in setuptools 50.1.0.
# Therefore, we stay on an older version of setuptools for Ansible 2.9
# (i.e. Python 2.7)
# The warning reoccurred since then under some (unknown) circumstances.
# setuptools>=66.1.0 is required on Python 3.12.
setuptools==39.0.1; python_version == '2.7'
setuptools==50.3.2; python_version == '3.5'
setuptools==59.6.0; python_version >= '3.6' and python_version <= '3.11'
setuptools==66.1.0; python_version >= '3.12'

wheel==0.30.0; python_version <= '3.6'
wheel==0.38.1; python_version >= '3.7'


# Direct dependencies for installation (must be consistent with requirements.txt)

# Keep consistent with table for 'minimum' packages in development.rst and with requirements.txt
ansible==2.9.27; python_version == '2.7' # sanity test is not run here
ansible==2.9.27; python_version >= '3.5' and python_version <= '3.7'
ansible==2.9.27; python_version == '3.8'
ansible==4.1.0; python_version == '3.9'
ansible==5.0.1; python_version == '3.10'
ansible==7.0.0; python_version == '3.11'
ansible==9.0.1; python_version >= '3.12'

# ansible-core is pulled in by ansible>=4.0, and is needed for pip-check-reqs checks.
ansible-core==2.11.12; python_version == '3.9'
ansible-core==2.12.10; python_version == '3.10'
ansible-core==2.14.2; python_version == '3.11'
ansible-core==2.16.0; python_version >= '3.12'

# other ansible or ansible-core dependencies
resolvelib==0.5.3; python_version == '3.9'
resolvelib==0.5.3; python_version == '3.10'
resolvelib==0.5.3; python_version >= '3.11'
distlib==0.3.6

requests==2.25.0; python_version <= '3.6'
requests==2.31.0; python_version >= '3.7'

pytz==2016.10; python_version <= '3.9'
pytz==2019.1; python_version >= '3.10'

zhmcclient==1.13.3


# Indirect dependencies for installation (must be consistent with requirements.txt)

# Jinja2 is used by ansible/ansible-core and by ansible-doc-extractor.
Jinja2==2.10.1; python_version <= '3.7'
Jinja2==3.1.3; python_version >= '3.8'

# MarkupSafe is used by Jinja2
MarkupSafe==1.1.0; python_version <= '3.7'
MarkupSafe==2.0.0; python_version >= '3.8'

cryptography==3.3.2; python_version == '2.7'
cryptography==3.0; python_version == '3.5'
cryptography==3.4.7; python_version == '3.6'
cryptography==42.0.2; python_version >= '3.7'

importlib-metadata==0.12; python_version <= '3.7'
importlib-metadata==4.8.3; python_version >= '3.8'

packaging==20.5; python_version <= '3.5'
packaging==21.3; python_version >= '3.6'
PyYAML==5.3.1; python_version <= '3.5'
PyYAML==6.0.1; python_version >= '3.6'

python-dateutil==2.8.2
jsonschema==3.0.1; python_version <= '3.6'
jsonschema==4.10.0; python_version >= '3.7'
urllib3==1.26.18; python_version == '2.7'
urllib3==1.26.9; python_version == '3.5'
urllib3==1.26.18; python_version >= '3.6'
-c minimum-constraints-install.txt


# Direct dependencies for development (must be consistent with dev-requirements.txt)
Expand Down

0 comments on commit 06edfb5

Please sign in to comment.