feat: persist requirements.txt as a build artifact

[jenstroeger]

I think it makes sense to add the generated requirements.txt file as a permanent build artifact. While it duplicates information in the SBOM, it’s also useful for downstream actions to use—think of, for example, a docker.yaml which is triggered on release and which produces a Python application image using the pinned, generated requirements from the build.

[jenstroeger]

Actually… we probably want a requirements.txt different than the one used for the SBOM. By default the package is installed with all additional dependencies

python-package-template/Makefile

Lines 83 to 85 in c754175

	python -m pip install --upgrade pip
	python -m pip install --upgrade wheel
	python -m pip install --upgrade --upgrade-strategy eager --editable .[hooks,dev,test,docs]

but we do not want these additional dependencies deployed to a production environment. The requirements.txt file should therefore be generated from

python -m pip install --upgrade --upgrade-strategy eager --editable .

and not contain the hooks, dev, test and docs extras.

[behnazh]

Actually… we probably want a requirements.txt different than the one used for the SBOM. By default the package is installed with all additional dependencies

Yes, agreed. SBOM will already container all the non-production dependencies. Maybe we should document this difference in the README.md file?

[jenstroeger]

I think this would work:

python -m pip uninstall --yes --requirement <(pip freeze --local)
python -m pip install --upgrade --upgrade-strategy eager pip wheel --editable .
$(MAKE) requirements

But we have a race-condition here, I think. If between building the SBOM and building this new requirements.txt any of the external packages change then we have diverging resource definitions 🤔

Should we build

pip install --editable .

first for the requirements, and then add the additional dependencies after

pip install --editable .[hooks,dev,test,docs]

which should use the existing packages. We may want to disable the --upgrade-strategy eager arg?

[jenstroeger]

Before I forget: I think we could take a look at the --no-index option:

1. install all packages and generate the SBOM (or as @behnazh calls it, iBOM, infrastructure BOM because it contains tests and checks and docs); then
2. clean out the installed packages (see above); then
3. install the production dependencies only and disable the PyPI index, thus forcing only already-downloaded packages to be installed.

Considering that the production dependencies are a subset of the all-inclusive dependencies, step 3 ought to install the same packages as for step 1. Worth exploring, methinks 🤔

[jenstroeger]

I added a prune goal to the Makefile which allows the user to prune the installed packages down to the required dependencies only, without the development dependencies. The point is to prune the packages down to a shipping production venv.

Then things get tricky, mainly because we still depend on the hashin tool until there’s a resolution to issue pip freeze with a hash (#4732).

Basically, we want to generate a requirements.txt file for the current venv which includes hashes for every package. So…

• Freeze the venv as-is and then add the hashin command back in to fetch package hashes;
• Finally, add our own package to the requirements.txt^* and, if available, generate hashes for it from the distribution assets.

To be honest, the changes to the requirements goal in the Makefile are ugly and I’m tempted to hoist them out of the Makefile into the build.yaml… 🤔

—————
^* The current implementation does not contain the package itself in the requirements.txt and therefore it’s also missing from the SBOM declaration.

[jenstroeger]

Hmm, I think we should consider replacing hashin with something like this:

for pkg in `python -m pip freeze --local --disable-pip-version-check --exclude-editable`; do
  echo -n $pkg;
  [[ $pkg =~ (.*)==(.*) ]] && curl -s https://pypi.org/pypi/${BASH_REMATCH[1]}/json | python -c "import json, sys; print(''.join(f''' \\\\\n  --hash=sha256:{pkg['digests']['sha256']}''' for pkg in json.load(sys.stdin)['releases']['${BASH_REMATCH[2]}']));";
done

which, given an arbitrary venv, produces

alabaster==0.7.12 \
  --hash=sha256:446438bdcca0e05bd45ea2de1668c1d9b032e1a9154c2c259092d77031ddd359 \
  --hash=sha256:a661d72d58e6ea8a57f7a86e37d86716863ee5e92788398526d58b26a4e4dc02
argcomplete==1.12.3 \
  --hash=sha256:291f0beca7fd49ce285d2f10e4c1c77e9460cf823eef2de54df0c0fec88b0d81 \
  --hash=sha256:2c7dbffd8c045ea534921e63b0be6fe65e88599990d8dc408ac8c542b72a5445
...
coverage==6.4.4 \
  --hash=sha256:e7b4da9bafad21ea45a714d3ea6f3e1679099e420c8741c74905b92ee9bfa7cc \
  --hash=sha256:fde17bc42e0716c94bf19d92e4c9f5a00c5feb401f5bc01101fdf2a8b7cacf60 \
  --hash=sha256:cdbb0d89923c80dbd435b9cf8bba0ff55585a3cdb28cbec65f376c041472c60d \
  --hash=sha256:67f9346aeebea54e845d29b487eb38ec95f2ecf3558a3cffb26ee3f0dcc3e760 \
  --hash=sha256:42c499c14efd858b98c4e03595bf914089b98400d30789511577aa44607a1b74 \
  ...
  --hash=sha256:e16c45b726acb780e1e6f88b286d3c10b3914ab03438f32117c4aa52d7f30d58
cyclonedx-bom==3.5.0 \
  --hash=sha256:da06ff3388ec0ffa874cf6bf1fe4da79ebc89e7cb69202bb1c0fb7744208fcb9 \
  --hash=sha256:ff12e69e3c79d41df203336eef0a29376c3bf8bb8fdd254222cbd788d09bd906
...

So this change works beautifully without any additional packages:

diff --git a/Makefile b/Makefile
index 6e8c9a3..7e42191 100644
--- a/Makefile
+++ b/Makefile
@@ -113,9 +113,10 @@ sbom: requirements
 requirements: requirements.txt
 requirements.txt: pyproject.toml
        echo -n "" > requirements.txt
-       REQUIREMENTS=`python -m pip freeze --local --disable-pip-version-check --exclude-editable`; \
-       python -m pip install hashin; \
-       for pkg in $$REQUIREMENTS; do hashin --verbose --algorithm sha256 --include-prereleases $$pkg; done
+       for pkg in `python -m pip freeze --local --disable-pip-version-check --exclude-editable`; do \
+         echo -n $$pkg >> requirements.txt; \
+         [[ $$pkg =~ (.*)==(.*) ]] && curl -s https://pypi.org/pypi/$${BASH_REMATCH[1]}/json | python -c "import json, sys; print(''.join(f''' \\\\\n    --hash=sha256:{pkg['digests']['sha256']}''' for pkg in json.load(sys.stdin)['releases']['$${BASH_REMATCH[2]}']));" >> requirements.txt; \
+       done
        echo -e -n "package==$(PACKAGE_VERSION)" >> requirements.txt
        if [ -f dist/package-$(PACKAGE_VERSION).tar.gz ]; then \
          echo -e -n " \\\\\n    `pip hash --algorithm sha256 dist/package-$(PACKAGE_VERSION).tar.gz | grep '^\-\-hash'`" >> requirements.txt; \

[behnazh]

So this change works beautifully without any additional packages:

That's neat! I think it's worth to give it a go. We can change it back if it faces issues.

[jenstroeger]

That's neat! I think it's worth to give it a go. We can change it back if it faces issues.

Commit 3792ace.

[behnazh]

It would be an interesting challenge to use jq instead of Python to process the JSON file returned by the API. But that can be explored later.