Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

conda: avoid py3.x-conda cross-contamination #38266

Merged
merged 1 commit into from
Dec 23, 2024

Conversation

dannf
Copy link
Contributor

@dannf dannf commented Dec 23, 2024

Anything in the top-level directory of this package will end up getting installed into each version of the python module. We already use the prevent-inclusion parameter to prevent those files from being copied in. Now that we're using "cleanup" as a holding area, we need to include that as well.

This was causing the usr/bin/* files from all previously built modules to be included in subsequent ones:

  # apk info -L py3.12-conda | grep cleanup
  WARNING: opening /work/packages: No such file or directory
  usr/lib/python3.12/site-packages/cleanup/3.10/bin/conda
  usr/lib/python3.12/site-packages/cleanup/3.11/bin/conda
  usr/lib/python3.12/site-packages/melange-out/py3.11-conda/usr/lib/python3.11/site-packages/cleanup/3.10/bin/conda

And, because melange adds dependencies for paths in the shebangs, we were getting dependencies on additional python versions.

Anything in the top-level directory of this package will end up getting
installed into each version of the python module. We already use the
`prevent-inclusion` parameter to prevent those files from being copied
in. Now that we're using "cleanup" as a holding area, we need to include
that as well.

This was causing the usr/bin/* files from all previously built modules
to be included in subsequent ones:

  # apk info -L py3.12-conda | grep cleanup
  WARNING: opening /work/packages: No such file or directory
  usr/lib/python3.12/site-packages/cleanup/3.10/bin/conda
  usr/lib/python3.12/site-packages/cleanup/3.11/bin/conda
  usr/lib/python3.12/site-packages/melange-out/py3.11-conda/usr/lib/python3.11/site-packages/cleanup/3.10/bin/conda

And, because melange adds dependencies for paths in the shebangs,
we were getting dependencies on additional python versions.

Signed-off-by: dann frazier <dann.frazier@chainguard.dev>
@dannf dannf added approved-to-run A repo member has approved this external contribution eng:os labels Dec 23, 2024
@dannf dannf self-assigned this Dec 23, 2024
@octo-sts octo-sts bot added the bincapz/pass bincapz/pass Bincapz (aka. malcontent) scan didn't detect any CRITICALs on the scanned packages. label Dec 23, 2024
@dannf dannf enabled auto-merge December 23, 2024 20:07
@dannf dannf merged commit 3c3419f into wolfi-dev:main Dec 23, 2024
14 checks passed
- runs: |
# We don't use this dir until further down, but py/pip-build-install
# will fail if a parameter to its `prevent-inclusion:` does not exist
mkdir -p ./cleanup
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we not modify py/pip-build-install rather than potentially modifying multiple packages in the same way?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had considered it a feature that this pipeline failed when given bogus input because, in most cases, it seems like an absent path would be due to a mistake. What if instead we detect this case and emit a warning? Something like this not-yet-tested change:

diff --git a/pipelines/py/pip-build-install.yaml b/pipelines/py/pip-build-install.yaml
index 8cfa9d5a5..6b6a4bf01 100644
--- a/pipelines/py/pip-build-install.yaml
+++ b/pipelines/py/pip-build-install.yaml
@@ -92,13 +92,23 @@ pipeline:
         fi
       fi
 
+      add_prevent() {
+        if [ -e "$1" ]; then
+          ( vr tar -Apf "$2" && vr rm -rf "$1" ) ||
+             { echo "ERROR: failed adding $1 to prevent-inclusion.tar"; exit 1; }
+          echo "prevented-inclusion of $1"
+          return
+        fi
+        echo "WARNING: prevents-inclusion path $1 not found, ignoring."
+      }
+
+      pitar="$tmpd/prevent-inclusion.tar"
       prevents="${{inputs.prevent-inclusion}}"
       if [ -n "$prevents" ]; then
+         # initialize empty tar file
+         tar -cf "$pitar" -T /dev/null
          # do not allow expansion of prevents
-         ( set -f; vr tar -cpf "$tmpd/prevent-inclusion.tar" $prevents &&
-           vr rm -rf $prevents ) ||
-           { echo "ERROR: failed creation of prevent-inclusion.tar with $prevents"; exit 1; }
-         echo "prevented-inclusion of $prevents"
+         ( set -f; for p in $prevents; do add_prevent "$p" "$pitar"; done )
       fi
 
       [ -d build ] && hadbuild=true || hadbuild=false

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved-to-run A repo member has approved this external contribution bincapz/pass bincapz/pass Bincapz (aka. malcontent) scan didn't detect any CRITICALs on the scanned packages. eng:os
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants