Skip to content

Add fpm to dev container image to avoid podman in docker in GitHub actions #2012

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ddelnano merged 7 commits into from
Sep 10, 2024
Merged

Add fpm to dev container image to avoid podman in docker in GitHub actions #2012

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7ce3517
Upgrade chef client to latest version
ddelnano Sep 7, 2024
c172f45
Install fpm in the dev docker image
ddelnano Sep 8, 2024
e8544c9
Revert "Use privileged container for running cli build release steps …
ddelnano Sep 9, 2024
901a387
Move packaging recipe to px_dev_extras
ddelnano Sep 9, 2024
996a82a
Update base image version file
ddelnano Sep 9, 2024
d2ba8fa
Remove genny to get a successful build on current go version and upda…
ddelnano Sep 10, 2024
20479f7
#ci:ignore-deps #ci:bpf-build-all-kernels
ddelnano Sep 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .github/workflows/cli_release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,6 @@ jobs:
needs: get-dev-image
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
# --privileged is needed in order for podman to work. Otherwise it fails
# to create a new namespace when the clone syscall happens.
options: --privileged
env:
ARTIFACT_UPLOAD_LOG: "artifact_uploads.json"
steps:
Expand Down
8 changes: 4 additions & 4 deletions docker.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
DOCKER_IMAGE_TAG=202405102250
LINTER_IMAGE_DIGEST=e9987c1c23d0d9c8443b3973f7f3482e5737b10053ce56f58fa59e4cb4f32291
DEV_IMAGE_DIGEST=7e38004cc30aa4f26b800ecad67c8e38fdffdfed8b3be0d3084e1706a60a4f21
DEV_IMAGE_WITH_EXTRAS_DIGEST=c405c9b28a8ed91785ad1115df146b80f275e88f93e4c5a516ad4bb7d4508112
DOCKER_IMAGE_TAG=202409092344
LINTER_IMAGE_DIGEST=18436897f607dd6b1926c12de13be7e9ed668a17cff133e4da90aad0112a406f
DEV_IMAGE_DIGEST=83eba02bf4ced64161276dee88dcd6f02de5951bb3ce5346a1182c70498156aa
DEV_IMAGE_WITH_EXTRAS_DIGEST=27ff4d9111b77516fdf73715ba6ea23d02de3b790d88f6f70a48c5e1ec53570c
2 changes: 1 addition & 1 deletion tools/chef/base_image_version.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
202405102217
202409092308
1 change: 0 additions & 1 deletion tools/chef/cookbooks/px_dev/recipes/golang.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@
execute 'install go binaries' do
ENV['GOPATH'] = "/opt/px_dev/gopath"
command %(go install github.com/golang/mock/mockgen@v1.5.0 && \
go install github.com/cheekybits/genny@v1.0.0 && \
Copy link
Member Author

@ddelnano ddelnano Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to install genny results in the following error:

$ go install github.com/cheekybits/genny@v1.0.0
go: finding module for package golang.org/x/tools/imports
go: toolchain upgrade needed to resolve golang.org/x/tools/imports
go: golang.org/x/tools@v0.25.0 requires go >= 1.22.0 (running go 1.21.0)

I wasn't able to get to the bottom of how this changed between the beginning on my PR and now or what dependency is causing this toolchain requirement.

Anyway, it appears that the last reference to genny was removed in 806ad3f. This opts to remove it instead of trying to work around this issue.

# Verify that genny is no longer used on that commit

ddelnano@dev-vm:~/code/pixie-worktree ((806ad3f67...)) $ git grep -i genny
go.sum:github.com/gobuffalo/genny v0.0.0-20190329151137-27723ad26ef9/go.mod h1:rWs4Z12d1Zbf19rlsn0nurr75KqhYp52EAGGxTbBhNk=
go.sum:github.com/gobuffalo/genny v0.0.0-20190403191548-3ca520ef0d9e/go.mod h1:80lIj3kVJWwOrXWWMRzzdhW3DsrdjILVil/SFKBzF28=
go.sum:github.com/gobuffalo/genny v0.1.0/go.mod h1:XidbUqzak3lHdS//TPu2OgiFB+51Ur5f7CSnXZ/JDvo=
go.sum:github.com/gobuffalo/genny v0.1.1 h1:iQ0D6SpNXIxu52WESsD+KoQ7af2e3nCfnSBoSF/hKe0=
go.sum:github.com/gobuffalo/genny v0.1.1/go.mod h1:5TExbEyY48pfunL4QSXxlDOmdsD44RRq4mVZ0Ex28Xk=
Binary file go_deps.bzl matches
tools/arc_addons/pixielabs/unit/ArcanistGoGenCheckerTestEngine.php:    'genny' => '/(?<=-out )(.*)(?=\.gen\.go)/',
tools/chef/cookbooks/pixielabs/recipes/base.rb:            github.com/cheekybits/genny@v1.0.0 \

# Go back a commit and see that XXX removed the final reference

ddelnano@dev-vm:~/code/pixie-worktree ((806ad3f67...)) $ git checkout HEAD~1
Previous HEAD position was 806ad3f67 Use informers for k8s metadata watcher
HEAD is now at b0d2c6c56 Add dataAccess flag to CLI

ddelnano@dev-vm:~/code/pixie-worktree ((b0d2c6c56...)) $ git grep -i genny
go.sum:github.com/gobuffalo/genny v0.0.0-20190329151137-27723ad26ef9/go.mod h1:rWs4Z12d1Zbf19rlsn0nurr75KqhYp52EAGGxTbBhNk=
go.sum:github.com/gobuffalo/genny v0.0.0-20190403191548-3ca520ef0d9e/go.mod h1:80lIj3kVJWwOrXWWMRzzdhW3DsrdjILVil/SFKBzF28=
go.sum:github.com/gobuffalo/genny v0.1.0/go.mod h1:XidbUqzak3lHdS//TPu2OgiFB+51Ur5f7CSnXZ/JDvo=
go.sum:github.com/gobuffalo/genny v0.1.1 h1:iQ0D6SpNXIxu52WESsD+KoQ7af2e3nCfnSBoSF/hKe0=
go.sum:github.com/gobuffalo/genny v0.1.1/go.mod h1:5TExbEyY48pfunL4QSXxlDOmdsD44RRq4mVZ0Ex28Xk=
Binary file go_deps.bzl matches
Binary file src/vizier/services/metadata/controllers/k8smeta/k8s_metadata_utils.gen.go matches
src/vizier/services/metadata/controllers/k8smeta/k8s_metadata_utils.go://go:generate genny -in=k8s_metadata_utils.tmpl -out k8s_metadata_utils.gen.go gen "ReplacedResource=Pod,Service,Namespace,Endpoints,Node"
src/vizier/services/metadata/controllers/k8smeta/k8s_metadata_utils.tmpl:       "github.com/cheekybits/genny/generic"
tools/arc_addons/pixielabs/unit/ArcanistGoGenCheckerTestEngine.php:    'genny' => '/(?<=-out )(.*)(?=\.gen\.go)/',
tools/chef/cookbooks/pixielabs/recipes/base.rb:            github.com/cheekybits/genny@v1.0.0 \

go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.4.1 && \
go install k8s.io/code-generator/cmd/client-gen@v0.20.6 && \
go install github.com/go-bindata/go-bindata/go-bindata@v3.1.2+incompatible && \
Expand Down
1 change: 1 addition & 0 deletions tools/chef/cookbooks/px_dev_extras/recipes/default.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@

include_recipe 'px_dev_extras::mac_os_x'
include_recipe 'px_dev_extras::gperftools'
include_recipe 'px_dev_extras::packaging'

pkg_list = [
'cmake',
Expand Down
27 changes: 27 additions & 0 deletions tools/chef/cookbooks/px_dev_extras/recipes/packaging.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Copyright 2018- The Pixie Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0

# Packaging utilities only need to run on Linux
if node.platform_family?('debian')
apt_package ['ruby-dev', 'rpm'] do
action :upgrade
end

execute 'install_fpm' do
command '/usr/bin/gem install fpm'
action :run
end
Comment on lines +23 to +26
Copy link
Member Author

@ddelnano ddelnano Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I retested using gem_package and it installs fpm within /opt/chef/embedded as I saw during my previous test. Below is the opensnoop output filtered by fpm files and the git diff showing what I tested:

$ git diff
diff --git a/tools/chef/cookbooks/px_dev_extras/recipes/packaging.rb b/tools/chef/cookbooks/px_dev_extras/recipes/packaging.rb
index 7d9177935..2292ec4d6 100644
--- a/tools/chef/cookbooks/px_dev_extras/recipes/packaging.rb
+++ b/tools/chef/cookbooks/px_dev_extras/recipes/packaging.rb
@@ -20,8 +20,11 @@ if node.platform_family?('debian')
     action :upgrade
   end

-  execute 'install_fpm' do
-    command '/usr/bin/gem install fpm'
-    action :run
+  gem_package 'fpm' do
+    gem_binary '/usr/bin/gem'
   end
+  # execute 'install_fpm' do
+  #   command '/usr/bin/gem install fpm'
+  #   action :run
+  # end
 end


$ grep fpm chef_opensnoop.txt | grep -v 'git\|php'
1759223 python3             5   0 /tmp/pip-build-tracker-dioyjfpm
1786655 chef-solo           9   0 /root/.local/share/gem/specs/index.rubygems.org%443/quick/Marshal.4.8/fpm-1.15.1.gemspec
1787105 ruby                6   0 /root/.local/share/gem/specs/index.rubygems.org%443/quick/Marshal.4.8/fpm-1.15.1.gemspec
1787105 request_set.rb*     6   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/cache/fpm-1.15.1.gem
1787105 ruby                6   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/cache/fpm-1.15.1.gem
1787105 ruby                6   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/cache/fpm-1.15.1.gem
1787105 ruby                6   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/cache/fpm-1.15.1.gem
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/CHANGELOG.rst
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/CONTRIBUTORS
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/LICENSE
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/bin/fpm
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/command.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/errors.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/namespace.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/apk.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/cpan.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/deb.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/dir.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/empty.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/freebsd.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/gem.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/npm.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/osxpkg.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/p5p.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/pacman.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/pear.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/pkgin.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/pleaserun.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/puppet.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/pyfpm/__init__.py
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/pyfpm/get_metadata.py
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/python.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/rpm.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/sh.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/snap.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/solaris.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/tar.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/virtualenv.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/package/zip.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/rake_task.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/util.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/util/tar_writer.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/lib/fpm/version.rb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/deb.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/deb/changelog.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/deb/deb.changes.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/deb/ldconfig.sh.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/deb/postinst_upgrade.sh.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/deb/postrm_upgrade.sh.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/deb/preinst_upgrade.sh.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/deb/prerm_upgrade.sh.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/osxpkg.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/p5p_metadata.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/pacman.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/pacman/INSTALL.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/pleaserun/generate-cleanup.sh
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/pleaserun/install-path.sh
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/pleaserun/install.sh
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/pleaserun/scripts/after-install.sh
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/pleaserun/scripts/before-remove.sh
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/puppet/package.pp.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/puppet/package/remove.pp.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/rpm.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/rpm/filesystem_list
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/sh.erb
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/templates/solaris.erb
1787105 ruby                6   0 /opt/chef/embedded/bin/fpm
1787105 ruby                7   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/gems/fpm-1.15.1/bin/fpm
1787105 ruby                6   0 /opt/chef/embedded/lib/ruby/gems/3.1.0/specifications/fpm-1.15.1.gemspec
1788739 rm                  4   0 fpm-1.15.1
1788739 rm                  4   0 fpm

end
2 changes: 1 addition & 1 deletion tools/chef/packer_docker_base.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
"recipe[px_dev::linux]"
],
"type": "chef-solo",
"version": "18.4.12"
"version": "18.5.0"
},
{
"inline": [
Expand Down
2 changes: 1 addition & 1 deletion tools/chef/packer_docker_dev.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
"recipe[px_dev::cleanup]"
],
"type": "chef-solo",
"version": "18.4.12"
"version": "18.5.0"
},
{
"inline": [
Expand Down
2 changes: 1 addition & 1 deletion tools/chef/packer_docker_dev_extras.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
"recipe[px_dev_extras]"
],
"type": "chef-solo",
"version": "18.4.12"
"version": "18.5.0"
},
{
"inline": [
Expand Down
2 changes: 1 addition & 1 deletion tools/chef/packer_docker_linters.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
"recipe[px_dev::cleanup]"
],
"type": "chef-solo",
"version": "18.4.12"
"version": "18.5.0"
},
{
"inline": [
Expand Down