Add a new manifest command to support multi-architecture builds #1967

WYGIN · 2023-11-09T04:57:41Z

Summary

Output

Before

After

Documentation

Should this change be documented?
- Yes, see #___
- No

Bumps [github.com/buildpacks/lifecycle](https://github.com/buildpacks/lifecycle) from 0.17.1 to 0.17.2. - [Release notes](https://github.com/buildpacks/lifecycle/releases) - [Changelog](https://github.com/buildpacks/lifecycle/blob/main/RELEASE.md) - [Commits](buildpacks/lifecycle@v0.17.1...v0.17.2) --- updated-dependencies: - dependency-name: github.com/buildpacks/lifecycle dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

Go minor/patch dependencies will now be grouped, using the new Dependabot grouping feature: https://github.blog/changelog/2023-08-17-grouped-version-updates-by-semantic-version-level-for-dependabot/ Major updates, as well as security updates will still be opened as separate PRs. I've not grouped GitHub Actions update PRs, since the volume is typically much lower for those. In addition, the schedule has been changed from daily to weekly. This reduces project maintenance toil (no more having to manually create combined update PRs), plus makes it less painful for contributors to subscribe to repository notifications (currently there is a lot of noise from Dependabot PRs being opened/auto-rebased etc). Signed-off-by: Ed Morley <501702+edmorley@users.noreply.github.com>

Signed-off-by: Natalie Arellano <narellano@vmware.com>

- the builder for `pack build` - the previous image for `pack rebase` Signed-off-by: Natalie Arellano <narellano@vmware.com>

…uilder create`, ensure the os/arch matches the builder Signed-off-by: Natalie Arellano <narellano@vmware.com>

Bumps the go-dependencies group with 6 updates: | Package | From | To | | --- | --- | --- | | [github.com/docker/cli](https://github.com/docker/cli) | `24.0.6+incompatible` | `24.0.7+incompatible` | | [github.com/docker/docker](https://github.com/docker/docker) | `24.0.6+incompatible` | `24.0.7+incompatible` | | [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) | `5.9.0` | `5.10.0` | | [github.com/google/go-cmp](https://github.com/google/go-cmp) | `0.5.9` | `0.6.0` | | [github.com/onsi/gomega](https://github.com/onsi/gomega) | `1.28.0` | `1.29.0` | | [golang.org/x/oauth2](https://github.com/golang/oauth2) | `0.12.0` | `0.13.0` | Updates `github.com/docker/cli` from 24.0.6+incompatible to 24.0.7+incompatible - [Commits](docker/cli@v24.0.6...v24.0.7) Updates `github.com/docker/docker` from 24.0.6+incompatible to 24.0.7+incompatible - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v24.0.6...v24.0.7) Updates `github.com/go-git/go-git/v5` from 5.9.0 to 5.10.0 - [Release notes](https://github.com/go-git/go-git/releases) - [Commits](go-git/go-git@v5.9.0...v5.10.0) Updates `github.com/google/go-cmp` from 0.5.9 to 0.6.0 - [Release notes](https://github.com/google/go-cmp/releases) - [Commits](google/go-cmp@v0.5.9...v0.6.0) Updates `github.com/onsi/gomega` from 1.28.0 to 1.29.0 - [Release notes](https://github.com/onsi/gomega/releases) - [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md) - [Commits](onsi/gomega@v1.28.0...v1.29.0) Updates `golang.org/x/oauth2` from 0.12.0 to 0.13.0 - [Commits](golang/oauth2@v0.12.0...v0.13.0) --- updated-dependencies: - dependency-name: github.com/docker/cli dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/go-git/go-git/v5 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/google/go-cmp dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/onsi/gomega dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: golang.org/x/oauth2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <support@github.com>

When using --publish it should say "saved to registry" not "saved to docker daemon" Signed-off-by: Natalie Arellano <narellano@vmware.com>

Signed-off-by: WYGIN <wygininc@gmail.com>

Signed-off-by: WYGIN <wygininc@gmail.com> Date: Thu Nov 9 03:33:42 2023 +0000

…nto manifest

Signed-off-by: WYGIN <wygininc@gmail.com>

WYGIN · 2023-11-10T09:41:01Z

@jjbustamante is there any possible way to build buildpacks into images directly without cli flags from pack build <image-name> [flags] so that any local imageIndex is pushed to the registry the local images to pushed into the registry without any action from users, for now i am going to notify user about the local image that need to be pushed to registry

jjbustamante · 2023-11-10T14:01:16Z

internal/commands/manifest_create.go

+		panic(fmt.Sprintf("error marking insecure as hidden: %v", err))
+	}
+	cmdFlags.BoolVar(&flags.publish, "publish", false, "Publish to registry")
+	cmdFlags.BoolVar(&flags.all, "all", false, "Add all of the list's images if the images to add are lists/index")


We actually leave the complexity of dealing with recursive Image Indexes because no all registries handle that correctly, it is possible according to the OCI Image spec, but in practice it is not very useful.

jjbustamante · 2023-11-10T14:03:51Z

internal/commands/manifest_create.go

+	}
+	cmdFlags.BoolVar(&flags.publish, "publish", false, "Publish to registry")
+	cmdFlags.BoolVar(&flags.all, "all", false, "Add all of the list's images if the images to add are lists/index")
+	cmdFlags.BoolVar(&flags.amend, "amend", false, "Modify an existing list/index if one with the desired name already exists")


I am not sure if I want to use the same create command to update an existing image index, I think it could be another command, if we actually need it.

jjbustamante

A few comments in the manifest create commands flags

jjbustamante · 2023-11-10T14:12:44Z

@jjbustamante is there any possible way to build buildpacks into images directly without cli flags from pack build <image-name> [flags]

I think I am not understanding what you want to do here, pack buildpack package will create a Buildpack OCI image that could be saved into the daemon, registry or in a file on disk with the .cnb extension (this is actually OCI layout format).

We do want to add capabilities to pack to handle the creation of Image Indexes from the beginning, but we are trying to define the behavior with the following RFC.

In this particular work, we just want to provide end-users with some commands to create the Image Index assuming they already have OCI images created. Check this RFC

WYGIN added 18 commits November 9, 2023 02:54

WIP: added manifest_add command

17ec0e6

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_annotate command

45f39bc

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_create command

5b75d95

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_exists command

9799f01

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_inspect command

1968518

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_push command

d8f6039

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_remove command

07b9150

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_rm command

a6355e1

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added add_manifest client

f24b167

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added annotate_manifest client

15ed584

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added IndexFactory to Client

8cf016e

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added create_manifest client

0ed954f

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added delete_manifest client

dfd4bc4

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added exists_manifest client

94ba3e3

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added inspect_manifest client

4ec02e7

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added push_manifest client

f2161a9

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added remove_manifest client

2080208

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added errors

5aa4b9f

Signed-off-by: WYGIN <wygininc@gmail.com>

github-actions bot added the type/enhancement Issue that requests a new feature or improvement. label Nov 9, 2023

github-actions bot added this to the 0.33.0 milestone Nov 9, 2023

dependabot bot and others added 10 commits November 9, 2023 11:06

Updating default lifecycle version to 0.17.2

32e5d1d

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

Add buildpacksio/pack:<version>-base images to delivery

bf88725

Signed-off-by: Natalie Arellano <narellano@vmware.com>

Add floating :base tag

31654b8

Signed-off-by: Natalie Arellano <narellano@vmware.com>

Ensure the run image os/arch always matches:

f70fdfe

- the builder for `pack build` - the previous image for `pack rebase` Signed-off-by: Natalie Arellano <narellano@vmware.com>

When downloading buildpacks or extensions for pack build or `pack b…

0133424

…uilder create`, ensure the os/arch matches the builder Signed-off-by: Natalie Arellano <narellano@vmware.com>

Fix misleading log message when publishing a buildpack package

71821d8

When using --publish it should say "saved to registry" not "saved to docker daemon" Signed-off-by: Natalie Arellano <narellano@vmware.com>

WIP added code to generate env files in buildConfigEnv dir

4624639

Signed-off-by: WYGIN <wygininc@gmail.com>

WYGIN added 21 commits November 9, 2023 11:07

WIP added manifest_create command

a76513d

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_exists command

128f178

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_inspect command

d238c64

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_push command

9006ad4

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_remove command

11d416d

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added manifest_rm command

749b6fe

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added add_manifest client

8b25d63

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added annotate_manifest client

6b8f798

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added IndexFactory to Client

4aa9362

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added create_manifest client

89763fd

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added delete_manifest client

a5b1faa

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added exists_manifest client

6090d11

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added inspect_manifest client

b8f3186

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added push_manifest client

b9dd8a4

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added remove_manifest client

54c3c6c

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added errors

553f927

Signed-off-by: WYGIN <wygininc@gmail.com> Date: Thu Nov 9 03:33:42 2023 +0000

Merge branch 'manifest' of https://github.com/WYGIN/buildpacks-pack i…

c7180d8

…nto manifest

WIP generate mock code with 'make generate'

4bbb483

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP added tests for add_manifest

8f81b6c

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP improved add_manifest client tests

f12c849

Signed-off-by: WYGIN <wygininc@gmail.com>

WIP refactor manifest_add client

4326812

Signed-off-by: WYGIN <wygininc@gmail.com>

jjbustamante reviewed Nov 10, 2023

View reviewed changes

jjbustamante added experimental Issue or PR refers to an experimental feature. epic/multi-arch labels Nov 20, 2023

jjbustamante removed this from the 0.33.0 milestone Nov 22, 2023

WYGIN closed this Jan 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a new manifest command to support multi-architecture builds #1967

Add a new manifest command to support multi-architecture builds #1967

WYGIN commented Nov 9, 2023 •

edited by jjbustamante

Loading

WYGIN commented Nov 10, 2023

jjbustamante Nov 10, 2023

jjbustamante Nov 10, 2023

jjbustamante left a comment

jjbustamante commented Nov 10, 2023

Add a new manifest command to support multi-architecture builds #1967

Add a new manifest command to support multi-architecture builds #1967

Conversation

WYGIN commented Nov 9, 2023 • edited by jjbustamante Loading

Summary

Output

Before

After

Documentation

Related

WYGIN commented Nov 10, 2023

jjbustamante Nov 10, 2023

Choose a reason for hiding this comment

jjbustamante Nov 10, 2023

Choose a reason for hiding this comment

jjbustamante left a comment

Choose a reason for hiding this comment

jjbustamante commented Nov 10, 2023

WYGIN commented Nov 9, 2023 •

edited by jjbustamante

Loading