Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manifest List Interface #191

Merged
merged 189 commits into from
May 7, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
189 commits
Select commit Hold shift + click to select a range
9eb407f
index: add Image Index interface
husni-faiz Apr 3, 2023
816341c
remote: index: function to create new index
husni-faiz Apr 3, 2023
fefd9bc
bugfix: index: Remove mutating media type to DockerManifestList
husni-faiz Apr 9, 2023
05456ff
index: optimize: get image from descriptor
husni-faiz Apr 9, 2023
13989a3
index: check for image architecture
husni-faiz Apr 9, 2023
dc5c024
index: add default path to Save()
husni-faiz Apr 9, 2023
02d354b
index: add keychain to new index
husni-faiz Apr 9, 2023
db82b69
index: add image index types
husni-faiz Apr 9, 2023
cdb04c9
indexOptions for creating image index
husni-faiz Apr 9, 2023
d87b7d1
index: path and mediatype options for creating index
husni-faiz Apr 9, 2023
7842116
index: add remove index method
husni-faiz Apr 9, 2023
390546a
remote: index: remove path option
husni-faiz Apr 25, 2023
aa43a59
remote: index: user docker mediatype by default
husni-faiz Apr 25, 2023
21a4790
remote: index: remove saving to layout
husni-faiz Apr 25, 2023
74f8e44
remote: index: save index to registry
husni-faiz Apr 25, 2023
d91432c
index: local: annotate and add functionality
husni-faiz May 2, 2023
bccd2af
remote/index: return error instead of calling panic
husni-faiz May 7, 2023
e24cebc
remote/index: return a wrapped error if fetcing an image fails
husni-faiz May 7, 2023
c17b6d9
local/index: return error instead of calling panic
husni-faiz May 7, 2023
add9675
remote/new_index: validate index name
husni-faiz May 7, 2023
db2e406
local/index: refactor interface functions to use the index path
husni-faiz May 8, 2023
3464e40
local/index_options: create new index with IndexManifest
husni-faiz May 8, 2023
08653dd
remote/index_options: create new index with IndexManifest
husni-faiz May 8, 2023
c941979
remote/new_index: rename file
husni-faiz May 8, 2023
9d284ea
get IndexManifest from ggcr index
husni-faiz May 15, 2023
2752977
remote/index/save: return error if platform information is missing
husni-faiz May 17, 2023
a4589d4
remote/index: ImageIndexTest to be used in unit tests
husni-faiz May 17, 2023
8c5d049
remote/index_test: setup a registry for test
husni-faiz May 17, 2023
4f9c64b
remote/index_test: basic tests for remote index
husni-faiz May 17, 2023
85b8899
remote/new_index: NewIndexTest function to create a test index
husni-faiz May 17, 2023
6a0c434
remote/new_index: use index in registry if already exists
husni-faiz May 17, 2023
8653514
remote/index: remove ManifestSize unused function
husni-faiz May 20, 2023
93861df
index: add brief comments to all index functions
husni-faiz May 21, 2023
afaa657
index: remove returning error when platform information is missing
husni-faiz May 21, 2023
2f6f5c9
local/index: remove AppendManifet wrapper function
husni-faiz May 21, 2023
d558685
local/index: save indent formatted json output
husni-faiz May 26, 2023
adae788
local/index save: use os module instead of layout package
husni-faiz May 27, 2023
7e6c885
local/index: delete method to remove index from local storage
husni-faiz May 27, 2023
54343a5
local/index: wrap GetIndexManifest errors
husni-faiz May 28, 2023
191f664
new_index: check for local index first
husni-faiz Jun 1, 2023
fcb5ade
local/index: copy referenced images to same registry as index
husni-faiz Jun 1, 2023
51609ae
index: fix github action tests failing
husni-faiz Jun 1, 2023
818f3a2
fix linter error: io/ioutil deprecated
husni-faiz Nov 20, 2023
8fdc89d
Merge branch 'main' into dev-image-index
husni-faiz Nov 25, 2023
b31a1ee
WIP added image-index methods
WYGIN Dec 8, 2023
186495b
fWIP fix lint and run format
WYGIN Dec 15, 2023
0c5a954
removed ,gitpod.yml file
WYGIN Dec 15, 2023
bd7af69
WIP removed ImageIndexHandler
WYGIN Dec 15, 2023
85fd5dc
WIP fix some missing logic
WYGIN Dec 15, 2023
f6f80f1
WIP added registry and insecure fields to IndexAddOptions
WYGIN Jan 10, 2024
20568f7
WIP added empty test cases
WYGIN Jan 11, 2024
51446eb
WIP added tests but failing
WYGIN Jan 13, 2024
334b72a
WIP removed oci layout test files
WYGIN Jan 13, 2024
4e79960
WIP removed cnb dir from local
WYGIN Jan 13, 2024
3b842ab
WIP refactor code to improve readability
WYGIN Jan 19, 2024
bd05301
WIP added fake index with empty test cases
WYGIN Jan 20, 2024
39a6fa3
WIP added few tests for fake index
WYGIN Jan 20, 2024
45d6b75
WIP fixed fake index tests
WYGIN Jan 20, 2024
e8c303a
WIP defined errors at one place
WYGIN Jan 20, 2024
00a8a80
WIP trying to fix error when saving index
WYGIN Jan 20, 2024
01db00c
WIP fix: imgutil#Save() and annotations bugs
WYGIN Jan 21, 2024
592a65c
WIP fix: all bugs fixed except image hash change when adding image to…
WYGIN Jan 21, 2024
9ec2c8c
WIP added tests for all indexes
WYGIN Jan 22, 2024
324a62a
WIP made index.NewIndex to return imgutil.ImageIndex instead of imgut…
WYGIN Jan 22, 2024
95b0df4
WIP improved test coverage
WYGIN Jan 26, 2024
3ca7018
WIP improved test coverage by adding #Save
WYGIN Jan 27, 2024
fe02008
WIP fix failing tests for #Add
WYGIN Jan 28, 2024
ab15c33
WIP fix all bugs except mutated configFile and Manifest of image when…
WYGIN Jan 29, 2024
fd3d91c
WIP fix bug except #Save urls
WYGIN Jan 30, 2024
13c3f65
WIP fix all bugs except #Save URLs
WYGIN Jan 31, 2024
d01eaf4
WIP fix: SetURLs for #Save
WYGIN Feb 5, 2024
e158f8d
WIP bug fixes
WYGIN Feb 6, 2024
8d76f28
WIP added goroutines to improve performance
WYGIN Feb 8, 2024
cdd16f8
WIP minor bug fixes
WYGIN Feb 9, 2024
921a22f
WIP added manifestOnly handler
WYGIN Feb 12, 2024
9edc6be
WIP minor changes to ImageIndex#Push
WYGIN Feb 12, 2024
e7cfa9c
WIP fix: use latest index when pushing index
WYGIN Feb 12, 2024
c98cea9
WIP added TaggableIndex to push IndexManifest only
WYGIN Feb 12, 2024
fad6445
WIP fix: added decribable methods to TaggableIndex
WYGIN Feb 12, 2024
0b018df
WIP added documentation, fix: IndexHandler#Push iissue
WYGIN Feb 13, 2024
1d563fd
WIP added tests for index.NewIndex
WYGIN Feb 13, 2024
8656617
Merge branch 'main' into dev-image-index
jjbustamante Feb 13, 2024
da80ffb
Merge branch 'dev-image-index' into image-index
WYGIN Feb 14, 2024
d071ec2
WIP formatted code after Merge upstream branch
WYGIN Feb 14, 2024
2f20186
WIP: fix bugs
WYGIN Feb 19, 2024
d57cc75
fix: tests making 'pack manifest' fail
WYGIN Feb 29, 2024
5f5ff95
fix: return error when setters called on unknown digest of fake index
WYGIN Feb 29, 2024
658b7ff
refactor: improved error messages
WYGIN Feb 29, 2024
363020d
fix: bug causing local images to fail annotate
WYGIN Mar 1, 2024
2b39c6a
fix: manifest handler not saved annotated images
WYGIN Mar 1, 2024
052db76
refactor: removed IndexHandler
WYGIN Mar 2, 2024
96558aa
refactor: remove dead code
WYGIN Mar 2, 2024
38da26f
refactor: improve ManifestHandler#Save readability
WYGIN Mar 2, 2024
2dad2cc
refactor: nit all methods except #Add
WYGIN Mar 2, 2024
0bf93b7
refactor: nit nil pointer handling
WYGIN Mar 2, 2024
3a656df
refactor: ManifestHandler tests
WYGIN Mar 2, 2024
1c3b05e
feat: added #Features #OSFeatures #URLs #Annotations methods to imgut…
WYGIN Mar 4, 2024
abfcabf
refactor: remove IndexHandler
WYGIN Mar 4, 2024
439fac9
fix: removed duplicate stringSlices for features, osFeatures, urls
WYGIN Mar 14, 2024
6e77fba
refactor: change imgutil.Platform to v1.Platform
WYGIN Mar 15, 2024
49941ee
refactor: local imaages added to index should implement EdditableImage
WYGIN Mar 15, 2024
3acfd71
fix: add local images return index not found error
WYGIN Mar 25, 2024
b5aab10
WIP: added tests for imgutil#MutateManifest
WYGIN Mar 28, 2024
5a8a519
WIP: added tests
SaikiranIndia Mar 30, 2024
3eb4bc6
fix: test
SaikiranIndia Mar 30, 2024
c709d08
WIP: refactor: improve code readability
SaikiranIndia Mar 30, 2024
ce6569d
Merge pull request #2 from WYGIN/image-index
jjbustamante Apr 2, 2024
9e2d51b
fixing some linting errors that are preventing the tests to be executed
jjbustamante Apr 2, 2024
9153420
Fixing failing unit tests
jjbustamante Apr 2, 2024
6550208
Fixing unit test
jjbustamante Apr 2, 2024
6745c1c
Running Manifest Handler tests in Parallel
jjbustamante Apr 2, 2024
d7596e5
Running the rest of the tests in Parallel
jjbustamante Apr 2, 2024
59056e1
fix: layout image concurrency issue while saving
SaikiranIndia Apr 3, 2024
9533f44
chore: revert changes from v1.Platform to imgutil.Platform
WYGIN Apr 3, 2024
75b7c54
Merge pull request #3 from WYGIN/husni-image-index
jjbustamante Apr 3, 2024
103e126
running layout and sparse tests in parallel
jjbustamante Apr 3, 2024
b85e83c
merging latest changes from main
jjbustamante Apr 9, 2024
ba1701d
restoring imgutil.MakeFileSafeName method to convert the image name t…
jjbustamante Apr 9, 2024
1db264b
skipping some test for windows because we don't have Indexes availabl…
jjbustamante Apr 9, 2024
b437d70
fixing windows error message and skipping one more test
jjbustamante Apr 9, 2024
eac7a17
Skipping test with alpine on windows
jjbustamante Apr 9, 2024
ae178ae
Updating doc comments and removing some logic that I think we don't n…
jjbustamante Apr 10, 2024
96301fa
Merge branch 'main' into dev-image-index
jjbustamante Apr 10, 2024
fc5c913
Fixing lint errors
jjbustamante Apr 10, 2024
897ad17
Fixing lint error
jjbustamante Apr 10, 2024
1d0faab
Fixing unit test after cleaning a little bit
jjbustamante Apr 10, 2024
5ff17bd
Refactoring packages, removing fake implemenatation because it is not…
jjbustamante Apr 11, 2024
6fd707e
Merging latest changes from main 04/11
jjbustamante Apr 11, 2024
b7d201c
Merge branch 'main' into dev-image-index
jjbustamante Apr 11, 2024
d8861b6
removing unused files
jjbustamante Apr 11, 2024
53dd3a9
Removing fake implementation, we don't need it right now, if we do ne…
jjbustamante Apr 11, 2024
1c7c3ec
Removing fake implementation, we don't need it right now, if we do ne…
jjbustamante Apr 11, 2024
a0a34ed
Removing more fake implementations
jjbustamante Apr 11, 2024
cedf3ad
restoring from main branch
jjbustamante Apr 11, 2024
50d54c8
refactoring layout image index implementation to be similar to our im…
jjbustamante Apr 12, 2024
ebde93a
Merge branch 'main' into dev-image-index
jjbustamante Apr 12, 2024
5cd3d3e
refactoring the image index options to follow the same pattern we use…
jjbustamante Apr 12, 2024
d038320
Merge branch 'main' into dev-image-index
jjbustamante Apr 12, 2024
0f3054e
adding assertions for checking an ImageIndex that was saved on disk
jjbustamante Apr 12, 2024
c87bd7a
adding #Save, #Add, #Push test scenarios
jjbustamante Apr 13, 2024
fd63c31
adding #Save, #Add, #Push test scenarios
jjbustamante Apr 13, 2024
530a05d
adding #Inspect and #Remove test cases
jjbustamante Apr 13, 2024
d0e9f35
Removing URL() method from index interface, it is not clear to me if …
jjbustamante Apr 15, 2024
1dfd895
removing index package all the implementation is under cnb_index.go, …
jjbustamante Apr 15, 2024
6eb1a70
Removing URLs method in other places
jjbustamante Apr 15, 2024
2c63503
Merge branch 'main' into dev-image-index
jjbustamante Apr 16, 2024
02e8bfa
Remove unneeded setters
natalieparellano Apr 17, 2024
718dd15
Remove unneeded things from local package
natalieparellano Apr 17, 2024
974fe62
Remove features, we don't care about it for now
natalieparellano Apr 17, 2024
0898f25
Remove annotations as a field on the image struct,
natalieparellano Apr 17, 2024
c98f11b
Merge pull request #5 from buildpacks/image-index
jjbustamante Apr 17, 2024
8426835
Remove mutate manifest functions, these should not be needed
natalieparellano Apr 17, 2024
ff8f39f
Remove focus
natalieparellano Apr 17, 2024
d8821bf
Fix failing test
natalieparellano Apr 17, 2024
531abe9
Merge pull request #6 from buildpacks/image-index
jjbustamante Apr 17, 2024
ffa9fbc
Cleanup imgutil.Image interface to organize methods by manifest, conf…
natalieparellano Apr 17, 2024
ad4eec6
Merge pull request #7 from buildpacks/image-index
jjbustamante Apr 17, 2024
9b2f042
Remove the option to add an index to another index,
natalieparellano Apr 17, 2024
bae2d77
Merge pull request #8 from buildpacks/image-index
jjbustamante Apr 17, 2024
88cde47
Prune more stuff from cnb_index.go, but add back setters!
natalieparellano Apr 18, 2024
ba33b6b
Merge pull request #9 from buildpacks/image-index
jjbustamante Apr 18, 2024
289f30c
Fix format
natalieparellano Apr 18, 2024
ea2a3cb
Revert "Fix format"
natalieparellano Apr 18, 2024
55ed46a
Remove CNBIndex.Format as it's not used
natalieparellano Apr 18, 2024
b25226d
Fix a few tests by updating the fixtures
natalieparellano Apr 18, 2024
811ca75
WIP: try to fix tests by overriding path.AppendIndex, but now there's…
natalieparellano Apr 18, 2024
31615f7
Revert "WIP: try to fix tests by overriding path.AppendIndex, but now…
natalieparellano Apr 18, 2024
0d6d63a
WIP: fix all but RemoveManifest test
natalieparellano Apr 18, 2024
c46fcc1
Merge pull request #10 from buildpacks/image-index
jjbustamante Apr 18, 2024
614338e
Fix in-memory part
natalieparellano Apr 18, 2024
e767856
Merge remote-tracking branch 'buildpacks/image-index' into dev-image-…
jjbustamante Apr 18, 2024
e9716db
fixing the remove manifest test, we need to create an empty index on …
jjbustamante Apr 18, 2024
725d3c9
Override Image method instead of renaming us
natalieparellano Apr 19, 2024
d382bdb
Remove insecure as it is not used
natalieparellano Apr 19, 2024
3255562
Try to simplify index options by putting them all on one struct
natalieparellano Apr 19, 2024
ddc5be8
Merge pull request #11 from buildpacks/image-index
jjbustamante Apr 19, 2024
697ac4c
Merge branch 'buildpacks:main' into dev-image-index
jjbustamante Apr 22, 2024
a6fda53
Move some stuff around
natalieparellano Apr 22, 2024
d508728
Move stuff around
natalieparellano Apr 22, 2024
4aedd4c
Merge pull request #12 from buildpacks/image-index
jjbustamante Apr 22, 2024
c2871d0
Fixing - these tests should push and pull from a local registry to av…
jjbustamante Apr 23, 2024
8412584
removing focus
jjbustamante Apr 24, 2024
cbecaa8
adding one more test case for adding multiple manifests, updating Pus…
jjbustamante Apr 24, 2024
f00946b
removing blobs from test data, after testing on pack we don't want to…
jjbustamante Apr 25, 2024
f79d933
when using a fake.Image for unit testing purpose, the Platform is und…
jjbustamante Apr 26, 2024
7a7ff3e
moving layout index tests to index_test.go and adding test coverage t…
jjbustamante Apr 29, 2024
d4a3c74
adding more test coverage for error cases in image index
jjbustamante Apr 29, 2024
2e42a19
fixing an issue when a docker media-type is updated and the value is …
jjbustamante Apr 30, 2024
73a2d7c
change a FIXME into a Note
jjbustamante May 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
WIP fix: use latest index when pushing index
Signed-off-by: WYGIN <wygininc@gmail.com>
  • Loading branch information
WYGIN committed Feb 12, 2024
commit e7cfa9cd8bf143e81dad9abc9951d8d574cf7d5a
2 changes: 1 addition & 1 deletion fakes/index.go
Original file line number Diff line number Diff line change
Expand Up @@ -544,7 +544,7 @@ func (i *Index) Inspect() (mfestStr string, err error) {
return string(mfestBytes), nil
}

func (i *Index) Remove(digest name.Digest) error {
func (i *Index) Remove(digest name.Reference) error {
if i.isDeleted {
return errors.New("index doesn't exists")
}
Expand Down
92 changes: 82 additions & 10 deletions index.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ type ImageIndex interface {
Save() error
Push(ops ...IndexPushOption) error
Inspect() (string, error)
Remove(digest name.Digest) error
Remove(ref name.Reference) error
Delete() error
}

Expand Down Expand Up @@ -3272,11 +3272,23 @@ func (h *ManifestHandler) Push(ops ...IndexPushOption) error {
}

if pushOps.Format != types.MediaType("") {
h.ImageIndex = mutate.IndexMediaType(h.ImageIndex, pushOps.Format)
if err := h.Save(); err != nil {
return err
}
}

layoutPath := filepath.Join(h.Options.XdgPath, h.Options.Reponame)
path, err := layout.FromPath(layoutPath)
if err != nil {
return err
}

h.ImageIndex, err = path.ImageIndex()
if err != nil {
return err
}

ref, err := name.ParseReference(
h.Options.Reponame,
name.WeakValidation,
Expand All @@ -3286,7 +3298,7 @@ func (h *ManifestHandler) Push(ops ...IndexPushOption) error {
return err
}

err = remote.WriteIndex(
err = remote.Put(
ref,
h.ImageIndex,
remote.WithAuthFromKeychain(h.Options.KeyChain),
Expand Down Expand Up @@ -3335,6 +3347,24 @@ func (i *IndexHandler) Push(ops ...IndexPushOption) error {
}
}

if pushOps.Format != types.MediaType("") {
i.ImageIndex = mutate.IndexMediaType(i.ImageIndex, pushOps.Format)
if err := i.Save(); err != nil {
return err
}
}

layoutPath := filepath.Join(i.Options.XdgPath, i.Options.Reponame)
path, err := layout.FromPath(layoutPath)
if err != nil {
return err
}

i.ImageIndex, err = path.ImageIndex()
if err != nil {
return err
}

ref, err := name.ParseReference(
i.Options.Reponame,
name.WeakValidation,
Expand Down Expand Up @@ -3405,10 +3435,31 @@ func (i *IndexHandler) Inspect() (string, error) {
return string(mfestBytes), nil
}

func (h *ManifestHandler) Remove(digest name.Digest) error {
hash, err := v1.NewHash(digest.Identifier())
if err != nil {
return err
func (h *ManifestHandler) Remove(ref name.Reference) (err error) {
var hash v1.Hash
switch v := ref.(type) {
case name.Tag:
desc, err := remote.Head(
v,
remote.WithAuthFromKeychain(h.Options.KeyChain),
remote.WithTransport(
getTransport(h.Options.InsecureRegistry),
),
)
if err != nil {
return err
}

if desc == nil {
return ErrManifestUndefined
}

hash = desc.Digest
default:
hash, err = v1.NewHash(v.Identifier())
if err != nil {
return err
}
}

if _, ok := h.Images[hash]; ok {
Expand Down Expand Up @@ -3441,10 +3492,31 @@ func (h *ManifestHandler) Remove(digest name.Digest) error {
return nil
}

func (i *IndexHandler) Remove(digest name.Digest) error {
hash, err := v1.NewHash(digest.Identifier())
if err != nil {
return err
func (i *IndexHandler) Remove(ref name.Reference) (err error) {
var hash v1.Hash
switch v := ref.(type) {
case name.Tag:
desc, err := remote.Head(
v,
remote.WithAuthFromKeychain(i.Options.KeyChain),
remote.WithTransport(
getTransport(i.Options.InsecureRegistry),
),
)
if err != nil {
return err
}

if desc == nil {
return ErrManifestUndefined
}

hash = desc.Digest
default:
hash, err = v1.NewHash(v.Identifier())
if err != nil {
return err
}
}

if _, ok := i.Images[hash]; ok {
Expand Down