Copy images between registries / Error: invalidData

[remarkov]

Probably more a question, not an issue. I hope this can be useful for others too.

When having more than one local image registry in the network, what would be the best way to copy/sync images between the registries without tart pull and further tart push?
So far I tried using Skopeo which works most of the times but sometimes the copied image gets corrupt and tart clone or tart pull give an error during the progress

pulling disk (160.6 GB compressed)...
9%
Error: invalidData

Most likely this is because Tart image format is not fully compatible with Docker images but that's just a guess. Any advice will be much appreciated.

[fkorotkov]

Do you have a hard requirement of having different registries? Have you considered a single registry with several read through proxies?

[edigaryev]

Error: invalidData

Which Tart version are you using to generate the VM images?

This error seems to be coming from the LZ4 decompressor and could be caused by the Skopeo reordering the manifest layers, which is an issue in the older V1 disk format.

[remarkov]

thanks for quick response!

@fkorotkov yes, unfortunately we need separate registries located in different locations/networks but some may share some of the images, not necessarily all.

@edigaryev that's on Tart 0.38.0. Do you by any chance know some workaround to prevent the layers reordering? Does it make sense to try other tools like regclient or this is something Tart-specific?

[fkorotkov]

We don't have experience with Skopeo or similar tools. Some other tools in our experience were Docker exclusive and not fully OCI compliment until the latest versions.

Another concerning part is that this issue is transient and only sometimes things getting corrupted. Definitely points to something fishy in the tooling. Hope you'll figure it out!

[fkorotkov]

I just doble checked and Disk V2 also depends on ordering but we can fix it in backward compatible way.

[edigaryev]

@remarkov I've tried reproducing this using a simple script that does tart push, skopeo copy and tart pull in a loop to no avail on the latest Tart version (2.11.0). So the initial premise that this issue is related to layer order is probably wrong.

Could try pulling your corrupted image using the latest Tart to see if the issue still persists? Ideally, push a new VM image using the tart push to get the newer V2 disk image.

It has countless improvements over the older version, including the V2 disk images. We've also just merged #835 to nail this error further down if it ever occurs again.

[remarkov]

@edigaryev Thanks for your time, I will try to do the tests with the newer version too.