fsck failure w/custom base images

[cgwalters]

When doing custom base images I'm seeing this now

[root@ibm-p8-kvm-03-guest-02 ~]# ostree fsck
Validating refs...
Validating refs in collections...
Enumerating commits...
Verifying content integrity of 3 commit objects...
fsck objects (1/66230) [             ]   0%
error: In commits f00559fb1659e6673295b02f2961b2a491127cc750cf28840e93bd779ab555ff: fsck content object 3858fbe9638069f39568aad5462f7989454f37b43c3e2795cef5244898d41a96: Corrupted file object; checksum expected='3858fbe9638069f39568aad5462f7989454f37b43c3e2795cef5244898d41a96' actual='0e7406b4ddbd2bc5d263a6ecf38b2e677c377c00dde460f737e75f862300aeda'
[root@ibm-p8-kvm-03-guest-02 ~]# ostree ls -R -X -C f00559fb1659e6673295b02f2961b2a491127cc750cf28840e93bd779ab555ff | grep 3858fbe9638069f39568aad5462f7989454f37b43c3e2795cef5244898d41a96
-00755 0 0 909224 3858fbe9638069f39568aad5462f7989454f37b43c3e2795cef5244898d41a96 { [(b'security.selinux', b'system_u:object_r:system_conf_t:s0')] } /usr/lib64/libfreeblpriv3.so
[root@ibm-p8-kvm-03-guest-02 ~]# ls -alZ /usr/lib64/libfreeblpriv3.so 
-rwxr-xr-x. 1 root root system_u:object_r:lib_t:s0 909224 Jan  1  1970 /usr/lib64/libfreeblpriv3.so

which is just completely baffling me

[cgwalters]

OK yeah, this is conceptually an existing bug, it's just that one is way more likely to hit it with custom base images as a single tar layer.

The commit we generate for the layer caches is in a less defined state because we don't have a reference base policy here.

The real fix here is composefs, where we can cleanly decouple labeling of the object store from labeling of the merged FS tree. (Though it'd still make sense to have some optimized caching here even so to avoid recomputing all labels on every update in general)

[cgwalters]

Just thinking about this one some more. Man this is all so messy because ostree was very much designed with the idea that labeling is done on the build server and we were doing that with the ostree-container bits and now we're trying to do it both ways, and this is fallout.

I think the shortest term hack here is to teach ostree fsck to skip the layer blob refs (annotation on the commit?)

The more medium term is probably to try to go deeper onto using composefs instead of ostree here; especially composefs-with-splitstream/tar-split instead of the layer blob refs.