SHA1 sum of package dists should be more reliable than a SHA1 on the zip result

[Seldaek]

(Replacing #1496 which has become a mess, references #5940)

If multiple servers create archives, then those archives can have different SHA1s which is problematic. Potential solutions:

• hash the contents in a reproducible way (might be very slow)
• just avoid recreating archives all the time (depends on ecosystem but preferable)

[fadoe]

Is there a solution until now?

[david0]

I'm not an expert in the ZIP format, but I guess the file access times (https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT) are changing and thereby messing up the checksum.
Would it be an option the mask out the relevant parts?

[Seldaek]

@david0 it's a good theory/idea, either that or even modification times. I am not sure if these metadata bits are part of the header or if they are also compressed together with the file contents. If it's in the headers then maybe we could more or less easily mask them and then hash the outcome yes.

If anyone is up for digging further in the spec and possibly trying to build a small proof of concept (it really does not need to be part of composer as a first step) that'd be great.

[david0]

My proof of concept.

I also noticed that file access times etc. are stored in an optional field "extra". It is possible to omit this field when creating zip files using zip -X. So if we would find a way to create the zip-files without "extra" for satis, the problem should be solved.

[Seldaek]

@david0 ideally we should support this for zips created by github as well though where we can't control the process, so if we can strip the headers it'd be best.

I am not sure about the proof of concept because I get the same shasums anyway for the testdata1/2 and 3/4 couples. So the sha.php code does not prove much but it does indeed produce different shas that are also consistent.

It'd be nice if we could reproduce the exact issue (two different zips of same content) and get the same sha.

[david0]

@Seldaek must be some difference on your system (with atime). I updated the poc and included my testfiles.

$ shasum *.zip
689595f5f847e9edc4c01fb2c81a17472f009df5  testdata1_extra.zip
0911f01200a2a9a0b7f251b7308fcbe3a18c8f56  testdata2_extra.zip
312d94af3ac4c929169e5e515b53ee69be4b1e1c  testdata3_noextra.zip
312d94af3ac4c929169e5e515b53ee69be4b1e1c  testdata4_noextra.zip
34ef754c68ffbb559954f7932b70e46b5d826cc3  testdata5_corrupted_extra.zip
823ced3f35d98acbb4cb33dd1b1bb25e2f611323  testdata5_corrupted_noextra.zip
$ php sha.php
testdata1_extra.zip: e122eb938e4a680503b97c278f71583d27537463
testdata2_extra.zip: e122eb938e4a680503b97c278f71583d27537463
testdata3_noextra.zip: 31b74a5869270919e41f185f593f14306d9dc8c9
testdata4_noextra.zip: 31b74a5869270919e41f185f593f14306d9dc8c9
testdata5_corrupted_extra.zip: 63eae002cb6deb46240d300308ea783f9fd41735
testdata5_corrupted_noextra.zip: 3b0d719237c790a00cf28fc998ebf6b7dc2a8f71

It would be also possible to make testdata 1-4 have the same hash by ignoring the central directory structure also.

[Seldaek]

OK I get the same hashes as you do so that's a good start ;) Not sure what central directory structure is, does that include paths/directory structure of the unzipped output? Because ignoring that might be risky.

[david0]

The central directory structure seems the be a repetition of the file headers for random access.
I updated the poc once again and now I get same hashes for testdata 1-4.

[sroze]

@Seldaek do you have some feedback about @david0's work ? It would be really interesting to fix these checksum "issue" IMO because it is causing some problems with private dependency management basically.