bpo-34990: Change pyc headers to use 64-bit timestamps

[ammaraskar]

This is a more permanent alternative to the compileall issue. As Benjamin originally suggested here in the PEP 552 pull request this changes both the timestamp and size to use 64 bit integers. In the hashed pyc files, we end up padding out the space that would be used with zeroes.

It is possible that I missed a few tests since the larger 8-byte field can masquerade as the two 4-byte fields easily.

As a bonus this also lets us support large source files, though I question if anyone is really compiling larger than 2/4GB python files.

https://bugs.python.org/issue34990

[bmwiedemann]

In practice, none of those fields really need to be 64 bit, as long as the handling of timestamps is done the way it is specified with &0xffff_ffff . And I guess the change makes backporting to existing releases hard. Though we all hope that nobody will use python-3.7 in 18 years, we cannot be certain.

[ammaraskar]

handling of timestamps is done the way it is specified with &0xffff_ffff

Sorry I might be missing something, is this specified in a PEP or some document? Or are you referring to the mask in _bootstrap_external.py?

And I guess the change makes backporting to existing releases hard

That's true, I don't particularly mind backporting the changes to 3.7 as I think this qualifies as a bugfix, though the release manager will have final say on that.

---

I pinged Stephane about their original PR and if they give me the go-ahead on taking it over I can update that for a comparison on which one would be better.

[bmwiedemann]

I learned it from https://bugs.python.org/issue34990#msg327750

I went through git history to find it:
5136ac0c added source_size = source_stats['size'] & 0xFFFFFFFF" for bpo-13645
and for PEP552, this concept was extended to source_mtime in 42aa93b8
though neither mentioned it in text anywhere.

For backporting, compatibility might be a consideration. What happens if such new .pyc files created by python-3.7.x are encountered by a python-3.7.0 ?

[ammaraskar]

I went through git history to find it

Thank you for investigating! I'm speculating but best I can tell the reason & 0xFFFFFFFF was added was to deal with the fact that stat->st_size could be a 64-bit value. The mask likely serves to make sure it fits.

What happens if such new .pyc files created by python-3.7.x are encountered by a python-3.7.0 ?

Aah I see what you mean. Now that I look at it I don't think there's any precedent for backporting changes to the bytecode format mid-release. I think what ends up happening is that because of the MAGIC value change we end up regenerating the pyc files, however this does break pyc only distributions.

I'll leave it to the experts but I don't this can be safely backported.

---

I've gotten the go ahead from Stephane to continue their work from the previous PR so worst case we can apply that to 3.7 and 3.8

[ammaraskar]

Closing in favor of #19708