Skip to content

Fix incorrect pyc cache loaded when test file is updated in short time #13293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Fix incorrect pyc cache loaded when test file is updated in short time #13293

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4e08f5c
fix https://github.com/pytest-dev/pytest/issues/13292
jnhyperion Mar 13, 2025
e4d70c7
fix https://github.com/pytest-dev/pytest/issues/13292
jnhyperion Mar 13, 2025
71fbe1a
Merge branch 'pytest-dev:main' into main
jnhyperion Mar 14, 2025
c573aa4
Add changelog
jnhyperion Mar 14, 2025
2431211
fix ut in `Windows` platform
jnhyperion Mar 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AUTHORS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -488,3 +488,4 @@ Zachary OBrien
Zhouxin Qiu
Zoltán Máté
Zsolt Cserna
Johnny Huang
1 change: 1 addition & 0 deletions changelog/13292.bugfix.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed stale pyc cache loading after rapid test file updates.
4 changes: 2 additions & 2 deletions src/_pytest/assertion/rewrite.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,7 @@ def _write_pyc_fp(
flags = b"\x00\x00\x00\x00"
fp.write(flags)
# as of now, bytecode header expects 32-bit numbers for size and mtime (#4903)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I missed it before, but this comment seems to negate this change -- it takes the lower 32 bits while st_mtime_ns needs at least 64 bits, e.g.

>>> import os
>>> st = os.stat('/home')
>>> print(st.st_mtime_ns)
1669828167393028173
>>> print(st.st_mtime_ns & 0xFFFFFFFF)
23537741
>>> print(int(st.st_mtime) & 0xFFFFFFFF)
1669828167

Basically, the pyc timestamp format is 32 bits, and we are trying to stay compatible with it (even though it's not absolutely necessary as we do our own loading). The same problem occurs with python itself, not just pytest.

See:
python/cpython#121376
https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815

Also related: #11418

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're correct, this is more complex than I expected, I used to run the same test without pytest, but it didn't reproduce somehow, so I thought it's only caused by pytest. Actually, the similar issue still exists in python's import system, and pytest's cache rewrite inherited the same issue. Looks like no proper solution for the issue until now, perhaps we should close the PR.

mtime = int(source_stat.st_mtime) & 0xFFFFFFFF
mtime = source_stat.st_mtime_ns & 0xFFFFFFFF
size = source_stat.st_size & 0xFFFFFFFF
# "<LL" stands for 2 unsigned longs, little-endian.
fp.write(struct.pack("<LL", mtime, size))
Expand Down Expand Up @@ -374,7 +374,7 @@ def _read_pyc(
with fp:
try:
stat_result = os.stat(source)
mtime = int(stat_result.st_mtime)
mtime = stat_result.st_mtime_ns
size = stat_result.st_size
data = fp.read(16)
except OSError as e:
Expand Down
34 changes: 32 additions & 2 deletions testing/test_assertrewrite.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1381,9 +1381,12 @@ def test_read_pyc_more_invalid(self, tmp_path: Path) -> None:

flags = b"\x00\x00\x00\x00"

mtime = b"\x58\x3c\xb0\x5f"
mtime = b"\x98\xf34\x10\x91\x8b,\x18"
mtime_int = int.from_bytes(mtime, "little")
os.utime(source, (mtime_int, mtime_int))
# set mtime_ns for win requires integer nanoseconds
os.utime(source, ns=(mtime_int, mtime_int))

mtime = (mtime_int & 0xFFFFFFFF).to_bytes(4, "little")

size = len(source_bytes).to_bytes(4, "little")

Expand Down Expand Up @@ -2374,3 +2377,30 @@ def test_saferepr_unbounded(self):
_saferepr(self.Help)
== f"<class '{Path(__file__).stem}.{self.__class__.__name__}.Help'>"
)


class TestIssue13292:
"""
Check the pyc cache generated in the 1st test execution
is not loaded in the 2nd test execution.
"""

def test_stale_pyc_cache_is_not_loaded(self, pytester: Pytester) -> None:
pytester.makepyfile("""
def test_dummy1():
def func():
pass
print(func.__qualname__)
""")
r = pytester.runpytest("-s")
assert r.ret == 0
assert "test_dummy1.<locals>.func" in r.stdout.str()
pytester.makepyfile("""
def test_dummy2():
def func():
pass
print(func.__qualname__)
""")
r = pytester.runpytest("-s")
assert r.ret == 0
assert "test_dummy2.<locals>.func" in r.stdout.str()