pathlib: Path.iterdir() is surprisingly not streaming

[nh2]

Bug report

Suprisingly (contrary to its name and being a generator), Path.iterdir() does not stream directory entries:

It reads all directory entries into memory before yielding the first entry.

This can cause excessive memory usage when "iterating" over very large directories.

Users would expect by default that .iterdir() operates in a streaming way, like UNIX find or readdir(), streaming e.g. the results of the underlying system calls such as getdents64() on Linux.

But it does entries = list(scandir_it) instead:

cpython/Lib/pathlib/__init__.py

Lines 835 to 843 in c419af9

	def iterdir(self):
	"""Yield path objects of the directory contents.
	The children are yielded in arbitrary order, and the
	special entries '.' and '..' are not included.
	"""
	root_dir = str(self)
	with os.scandir(root_dir) as scandir_it:
	entries = list(scandir_it)

This should be documented, and can hopefully be fixed without too much breakage.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Related issues

• Pathlib.iterdir semantics change dramatically under Python 3.13 #129871

Linked PRs

• gh-136059: docs: pathlib: Mention that iterdir() is surprisingly not streaming #136060

[LamentXU123]

I am not very fond of fixing it since it is not an optimization (it might change the behaviour a lot) and will effect the backward compatibility here.

And the doc change, I suggest that we can mention this but maybe not as a warning, I think it's just not serious enough

[nh2]

Another drawback of current .iterdir() is that it is uninterruptible, e.g. Ctrl+C during a large .iterdir() has no effect.

Ctrl+C works well during a proper streaming os.scandir().

I do not know where this difference comes from, as theoretically list() is also just reading from the proper streaming os.scandir(); maybe list() is implemented in a low-level uninterruptible way?

[nh2]

I am not very fond of fixing it since it is not an optimization (it might change the behaviour a lot) and will effect the backward compatibility here.

Adding another function with a new name and different semantics might be the better option here.

[jaraco]

I think I agree - that line 843 is a bad code smell. It would be better to consume that iterable lazily, irrespective of the ability to interrupt the operation. I'd be interested to know what behaviors rely on that eager consumption of the iterable.

Acknowledged it may be difficult to handle the backward compatibility concerns about this behavior.

In my opinion, it's a bug that a method called iterdir() isn't yielding the items lazily, based on its namesake and documentation. I'd rather see this behavior fixed than simply documented.

[emmatyping]

I believe this behavior was actually changed in 3.13 with #117728. Previously iterdir was lazy via returning a generator. But it was changed to materialize all entries at once. Since I think we should revert the semantics change due to breaking anyio (mentioned in #129871), it seems that we should revert to the pre-3.13 behavior which is what @nh2 expects.

[ronaldoussoren]

I think I agree - that line 843 is a bad code smell. It would be better to consume that iterable lazily, irrespective of the ability to interrupt the operation. I'd be interested to know what behaviors rely on that eager consumption of the iterable.

Acknowledged it may be difficult to handle the backward compatibility concerns about this behavior.

In my opinion, it's a bug that a method called iterdir() isn't yielding the items lazily, based on its namesake and documentation. I'd rather see this behavior fixed than simply documented.

The old version before #117728 used os.listdir which also eagerly reads the directory, line 843 maintains the semantics of that. The actual pathlib.Path instances are still created lazily. The main difference w.r.t. laziness is that the old version only called os.listdir when actually consuming the iterator, while the new implementation always calls os.scandir.

I guess there's no real good choice here: the current behaviour uses more memory when procesing large directories. Lazily processing the result from os.scandir uses more open file descriptors when processing nested filesystem structures (e.g. in a naive reimplementation of os.walk). The latter can also run out of file descriptors when the result from iterdir is kept alive without fully consuming it.