Local HTTP cache is confusing for package maintainers

[di]

I've seen this issue raised multiple times in different formats and mediums. Most recently on #pypa today, this question was asked:

Tried this twice today, is it unusual? New releases to PyPi take 10+ minutes before they can be installed via pip.

The reason for this is that pip uses CacheControl to maintain a local HTTP cache, and it will not retry requests made within the last 10 minutes:

pip/src/pip/_internal/index.py

Lines 781 to 787 in e818039

	resp = session.get(
	url,
	headers={
	"Accept": "text/html",
	"Cache-Control": "max-age=600",
	},
	)

Unfortunately, for a lot of package authors/maintainers, this is a common workflow:

1. The have recently (< 10 minutes ago) runpip install my_package
2. They publish a new version of my_package
3. They then immediately run pip install my_package to try and install the new version
4. pip says it can't find the new version (because the cache isn't old enough)

I think that this creates enough confusion about PyPI being "slow" or some type of user error that it's worth addressing in some way.

Some ideas for how this could be addressed:

1. Disable the local cache entirely: this would probably be overkill, I don't think we should do that.
2. Shorten the max-age for cache entries: This just shortens the window for when this could happen, but doesn't fix the underlying issue.
3. Leave the cache, but add some extra API to determine if a project has been updated in the last 10 minutes: this would be a lot of work for both pip and PyPI maintainers, plus the response for this endpoint would be essentially un-cacheable (from both pip and PyPI's perspective)
4. Tell users to use --no-cache: This works, but there isn't a great way to inform new users "this is what's happening" and "this is how to fix it" aside from doing it ad-hoc -- it's essentially where we are right now.
5. Exclude the project pages from cache: project pages are pretty light, and always fetching them shouldn't cause an excessive increase in bandwidth or response time. I'm not sure if this is easy to do within pip, but seems like the best approach to me.

[benjaoming]

Agreed that caching is great and very useful! Also for maintainers and developers! For instance when running Tox, we don't want to eagerly hit the online endpoints for each and every dependency.

Suggestion to improve cache invalidation: If using editable mode (pip -e) and a version of a requirement specification isn't found in the local cache, then switch off the cache and try again (or add an error message hint to use --no-cache).

[njsmith]

Slightly puckish suggestion: pip upload should clear the cache.

[dstufft]

(5) is pretty easy to do (in fact we already do this! We just have it limited to 10 minutes instead of setting no-cache or whatever).

The main reason it caches at all is just to avoid the case of where tox might hit the same page 4-10 times in a row for each environment, when generally the page never changes. This is more useful in places with high latency internet connections where installing a large number of items could easily add a non trivial amount of time to the install process. (e.g. if you have 100 things, and it takes 500ms to round trip to PyPI, you're looking at an extra ~minute per tox environment.

I originally chose 10 minutes because I thought it represented a good trade off in terms of how quickly a new package is available and how rarely people release new packages. I'm not married to do the idea though, just explaining the thought process of why it is like it is.

If we do change it, then I suggest changing it in such a way that CacheControl is still caching the page, and doing a conditional GET with the ETag header. While that won't solve the latency issue, it is still friendlier to people with metered internet to limit the amount of bandwidth we're consuming.

[benjaoming]

@dstufft I think everybody is on the same page on the benefits of caching! :) I really <3 that pip is sensitive to all sorts of internet connections.

@njsmith

Slightly puckish suggestion: pip upload should clear the cache.

This isn't enough, since us confused maintainers will release project A and then wonder why it isn't found in CIs and in other project B that depend on A.

For my use, it would be enough if pip simply said oh I couldn't find a specified version (pinning or version range), let me try without the local cache. So not entirely switching off caching or changing the timeouts - but automatically switching off caching when it seems evident that a newer version was requested and not found in the local cache.

[di]

Another instance of this confusing a user: pypi/legacy#739 (comment)