Version of released package missing from `releases` in JSON API

[emcd]

Describe the bug

https://pypi.org/project/importlib-resources/ shows importlib-resources-5.9.0 as latest.
However, curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/importlib-resources/json | jq '.releases' | grep '5\.9\.0' does not find it.

This discrepancy is causing errors with package dependency upgrades when there is attempt to associate importlib-resources-5.9.0 with a particular file to download.

$ curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/importlib-resources/json | jq '.releases' | grep '5\.8\.1' || echo $?
  "5.8.1": [
      "filename": "importlib_resources-5.8.1-py3-none-any.whl",
      "url": "https://files.pythonhosted.org/packages/be/1f/f5a73ba2612c0b9efb4babd07163feed72390c42655e2eb2399b3eb9ab89/importlib_resources-5.8.1-py3-none-any.whl",
$ curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/importlib-resources/json | jq '.releases' | grep '5\.9\.0' || echo $?
1

Expected behavior

releases list from JSON API should reflect what is considered the latest release in the Warehouse UI as long as releases is still being used at all.

To Reproduce

See description.

My Platform

Not particularly relevant, but I can see same problem with both curl and Python's urllib.

Additional context

I understand that the releases portion of the response has been marked as deprecated in the JSON API, but my understanding is that it has not yet been obsoleted. Unsure if this is related to recent changes in the Warehouse code base or not - cursory inspection show there is some schema change related to releases but I have not dove deep enough to understand its relevance and possible effects.

[emcd]

Issue has disappeared during the time since I filed this ticket. Closing.

[emcd]

Reopening as this is still occurring (with some high profile Python packages at that).

$ curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/virtualenv/json | jq '.releases' | grep '20\.15\.1' || echo $?
  "20.15.1": [
      "filename": "virtualenv-20.15.1-py2.py3-none-any.whl",
      "url": "https://files.pythonhosted.org/packages/6f/43/df7c7b1b7a5ac4e41fac24c3682c1cc32f2c1d683d308bba2500338d1e3e/virtualenv-20.15.1-py2.py3-none-any.whl",
      "filename": "virtualenv-20.15.1.tar.gz",
      "url": "https://files.pythonhosted.org/packages/a4/2f/05b77cb73501c01963de2cef343839f0803b64aab4d5476771ae303b97a6/virtualenv-20.15.1.tar.gz",
$ curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/virtualenv/json | jq '.releases' | grep '20\.16' || echo $?
1

[emcd]

@di: Also happened with Sphinx 5.1.0 yesterday although that one is now caught up. If you want to investigate current breakage, then virtualenv 20.16.0 is a good candidate.

[miketheman]

I improved the jq command a bit to get the last one based on upload_time:

$ curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/virtualenv/json | jq '.releases|to_entries| sort_by(.value[].upload_time)|.[].key' | tail -1
"20.16.0"

This is the current version. This may have been impacted by the linked (resolved) issue - is this still a problem?

[emcd]

curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/virtualenv/json | jq '.releases|to_entries| sort_by(.value[].upload_time)|.[].key' | tail -1

That's not the result I'm getting:

$ curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/virtualenv/json | jq '.releases|to_entries| sort_by(.value[].upload_time)|.[].key' | tail -1
"20.15.1"

Also, the linked issue was allegedly resolved many hours before I posted my update to say that there was still a problem.
If this is somehow CDN-related, you may be hitting an edge node that is more up-to-date than the one I am hitting.

Replacing --silent with --verbose, I see the following lines of interest:

* Connected to pypi.org (151.101.0.223) port 443 (#0)
< etag: "ANW0jPypeXeBgfyaBuyx3g"
< x-pypi-last-serial: 14279337
< date: Tue, 26 Jul 2022 04:33:17 GMT
< x-served-by: cache-iad-kiad7000176-IAD, cache-bfi-krnt7300034-BFI
< x-cache: HIT, HIT
< x-cache-hits: 7691, 1

[miketheman]

@emcd That does indeed appear to be the case.

You can pass a custom header to the request to get more debug information from the Fastly CDN service - this may point out which TTL and path are used.

For example, from my location, I see this:

$ curl -I --silent --header 'Accept: application/json' --header 'Fastly-Debug: 1' https://pypi.org/pypi/virtualenv/json| grep fastly-debug
fastly-debug-path: (D cache-bos4648-BOS 1658970394) (F cache-bos4646-BOS 1658969969) (D cache-iad-kiad7000176-IAD 1658969969) (F cache-iad-kjyo7100051-IAD 1658958863)
fastly-debug-ttl: (H cache-bos4648-BOS - - 424) (H cache-iad-kiad7000176-IAD 75293.859 86400.000 11106)
fastly-debug-digest: 8573210ed01d067b56b65df9dd5217a26d4ebb7799a7cbde1cf74b30c58968c6

ref: https://developer.fastly.com/reference/http/http-headers/Fastly-Debug/ and https://developer.fastly.com/reference/http/http-headers/Fastly-Debug-TTL/

In my example, I can see that the cache-iad-kiad7000176-IAD host will continue to serve this object for another 75293.859 seconds (remainingTTL). This is of course unless there's another upload which is supposed to trigger the cache invalidation behavior - which the linked ticket of "CDN purging broken.." addressed.

Sine the default cache period is one day, it's possible that this misbehavior persisted for a bit after resolution - but I think enough time has passed that this ought to work now - can you confirm?

[emcd]

@miketheman: Oh, in case it wasn't clear, it's been working for some time now. Eventually the caches do update, sometimes after 18 hours or more, similar to what is being observed by others still posting to the closed ticket:

• CDN Purging is broken, stale JSON responses are served instead #11949 (comment)
• CDN Purging is broken, stale JSON responses are served instead #11949 (comment)
• CDN Purging is broken, stale JSON responses are served instead #11949 (comment)
• CDN Purging is broken, stale JSON responses are served instead #11949 (comment)

This is the reason I'm leaving my ticket open and have not closed it. It is possible that the PyPA folks are not noticing the activity in the closed ticket. I didn't want to pester @di, @dstufft, or @ewdurbin too much, but given the number of folks affected (which is surely a much larger number than who are being vocal), this issue needs additional scrutiny. I think PyPA has someone who is on the PSF payroll (i.e., not a volunteer) - would maybe be nice to see her look at this.

[dstufft]

It would be useful to know for someone who is getting a stale response, can they add the Fastly-Debug: 1 header, and see if there is a Surrogate-Key header, and if so, what the value is of that header?

[emcd]

@miketheman @dstufft : Here's one that my daily Github Actions task turned up a little while ago and which I just confirmed locally (i.e. from a different machine than the one that caught it, so possibly hitting a different cache node):

$ curl --head --verbose --location --header 'Accept: application/json' --header 'Fastly-Debug: 1' https://pypi.org/pypi/jsonschema/json | egrep '(surrogate|fastly-debug|pypi|etag)'
etag: "MSWJBwrgR1jE+dkxcgGFlw"
surrogate-control: max-age=86400, stale-while-revalidate=300, stale-if-error=86400
surrogate-key:
x-pypi-last-serial: 14406929
fastly-debug-path: (D cache-bfi-krnt7300063-BFI 1659063631) (F cache-bfi-krnt7300080-BFI 1658588462) (D cache-iad-kjyo7100124-IAD 1658588462) (F cache-iad-kjyo7100124-IAD 1658550841)
fastly-debug-ttl: (H cache-bfi-krnt7300063-BFI - - 424) (H cache-iad-kjyo7100124-IAD 48778.980 86400.000 37621)
fastly-debug-digest: c2b71dfdc359c26a52aa1c6f362af62292fdbc19166c67968435e503b1e77686

Edge node is: * Connected to pypi.org (151.101.192.223) port 443

Note that this is for jsonschema-4.8.0 which was reported broken by @dalthviz 10 hours ago.

$ curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/jsonschema/json | jq '.releases' | grep '4\.8\.0' || echo $?
1