Lack of Python package cache invalidation causes issues with Pyodide updates

[rblank]

Polyscript provides package loading functionality for the Pyodide backend through the packages config key (the link is to the PyScript docs; I couldn't find this documented for Polyscript). My understanding is that the feature works as follows:

• The first time, Polyscript loads the packages via micropip. It then generates a lockfile via micropip.freeze(), and caches it in IndexedDB with a key corresponding to the string representation of the array of packages loaded. This happens in importPackages().
• On later page loads, Polyscript looks up the package list in IndexedDB, and if it finds it, it creates a blob for the cached package list and sets the options.lockFileURL to point at that blob. It also sets the package list as options.packages, so that Pyodide loads the packages during startup. This happens in engine().

AFAICT, the cache of lockfiles is never invalidated, except when setting package_cache: 'never' in the config (which clears the cache completely). This causes issues in at least two situations:

• When updating Pyodide: The lockfile contains the Pyodide version. Pyodide checks it against its own version, and fails to load in case of mismatch.
• When updating packages: While the cached lockfile ensures that always the same package versions are used, sometimes I do want a newer version of a package to be picked up.

I haven't found a good solution to either of these cases. Getting the current Pyodide version requires loading Pyodide, but this fails due to the version mismatch, so I cannot detect beforehand if I should set package_cache: 'never'. And for the packages, I would have to compare the cached lockfile to the current one; but the cache is an implementation detail, so that's not ideal.

My current workaround, also not ideal, is to not set packages, and instead pass the package list as a different config key my_packages, then retrieve that in Python and call await pyodide_js.loadPackages(polyscript.config.my_packages) from there. This increases the latency until Pyodide is ready, because packages cannot be loaded concurrently with Pyodide initialization.

My use case actually doesn't need the cache at all: I would be fine with Polyscript never loading packages, and instead passing packages directly as options.pacakges to Pyodide, without using micropip and freezing. But there is currently no way to do that. Setting package_cache: 'never' causes the packages to always be loaded with micropip, which has even worse latency than my workaround.

So this issue report is really two things:

• A bug report for the cache invalidation issues related to Pyodide updates. Both ideas below require getting the Pyodide version before loading it, which I haven't found how to do, and they don't solve the issue with package updates.
  • Have Polyscript check the version in the cached lockfile against the Pyodide version, and invalidate the cache entry if there is a mismatch.
  • Include the Pyodide version in the cache key.
• A feature request for disabling the package loading in Polyscript and pass the package list directly to Pyodide. Ideas:
  • Accept a pyodide_options config key that is merged with options. This would have the additional benefit of allowing more customization of Pyodide, e.g. setting fullStdLib, stdLibURL, env.
  • Accept package_cache: 'passthrough', and when this is set, set options.packages = packages and don't load packages.

I would be happy to send a PR for at least the feature request. For the bug, I'm still missing the bit about getting the Pyodide version, and how to handle package updates.

[WebReflection]

AFAICT, the cache of lockfiles is never invalidated, except when setting package_cache: 'never' in the config

just FYI, as soon as the list of packages is different, that cache gets invalidated too

The lockfile contains the Pyodide version. Pyodide checks it against its own version, and fails to load in case of mismatch.

good call ... we should guard against Pyodide versions provided by the user (it could also be a local Pyodide or custom URL)

While the cached lockfile ensures that always the same package versions are used, sometimes I do want a newer version of a package to be picked up.

when that's the case, you pin your package so you have full control of the version you want via your own config ... because like I've said in the first point, as soon as the packages list contains a single different char, the cache is invalidated.

My use case actually doesn't need the cache at all

then package_cache: 'never' it is? ... or maybe not ...

Setting package_cache: 'never' causes the packages to always be loaded with micropip, which has even worse latency than my workaround.

have you talked to Pyodide folks about it? Would a package_direct = true solve your issue?

Accept a pyodide_options config key that is merged with options. This would have the additional benefit of allowing more customization of Pyodide, e.g. setting fullStdLib, stdLibURL, env.

this is an intriguing idea if only people wouldn't end up piling up bugs in PyScript when things they changed wouldn't work as expected ... I'll think about it, it's a good hint though

Accept package_cache: 'passthrough', and when this is set, set options.packages = packages and don't load packages.

I love it ... that's the reason package_cache wasn't shipped as boolean in the first place, I think this would be my best pick for the time being, the other idea feels like opening a can of worms instead to me, happy to hear from @ntoll or others.

[WebReflection]

hah .. another important point I've just realized ...

The lockfile contains the Pyodide version. Pyodide checks it against its own version, and fails to load in case of mismatch.

That's not accurate because if you are loading a different interpreter, coming from a different URL, we do guard against versions' changes:

const storage = new IDBMapSync(indexURL);

That indexURL is granted to be unique, if you are using a custom url as interpreter and it provides different versions of Pyodide behind the scene, I suggest you don't do that and provide a url that reflects in any meaningful way the version of Pyodide you are using, as we already have that guard internally and it works because otherwise every new release would've raised tons of issues from everyone, which hasn't been the case so far, not even for our own testing, so that customizing the interpreter and failing at disambiguating with it feels like more on your shoulders than ours, as issue ... I hope we agree here because otherwise we need to slow down even further the slowest bootstrap we have in polyscript stack to date and that's hard to sell/convince me we should be defensive about power users "abusing" their power, imho.

[WebReflection]

@rblank we're here #138 ... I hope that fixes all your use cases and concerns.

[rblank]

just FYI, as soon as the list of packages is different, that cache gets invalidated too

The cache is not invalidated when the list of packages changes, the list of packages is the cache key. If I change the list of packages, then change back to the original list, I still get the lockfile of the original request.

when that's the case, you pin your package so you have full control of the version you want via your own config ... because like I've said in the first point, as soon as the packages list contains a single different char, the cache is invalidated.

Interesting, so instead of specifying e.g. sqlite3 as a package, I would use sqlite3==1.0.0? That would work, and I can get the version from the full lockfile so my users will still only have to provide the package name.

then package_cache: 'never' it is? ... or maybe not ...

This by itself has very bad performance, because 1) micropip is slower than loadPackage() and 2) package loading starts only after the interpreter is fully initialized.

this is an intriguing idea if then people wouldn't end up piling up bugs in PyScript when things they changed wouldn't work as expected ... I'll think about it, it's a good hint though

Yeah, flexibility is a double-edged sword. It would have been my preference, but I understand your concern and I don't need any of the other options fields yet, so I don't mind.

I love it ... that's the reason package_cache wasn't shipped as boolean in the first place, I think this would be my best pick for the time being

Sounds good. I'll wait a bit for feedback from others, then implement this (or whatever comes out of the discussion).

... aaaand you've added the option already. Thanks! My offer to do the work was serious, though.

That's not accurate because if you are loading a different interpreter, coming from a different URL, we do guard against versions' changes: (...)
That indexURL is granted to be unique

Ah, interesting, now I understand why this issue wasn't discovered before. So you are assuming that Pyodide is always loaded from a CDN, or at least a versioned URL. That's indeed not the case for me: I deliver Pyodide through a static site generator, and the URL doesn't contain the version.

if you are using a custom url as interpreter and it provides different versions of Pyodide behind the scene, I suggest you don't do that and provide a url that reflects in any meaningful way the version of Pyodide you are using

I understand how this invalidates your assumptions, but there's no fundamental reason that URLs should contain the version numbers of whatever they point to (except possibly for HTTP cache-busting, but there are other ways to do that). Pyodide knows what version it is, and it would be less roundabout for Polyscript to get the version from Pyodide directly (if only you could do that before instantiating it) than to rely on the URL.

Anyway, the packages_cache: 'passthrough' option fully solves the issue for me, so I'm fine with leaving it at that. I would recommend documenting the reliance on versioned URLs, though.

[WebReflection]

The cache is not invalidated when the list of packages changes, the list of packages is the cache key. If I change the list of packages, then change back to the original list, I still get the lockfile of the original request.

it's a matter of "best effort" ... you could use different packages and expect the cache to work in all variants, otherwise we'd be greedily erasing desired cache for multiple scenarios ... you, as power user, have the opportunity to dig into the IndexedDB storage quotes in your devtools and decide what to keep and what to erase, we, as PyScript team, try to provide the most expected, least surprise, least technical, solution to our users, I hope this reasons our current choice of not dropping all caches when a thing changes, but I'd like to hear how to tackle this otherwise, open to discussion, imho.

Interesting, so instead of specifying e.g. sqlite3 as a package, I would use sqlite3==1.0.0? That would work, and I can get the version from the full lockfile so my users will still only have to provide the package name.

we have never stressed enough that packages should be pinned because it's not just about our cache, if you update Pyodide and it comes with a new sqlite3 package that breaks (I know, it won't likely ever happen for sqlite3, but it might for all others) your logic trusting the behavior of that package is lost, same way you pin PyScript version to be sure Pyodide and its packages are pinned too, you should pin your packages for PyScript, and Pyodide, updates / portability.

This by itself has very bad performance, because 1) micropip is slower than loadPackage() and 2) package loading starts only after the interpreter is fully initialized.

fair ... still it looks like a good conversation to have in Pyodide space, as we just use primitives and because they suggested us to use micropip.freeze() to improve performance on bootstrap, it's a chicken/egg situation because we don't have micropip until we have Pyodide so ...

... aaaand you've added the option already. Thanks! My offer to do the work was serious, though.

the PR is open, I think it has a reason to exist (people trusting Pyodide to do the fastest/best thing on bootstrap as opposite of us using Pyodide hints to do the fastest/best thing on bootstrap) so it might stick in there or be up for discussions, but I think I like that idea, the semantic of the option, and the fact we won't need to "mess up ourselves" with the bootstrap cache.

So you are assuming that Pyodide is always loaded from a CDN, or at least a versioned URL. That's indeed not the case for me: I deliver Pyodide through a static site generator, and the URL doesn't contain the version.

BINGO ... then problem solved, your static site generator should hint which version of Pyodide is using for gazillion reasons, including future/potential bug reports and what not: "I am using Pyodide, stuff doesn't work" ... the first question to file an issue is which version of Pyodide and for which browser with which package? ;-)

I understand how this invalidates your assumptions, but there's no fundamental reason that URLs should contain the version numbers of whatever they point to ...

Hopefully I've already answered this part right before this other part was quoted.

[WebReflection]

forgot last quote:

I would recommend documenting the reliance on versioned URLs, though.

@ntoll we should do that ... because that's the power of URLs, they tell a story, a time, a version, a software, and whatnot ... hide everything behind the same URL and nobody could, or would, trust that URL anymore as it might point to crickets any day, any time, any hour ... the reason even the biggest package manager, which is npm, ends up in CDNs that give you full control to the version you are pointing at, and respect that decision so that pinned software won't ever break, everything else is around latest expectations that break in every single major release.

---

edit watch out ... in semver waters, anything behind 0.x means that 0.y is major

[rblank]

Thanks again for #138, I would be very happy to see it land. It does solve my issue, so everything below is only of academic value.

your static site generator should hint which version of Pyodide is using for gazillion reasons, including future/potential bug reports and what not

It does, but the version doesn't need to be in the URL for that. It can be on a status page, or in an error report, or many other places. And I can get it from programmatically from Pyodide, which is the authoritative place for it.

The site generator was carefully engineered to generate self-contained sites, without external dependencies. Included components like Pyodide, Polyscript and others are pinned and all pages expect the versions used at generation time. So the site is always internally consistent, except if it happens to be regenerated during the very small window between loading an HTML page and loading all resources (but fixing this would require much more than having the Pyodide version in its URL). For this use case, adding the version to the URL is unnecessary additional complexity.

[WebReflection]

@rblank it's tested and working so it could land but we're kinda waiting for a few things to happen before releasing, so even if it lands you'll need to wait for PyScript release, which might happen next week, @ntoll ?

[rblank]

Oh, sure, the timing isn't critical for me since my workaround works. I meant that I'd be happy to see it land eventually. No rush. I'm not using PyScript, though, so all I need is a Polyscript release.

BTW, it seems that I'm blind, because Pyodide does indeed export its version as the very obvious version module attribute. Why not use that instead of, or in addition to, the indexURL when instantiating the IDBMapSync? Something like:

    async engine({ loadPyodide, version }, config, url, baseURL) {
        // ...
        const storage = new IDBMapSync(`${version}|${indexURL}`);

This would solve the cache invalidation issue without having to rely on the version being part of the indexURL.

[WebReflection]

Why not use that instead of, or in addition to, the indexURL when instantiating the IDBMapSync?

it's a good idea, I'll do that in the MR as it should not change/affect much but it might grant better caching and less gotchas with always-same URLs, thanks 👍