Handling the learning & interoperability testing use cases for TestPyPI

[ncoghlan]

Split out from #726 to cover the additional TestPyPI use cases mentioned by @takluyver in that thread:

• folks just learning & experimenting with the Python packaging toolchain
• toolchain developers testing the interoperability of their publication process

I believe the current TestPyPI implements this by periodically purging the entire instance database, but it might be beneficial to instead implement this as a configurable "mode" in Warehouse, where it uses the regular Warehouse account management backend, but all the state related to packages themselves is purely ephemeral and auto-deletes after a "while".

[dstufft]

To be clear, TestPyPI basically never gets purged. I think it's happened once or twice ever and it is an entirely manual process. It's basically just a second deployment of the relevant codebases with a slightly different theme at this point.

[ncoghlan]

@dstufft Does the list-of-lists deleter still get run against the main DB? I know @r1chardj0n3s used to have to do that regularly thanks to an old tutorial that suggested publishing practice packages to the live service.

[r1chardj0n3s]

The lists-of-list purger was run against the production database, since the URL in the tutorial was the prod URL, not test. I ran it a few months back, after quite a break. The code is closely tied to the legacy codebase, but should be reasonable to migrate to the new, I should think. It's in the admin.py script.

[ncoghlan]

Responding to pypa/packaging-problems#114 and thinking about #720 (comment), I'm wondering if a possible option here might be to offer per-user personal indices that follow the same model as the proposed staging index, only even more permissive:

• only warn about name conflicts with the main PyPI index, don't prohibit them
• personal releases are permanently mutable (no staging required)

In essence, every user would get their own optional scratchpad to do whatever they want, and the inherent release mutability and barriers to use (i.e. needing to pass --extra-index-url) would encourage publishing to the main index once usage started to expand beyond the original author.

[taion]

Sorry to jump in here – I'm curious why TestPyPI is necessary at all. I'm not aware of a "test npm", a "test Crates", or a "test RubyGems". Is it possible that TestPyPI is solving a problem that generally doesn't exist any more?

[ncoghlan]

@taion See #726 for the background discussion here. Test PyPI currently gets used for 3 different use cases:

• Testing legacy PyPI changes before they get pushed live (replaced in Warehouse by a proper CI arrangement and support for local testing of changes)
• Testing package updates before pushing them to the main index (proposed to be replaced by an integrated staging index feature)
• A sandbox where students and other folks just learning Python can be introduced to the package publication toolchain without generating noise on the main index server (which I'm now suggesting would be a good reason to introduce support for per-user indices)

And as that list shows, keeping an entire separate sandbox instance around probably isn't the right long term answer to any of those situations - it was just an expedient answer given the limitations of the original PyPI code base.

[taion]

Makes sense. You're probably already aware of this, but I will note that "testing" seems like a less interesting consequence of per-user indices than the people who will be offering you money to get private per-user indices.

[ncoghlan]

@taion We're fairly wary of fee-for-service arrangements that may not only jeopardise the PSF's public interest charity status with the IRS, but also potentially cause concerns for the providers of the in-kind infrastructure donations that allow PyPI to operate at all. (That's very different from NPM's situation, since that's structured as a private for-profit company, albeit one that seems to be run by some genuinely lovely people).

Since we advise folks to run caching proxies anyway, that means the "private Python package indices" market is one we're largely leaving to language independent artifact management vendors like JFrog and Sonatype.

[taion]

Yeah, I understand. I just mean it's sort of funny – you'd have support at a technical level for a really useful feature, and one that people usually pay for, but not really a way to deploy it.

Like, Gemfury's great, but I'm way too afraid of accidentally uploading my private packages to PyPI.

---

To get back to the point, though, even for the "learning" case, is there a lot of harm to users uploading "hello world" packages to the main repository? Again, to take npm as an example, it has about 5x the number of packages as PyPI, and while it does support scoped packages, their tutorial for publishing packages literally shows just dropping something into the root scope: https://docs.npmjs.com/getting-started/publishing-npm-packages.

In practice I can't think of any trouble that's caused for me as either a consumer or a publisher of npm packages.

[dstufft]

Yea it's an issue. There is a book that has people upload a package to live PyPI that we semi-regularly have to delete their packages to avoid the spam in the main repository. You can normally find them by searching for "A simple printer of nested lists" (see https://pypi.org/search/?q=A+simple+printer+of+nested+lists).

I don't like the per-user indexes here, because I think that:

(A) They're not really needed for PyPI's core use case
(B) They're going to be much harder to scale and keep available since less traffic to each individual list means less likely they're going to be in CDN cache.
(C) It adds additional complexity and functionality to PyPI that is likely best handled by DevPI or similiar.

[taion]

The official npm docs include a screencast that demonstrates uploading a dummy package to npm. Funny enough, that package is actually still around: https://www.npmjs.com/package/npm-demo-pkg.

Do the spammy demo packages cause maintenance issues for PyPI? It looks like those are about 1.5% of all the packages on PyPI, which seems like a lot – but I've never noticed them, which I guess is one data point that suggests that they may not be a problem for users.