PyPI package release plan for plugins?

[ionelmc]

Just putting this here.

Things to care about:

• versioning
• testing (also Improve tests and testing environment #25)
• release management (eg: when to make releases)

[ionelmc]

Also, where to actually store the code: here or individual repositories.

[kylef]

Just linking this here since it's slightly related and would make this easier: getpelican/pelican#1488

[FedericoCeratto]

• Licensing (some plugins might have missing/inconsistent licenses)

[FedericoCeratto]

Also see #39

[justinmayer]

Anyone who wants to help make this a reality is encouraged to post a comment on this related issue: https://bitbucket.org/pypa/pypi/issues/238/request-for-new-pypi-trove-classifier-for

[MinchinWeb]

It's take a bit, but trove classifiers for PyPI have now been added!

[moorepants]

Adding my two cents: it would be much nicer for the end user to have versioned plugins that are install-able from pypi.

[justinmayer]

No promises, but hoping to rally the community and solicit help to implement this for 5.0.

[dreamalligator]

I'd be so happy with modularizing this and importing individual plugins even though there would be the extra bloat of the unused projects. For years my practice has been copy-pasting pieces, which isnt maintainable.

@justinmayer you can ping me to help if you need assigning sections of the codebase etc. :D

[dreamalligator]

maybe this PR (#1215) will help get us going? I'm attempting to do the initial steps of @ionelmc's and @moorepants suggestions (versions / CI / pypi installation) and allow folks to install pelican-plugins via pip and pipenv which covers 2/3 of those things. Since pelican-plugins already uses Travis-CI, there are some relatively simple steps to auto-release.

[Lucas-C]

Thanks for helping on this issue !
I commented on your PR my main concerns.

[moorepants]

I think the first step is to move each plugin into its own repository (if not already). I don't think it is a good idea to have a pip installable mega distribution, because you have to lock each plugin at a specific version for the collection. This will be problematic if a user wants to specify specific versions of different plugins. If you want a mega distribution you need to work out how to pin versions for each plugin in a compatible set.

[dreamalligator]

ah I see @moorepants. we should do the more long goal thing with so many people using this project. I'll close my PR and help with the splitting apart.

I wrote one of the liquid tag plugins (gram.py) ages ago, maybe I'll refresh it...

[dreamalligator]

Question about breaking things off. I'm going to do such for that old plugin I mentioned the other day, but what about helping with plugins I didnt write?

Should I break it off, and then make a transfer offer to getpelican?

or if I register it with PyPI, should I add one of you, such as Justin as a maintainer? or maybe just best to skip that and let y'all handle that...

[Lucas-C]

Should I break it off, and then make a transfer offer to getpelican?

It was suggested in getpelican/pelican#1488 to use https://github.com/pelican-plugins

if I register it with PyPI, should I add one of you, such as Justin as a maintainer?

Currently all the Pypi packages with classifier Framework::Pelican::Plugins seem to have been uploaded by individuals. Maybe an account could be created on Pypi and linked to the GitHub team @getpelican/moderators ? Then some Travis CI pipelines could be configured to publish packages with a token linked to this shared account. What do you think @getpelican/reviewers ?

[avaris]

I had some thoughts about PyPi installable plugin structure for a while. This issue looks like a good place to share for comments.

The organization from this comment (https://github.com/pelican-plugins) is better for this.

Instead of installing plugins haphazardly, namespace packages sounds appealing. We could make pelican.plugins a namespace package for plugins and individual plugins would install there. i.e. assets plugin would go into pelican.plugins.assets etc. This could also be used later in pelican for plugin discovery instead of PLUGIN_PATHS.

A standard metadata for a plugin that is accessible when imported would be good. Nothing complicated: name, version, short description, supported pelican version (validation in pelican using this could be added) etc.

[moorepants]

How do we get access to push a plugin to https://github.com/pelican-plugins?

I've converted render_math to a standalone repo: https://github.com/moorepants/render_math and can transfer the repo.

[moorepants]

One tip. If you clone this repo you can do this:

git filter-branch --prune-empty --subdirectory-filter render_math/ master

to retain the git history of that plugin. You just change the directory from render_math to the plugin you are converting.

[justinmayer]

I would like to take this opportunity to thank everyone who has stepped up to discuss, participate, and help enhance Pelican’s modularity. The recent influx of people volunteering to work on this endeavor is exciting, motivating, and deeply appreciated. Bravo! 👏

@avaris: I like your excellent suggestion regarding plugin structure. What changes, if any, would need to be made to Pelican core to support native namespace packages?

@moorepants: I’d be happy to ensure you (and anyone else who wants it) have access to a new render-math repository within the pelican-plugins organization. And the same goes to @nebulousdog and anyone else who wants to create new plugins and/or help migrate existing plugins to the new organization and structure.

Speaking of structure, I think now is the perfect time to talk about other standard conventions we might want to codify before beginning the process of adding plugin repositories to the new pelican-plugins organization. @FedericoCeratto helpfully kicked off that discussion in #436, and I just now noticed that @nebulousdog took the initiative and created an example Pelican plugin template. Awesome! 😁

I’d like to suggest taking that one step further… For some time now I have been thinking about creating a cookiecutter-pelican-plugin repository that contains a template for creating Pelican plugins, replete with the aforementioned standardized conventions on which we as a community have reached consensus. So rather than forking an example plugin template, changing the example project name in various places, et cetera, the CookieCutter template could produce a well-scaffolded plugin with minimum effort. Much of the work @nebulousdog did could be repurposed in CookieCutter form, as well as incorporating other agreed-upon standardization and conventions.

@ionelmc, @nebulousdog, and others have raised the question of how to handle release management. My strong preference would be for plugin packages to be automatically published to PyPI. It just so happens that I developed a solution that does exactly that, AutoPub, which is currently deployed for Pelican core and which has already auto-published multiple Pelican releases with aplomb. 🎉 So to answer the question of how to handle PyPI user accounts in this new scenario… At a minimum, adding the autopub PyPI user account as an Owner for any Pelican plugin package registered on PyPI would ensure that anyone with a commit bit for that plugin’s repository would be able to publish new package version releases simply by tapping the “Merge Pull Request” button. 🚀

What does everyone think about these ideas so far?

[moorepants]

Those ideas sound great. I support them all.

I would like to suggest that we allow plugins to migrate now even if they don't necessarily fit a future template; we can always update them later. The efforts should be in parallel (migrating plugins and developing the infrastructure templates/tools/etc), otherwise it will slow any momentum to have to wait on various infrastructure to develop. Also migrating some plugins will help us understand what infrastructure is needed and what might constitute best practices.

At the minimum we can prune a plugins directory, push that directory to the plugin org as a new repo, and replace the directory in this repo with a submodule link. That would keep everything the way it is but start the separation. Then PRs can be opened on the new repos to turn it into a proper tested package.

The conda-forge community has a beautiful, mostly automated, system in place for submitting on a per repo basis with a common testing, linting, and publishing system. Ideas and even some tools could be used to do something similar for this collection of plugin repositories.

[dreamalligator]

So rather than forking an example plugin template, changing the example project name in various places, et cetera, the CookieCutter template could produce a well-scaffolded plugin with minimum effort.

great! That was the main clunky thing I was seeing in my attempt at creating a minimum structure. I haven't used cookiecutter before, so I'll do some reading :)

[justinmayer]

Since I posted my comment yesterday, I've been working on a CookieCutter template, the initial version of which I just pushed: https://github.com/getpelican/cookiecutter-pelican-plugin

It is still very bare-bones, but it's a start. To test it out, install CookieCutter and then:

cookiecutter gh:getpelican/cookiecutter-pelican-plugin

If you don't want to install CookieCutter, but you have Pipx installed, you can run CookieCutter dynamically via:

pipx run cookiecutter gh:getpelican/cookiecutter-pelican-plugin

You can use CLI context arguments to override the defaults specified in cookiecutter.json, appending key/value pairs such as the following to the above invocations:

[…] plugin_name="Render Math" dev_status="6 - Mature"

[justinmayer]

When plugins are moved over to https://github.com/pelican-plugins, what do we think the standard naming convention should be for plugins? My inclination is to use dashes instead of underscores, only using underscores when they are necessary (e.g., for Python package names). To use the Render Math plugin as an example, the current CookieCutter template generates the following values:

plugin_name [Plugin to My World]: Render Math
repo_name [render-math]:
package_name [render_math]:
distribution_name [pelican-render-math]:
repo_url [https://github.com/pelican-plugins/render-math]:

So other than source-code-level names, everything consistently uses dashes, including the pelican-prefixed PyPI distribution name.

What does everyone think?

[moorepants]

If all of the plugins are turned into packages of a python distribution, then the distribution could be a new name that follows a new guideline, but the package name should probably stay as it was for backwards compatibility sake.

So render math would work like this:

$ pip install render-math
$ python
>>> import render_math

You can require that the package name matches the distribution name for new packages, but I would be very hesitant about breaking everyone's pelican config files.

[dreamalligator]

re #425 (comment): I think this looks good. Besides directories being named with underscores, are there any other exceptions? How does the ideal import look like?

[avaris]

If the namespace idea can become reality, actual import would look like:

import pelican.plugins.render_math

We can do some magic in pelican to keep PLUGINS = ['render_math'] working (instead of 'pelican.plugins.render_math').

If people want to switch, they need to move from this collective repo to individually installing plugins. Some adjustment is already needed in that case, so I'm OK with having people modify their settings a bit.

[justinmayer]

Random thought… Could Pelican core be modified to discover plugins within the pelican.plugins.* namespace and automatically load them, such that the PLUGINS setting is no longer required? If so, it seems to me that would be a better end-user experience than mandating everyone manually add entries to PLUGINS = ['plugin_1', 'plugin_2'] every time they Pip-install a plugin.

[avaris]

Yes, it is possible. I made a simple example for testing. See: https://github.com/avaris/namespace_plugins

So, people would manage plugins by installing/uninstalling them? Sounds good. We would still need PLUGINS for the ones that do not adhere to this schema.

[ionelmc]

Take a look at how pytest does the plugins (setuptools entrypoints discovery). Tho plugin ordering and blacklisting is tricky.

Regarding the plugin cookiecutter - I'd just fork an existing cookiecutter template and avoid solving again all the packaging and boilerplate (I hope you find one with a src layout 😆)

[justinmayer]

@ionelmc: I know it's been a long time since you originally submitted this issue. Thanks for chiming in! 😁

I actually referred to your PyLibrary template and corresponding Nameless repository when putting together my initial plugin template draft. Seeing such a detailed example was extremely helpful, so many thanks for all your work in those repositories!

As a general practice, I often prefer to start with an empty slate and add bits incrementally as needed, which is why I didn't fork an existing template. For example, I wanted to experiment with Poetry and pyproject.toml, CI systems besides Travis, et cetera. And it can take time to understand the underlying rationale and purpose when you're looking at an existing template, and what all the various bits are for (such as ci/bootstrap.py). So rather than inherit everything without needing/understanding it, it's just easier for me to build incrementally.

I've read the long src/ layout discussion, as well as the many linked articles within, but it's not entirely clear to me how that applies in a namespaced Pelican plugin context. I'm not saying that it doesn't apply — only that the applicability eludes me. 😊

[justinmayer]

@avaris said:

So, people would manage plugins by installing/uninstalling them? Sounds good. We would still need PLUGINS for the ones that do not adhere to this schema.

I think perhaps the PLUGINS setting still has some life left, even post-migration. If that setting isn't found, Pelican could auto-discover and enable any installed plugins it finds. If the setting is found, then Pelican would only enable the plugins found in that list.

This has the side benefit of potentially allowing for plugin load ordering in a future version of Pelican. @avaris reminded me that plugins are currently supported via Blinker, which does not support ordering. If we ever switch to another plugin mechanism that supports ordering, we could load them in the specified order, use plugin loading priorities, et cetera.

[MinchinWeb]

I think auto-discovery of plugins is good, but it's probably also helpful to have a mechanism to explicitly either enable or disable plugins. This will also make the transition easier, as plugins that haven't been moved to the new namespace will remain supported as well.

Another advantage: Good form is to use a virtual environment, but on Windows I don't have to worry about breaking the system Python so I sometimes get a little lazy. Under the current setup (with an explicitly defined plugin list), I can generate three different Pelican sites (each with a different set of plugins) without having to much around with virtual environments.

---

I do like the idea of proposing a namespace for plugins, as I expect a couple would organically develop otherwise; hopefully an "official" namespace will help avoid confusion.

However, namespace plugins are a little esoteric (just from lack of common use) and so I worry a little about them breaking in unfamiliar ways. Particularly, I worry a little about having the plugin namespace be within pelican. and someone making a misbehaving plugin that hides or breaks everything within the pelican namespace, and thus pelican (the program) as well; I think this would be painful for a new user to debug. To this end, would it be better to have the designate namespace be "beside" pelican, rather than within it? I.e. propose pelican_plugins.<plugin_name> (or something similar)?

[justinmayer]

@MinchinWeb: I'm not very familiar with namespace packages nor do I comprehend the potential threat model, so I'm going to let others address that concern.

[justinmayer]

I made a number of changes to the Pelican plugin template and then, as an initial experiment, I used that template to set up the Simple Footnotes plugin at its new home. There is still plenty of work left to do here, including auto-publishing to PyPI, but it's a decent start.

I also created a new repository for the Render Math plugin, as well as an initial issue to track migrating the work @moorepants has generously performed on that plugin over to the new shared repository: pelican-plugins/render-math#1

[justinmayer]

In the spirit of Hacktoberfest, some Pelican maintainers will be participating in a Pelican sprint this Sunday. One of the goals will be to help move this packaged plugin release plan forward, so please come join us! https://blog.getpelican.com/sprint-2019-fall.html

[MinchinWeb]

@MinchinWeb: I'm not very familiar with namespace packages nor do I comprehend the potential threat model, so I'm going to let others address that concern.

As I think about this, I think the "threat" is someone will accidentally put something in the pelican folder and Python might load that as "pelican", or similarly, someone provides an __init__.py file in the pelican/plugins folder that causes weirdness. The solution might be simple, especially if most plugins are hosted by "us": add two tests: 1) that there is nothing but the plugins folder in the pelican folder, and 2) that the pelican/plugins folder does not contain an __init__.py file.

---

How does one go about moving plugins to the new organization? I have several plugins I maintain that I'd be happy to move over.

[MinchinWeb]

To my previous comment, it seems that I would have to have the ability to create repos to transfer it to the organization (help page).

The repos I would transfer:

• https://github.com/MinchinWeb/minchin.pelican.plugins.cname
• https://github.com/MinchinWeb/minchin.pelican.plugins.summary
• https://github.com/MinchinWeb/minchin.pelican.plugins.post_stats
• https://github.com/MinchinWeb/minchin.pelican.plugins.optimize_images
• https://github.com/MinchinWeb/minchin.pelican.plugins.nojekyll
• https://github.com/MinchinWeb/minchin.pelican.jinja_filters

[oulenz]

If I understand it correctly, the cookiecutter template is intended for new plugins, not for transferring old ones, correct?

Do we want to transfer all plugins tomorrow regardless of how up-to-date they are (and sort that out afterwards), or do we want to be more selective and ensure that all the plugins which we transfer work correctly?

[avaris]

@MinchinWeb True, there are some "threats" to break pelican. Folder structure for a plugin has some restrictions, it needs to be something like this:

plugin1
├── pelican
│   └── plugins
│       └── plugin1
│           ├── __init__.py
│           ├── plugin1.py
│           └── ...
└── setup.py

Yes, you can't put anything else inside pelican or pelican/plugins. Otherwise you risk overriding and potentially breaking things. Could a plugin developer while writing the plugin hit these? Maybe... I'd like to think with good documentation, chance of this will be minimal if not zero. However, I doubt this will ever reach to a user.

I mean, if you accidentally override pelican/__init__.py in a plugin, pelican will flat out won't work at all. I can't think of any other standard pelican module behave any differently when overridden. That's basically a broken plugin at this point. To affect a user, the developer needs to package and publish that in its broken state and then a user needs to install it.

Having a pelican/plugins/__init__.py that's not compatible with namespace requirements would be a lot more tricky issue to debug. But, that's the nature of namespaces. It's still there even if you did pelican_plugins as namespace instead.

[justinmayer]

@MinchinWeb: Many thanks for reaching out. Let's start with MinchinWeb/minchin.pelican.jinja_filters. I just created pelican-plugins/jinja-filters and sent you an invitation. Once you accept it, you should be able to push those 16 commits into the new repository. Once that's completed, we can move on to the others. Sound good? 😸

[justinmayer]

@oulenz said:

If I understand it correctly, the cookiecutter template is intended for new plugins, not for transferring old ones, correct?

I intended it for both for new plugins as well as a model for how we might re-format existing plugins, benefiting from the consistency that comes from applying a shared set of best practices. So to take the Simple Footnotes plugin as an example, after importing the relevant commits from the legacy monolithic repository, I applied all of the enhancements and formatting conventions that would have been present had I generated the plugin from the template.

Do we want to transfer all plugins tomorrow regardless of how up-to-date they are (and sort that out afterwards), or do we want to be more selective and ensure that all the plugins which we transfer work correctly?

The latter. I would much rather take our time and feel good about the state of each plugin as we add them.

[oulenz]

One tip. If you clone this repo you can do this:

git filter-branch --prune-empty --subdirectory-filter render_math/ master

to retain the git history of that plugin. You just change the directory from render_math to the plugin you are converting.

In case others find it useful, I've used to following steps to transfer the more-categories plugin. (Replace <plugin-name>.)

-1. Choose a local directory where you want pelican plugins to reside (distinct from the directory containing the old pelican-plugins repository).

0. Create the new repository for <plugin-name> in the pelican-plugins organisation.
1. git clone git@github.com:getpelican/pelican-plugins.git <plugin-name>
   1.1. cd <plugin-name>
2. git filter-branch --prune-empty --subdirectory-filter <plugin_name>/ master
3. git remote set-url origin git@github.com:pelican-plugins/<plugin-name>.git
4. If the plugin did not have a separate license, re-add the license file from the pelican-plugins repo (minus the top four lines)
5. Move all files except the readme and the license to ./pelican/plugins
6. (Further modifications, tbd)

[moorepants]

Don't forget adding the AGPL license too. That is only in the top level of the pelican-plugins repo.

[oulenz]

Thanks, I've edited the list. (I guess new templates are free to choose a different license.)

[moorepants]

I guess new templates are free to choose a different license.

If you mean new plugins, yes. I would think so. Maybe there should be a recommended license though, to help ensure compatibility with pelican (if that's needed).

We'll have to maintain the AGPL license for any that are copied into new repos.

[kdeldycke]

For the record, there is a very detailed article on how to migrate Pelican plugins to their own independent packages at: https://blog.getpelican.com/namespace-plugin-migration.html