Repository navigation
Conversation
First commit of #88. Describes the file format before changing any code, so the rest of the work has a written contract to implement against. Two schemas under src/repoplone/schemas/, both Draft 2020-12: - repository-v1.json describes what repoplone accepts today. It is deliberately permissive -- unknown keys pass, so a repository that works now cannot be failed by the schema. It tolerates the [cookieplone] table that cookieplone writes into generated files, but rejects the spec 2 [[package]] array. - repository-v2.json describes the flat [[package]] array, with `type` as a discriminator. Per-type key validity falls out of that: plone_versions on a node package, or base_package on a generic one, are errors rather than keys that are silently ignored. node-aurora has no base package default, so it must declare one. The schemas gate shape only. Duplicate package names, the one-primary-per- family rule and paths existing on disk stay in Python, where the error messages can be good. Both are meant for collective/pytest-jsonschema, which loads a schema by file name. Until a release there bundles them, the tests validate with jsonschema directly, added to the test dependency group -- repoplone gains no runtime dependency. Fixtures live in tests/_resources/repository_toml/, one per rule, and the suite derives its parametrize sets from the filenames: a fixture carrying "invalid" must be rejected, every other must be accepted. A coverage test fails if a fixture is left out of both sets. No behavior change: nothing in src/ reads the schemas yet.
That release bundles repository-v1.json and repository-v2.json under the same names this repository uses, so the two copies can now be compared and the published one exercised. Both copies stay: repoplone reads its own at runtime, and the plugin serves the published one to other projects. A drift test asserts they are equal, which is the only thing that catches one being edited without the other. They were byte-identical when this landed. Direct jsonschema validation stays too. The plugin's fixtures answer with a bare boolean, while iter_errors says which rule a fixture broke -- worth keeping for the invalid-fixture matrix, where "it failed" is not the same as "it failed for the right reason". The new plugin tests are smoke tests: they prove the schemas work through pytest-jsonschema's own loader and validator, not only through ours.
Second commit of #88. Preparatory refactor: spec 2 lets a repository declare several packages, so nothing may reach the package it operates on by reading settings.backend or settings.frontend. Package construction: _build_backend_package and _build_frontend_package take one raw package table instead of the whole Dynaconf object, so the spec 2 parser can call them once per entry. get_backend and get_frontend keep their signatures and are now thin wrappers. Release and changelog helpers take the package as an argument: release_backend(settings, package, version, dry_run) release_frontend(settings, package, project_version, dry_run) update_backend_changelog(settings, package, draft, version) update_frontend_changelog(settings, package, draft, version) stamp_volto_version(settings, package) Callers pass settings.backend or settings.frontend, so behaviour is unchanged. The changelog helpers now read package.towncrier rather than looking the section up as settings.towncrier.backend, which drops their dependency on the two fixed section ids -- spec 2 adds one section per package. The frontend workspace root was derived two different ways: get_frontend used <root>/frontend, while the changelog copy, the distribution metadata and the mrs.developer updates walked two levels up from the package path. For the frontend/packages/<name> layout both agree, and that is the only layout the fixtures cover. They disagree elsewhere. A package declared as path = "frontend" sits one level below the root, so walking up twice lands on the repository's *parent* -- the frontend CHANGELOG.md copy and distribution.json would be written outside the repository. One helper, utils._path.frontend_root, now serves every call site and falls back to <root>/frontend when walking up leaves the repository. It lives in the leaf path module so the dependency modules can import it without a cycle. The new test pins all three layouts, including the two that previously disagreed. No fixture changed: the existing suite is what shows nothing moved.
Third commit of #88. A top-level spec_version selects the specification; spec 2 replaces the component tables with a flat [[package]] array whose entries carry a type. settings/spec.py holds what a JSON Schema cannot express: which spec a file selects, the rules spanning several packages, and messages naming the entry at fault. Its errors are RepoPloneException subclasses, so the CLI already prints them as "Error: ..." and exits 1. A package type encodes two things. The language prefix decides mechanics and is what packages group by -- the python and node families. The ecosystem suffix decides which extra keys apply and what base package is assumed. The generic python and node types are the same mechanics with the extras off, which is what a repository releasing a single plain package needs (#87). Spec 1 files are read into the same list, tagged python-plone and node-volto, so everything downstream works the same for both specs. settings.backend and settings.frontend stay dataclass fields holding the family's primary package, so project release hooks, `settings dump` and roughly sixty call sites are untouched; a family with no packages still answers with the disabled placeholder built from the shipped defaults. Two details the fixtures forced out, both instances of defaults being replaced rather than merged when a file declares a section: - settings/default.toml always preloads an empty [backend.package], so a spec 2 file appears to declare one. Only a package carrying a name was written by a person, which is the same signal `enabled` already uses. - repository.compose was read as an attribute, so a file that declares [repository] without it raised a dynaconf AccessError. A plain Python package ships no compose files, so it is now optional. This only affects files that used to crash. Two fixture projects: fake-multi-package, shaped like pas-plugins-identity with two packages per family, and fake-root-package, the single root-level Python package with no frontend that #87 needs.
Fourth commit of #88. The release steps now loop over their family's packages in document order, and follow the families in name: release_backend is release_python, release_frontend is release_node. The step modules were renamed to match. Those ids are user-facing -- they appear in [repository.release].steps, in registry function bindings and in --start-step -- so the rename comes with a path that breaks nothing. Spec 1 files accept either spelling. Spec 2 rejects the old one, naming the replacement and pointing at `settings migrate`, which is how spec 2 treats every other legacy name. --start-step always accepts both: typing the old name is muscle memory, not a configuration error. Towncrier sections follow the families too: `python` and `node` for the primaries, `python:<name>` for the rest. TowncrierSettings.get() reaches the ids that are not attribute names, and `backend` / `frontend` stay as aliases so project hooks reading settings.towncrier.backend keep working. Section titles are what people read in CHANGELOG.md, so they are spec-gated. Spec 2 uses the package name -- the only thing that reads well once a family holds several packages -- while spec 1 keeps Backend and Frontend, so upgrading repoplone never rewrites an existing project's headings. An explicit per-package `section` key overrides both. Authentication is a property of a registry, not of a package, so it is now checked once per family that publishes anything. Two collisions closed, both of which reported the same change twice: - The frontend-wide CHANGELOG.md copy now runs for the primary node package only. With several node packages the last one to run would have won. - A package at the repository root points its changelog at the repository changelog, so its own towncrier build writes that file. The aggregated entry now skips such a package. This was the open question the plan left for this commit; running it showed the change appearing twice under two headings, so the answer was yes, it collides. Also: the shared `settings` test fixture never busted the cwd cache, so it answered with whichever project a previous test had chdir-ed into. Adding a test module that loads a different project is what exposed it.
Fifth commit of #88, closing the issue. `repoplone settings migrate` rewrites a spec 1 repository.toml into spec 2: it adds spec_version, turns each component's package table into a [[package]] entry with its type and primary flag, normalizes compose to a list, drops the deprecated keys and renames the release steps. It validates the result against the spec 2 schema before writing anything, reports every change, and refuses a file that already declares a spec version. --dry-run prints instead of writing. It goes through tomlkit, so comments and formatting survive -- a config file is something a person reads. One case does not survive: TOML stores a comment written to introduce a table inside the *preceding* table, so a comment sitting between [frontend.package] and the next table is removed with the component. tomlkit cannot place an unkeyed item at a chosen position, and appending it puts the comment below whatever it introduced, which is worse than losing it. Such comments are reported instead, for the author to restore where they meant them. `versions current` reports one row per package rather than one per family, and `deps upgrade` / `constraints` fail with a clear message for a generic python or node package, which builds on no ecosystem and has no version to track. `deps stamp-volto-version` takes --package for a repository with several node packages. Both commands keep spec 1 output byte-identical: the versions table still says Backend and Frontend for a spec 1 file, the same rule the changelog headings follow. The two pre-existing tests asserting those labels are untouched, which is what shows it. README documents spec 2: the package array, the type table with both axes, families and the primary package, per-package options, a table of what changed from spec 1, the schemas, and the migration guide. The old specification section is now labelled Specification 1. Two details worth naming: - The dry-run output is echoed raw rather than through the rich console, which reads [repository] and [[package]] as markup tags and prints the file without its table headers. - `settings dump` gains spec_version and packages; backend and frontend still hold each family's primary package, so nothing reading them had to change.
Sixth and last commit of #88, the optional one. Restarting with --start-step re-runs a whole step, and a step now covers every package of its family. Without a check, the packages that reached PyPI or npm before the failure would be uploaded again and rejected -- turning a restart into a second failure rather than a resumption. That hazard existed before, but looping over a family makes it the normal case rather than the exception. already_published() asks the registry that matches the package's family, so the guard follows the same language prefix everything else does. The check runs before the version and changelog writes, not just before the upload. Rebuilding a changelog whose news fragments were consumed by the first run adds a second, empty entry, so a package that is already published is skipped whole. An unreachable registry answers False. Publishing and letting the registry reject a duplicate is a better failure than skipping a package that was never published, which would leave it unreleased while the run reports success. Dry runs never ask, and neither do packages with publish = false. The test that pins the skip was checked against a neutralised guard -- via monkeypatch, so nothing was mutated on disk -- to confirm it fails when the guard does not fire, rather than passing for some unrelated reason.
The one thing left unresolved in #88. node-aurora shipped with base_package required, because repoplone must not guess what an Aurora package builds on. With the answer settled it becomes a default like the other ecosystem types, and the key is optional again. This puts repository-v2.json ahead of the copy pytest-jsonschema 1.1.0 ships, which still requires the key. Rather than let that expected mismatch hide any *other* divergence, the drift check is now two tests: the spec 2 equality is a strict xfail naming the pending change, and a second test pins exactly how the two copies differ -- one conditional branch, everything else identical. The same marker covers the fixture the published schema still rejects. Strict xfail means these turn into failures the moment a pytest-jsonschema release carries the change, so the markers cannot be forgotten.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds specification 2 of
repository.toml, which lets a repository declare several packages, of several kinds.Closes #88
The format
A top-level
spec_versionselects the specification. A file without it is a specification 1 file and behaves exactly as before — that is the compatibility promise this PR keeps throughout.Specification 2 replaces the
[backend.package]and[frontend.package]tables with one flat array:typehas two axesThe language prefix decides mechanics — how a package is built, versioned and published. The ecosystem suffix decides which extra options apply and what base package is assumed.
typepython-plonepythonbase_package(Products.CMFPlone),python_version(s),plone_versionspythonpythonpython_version(s)node-voltonodebase_package(@plone/volto)node-auroranodebase_package(@plone/aurora)nodenodeReading it this way makes behaviour derivable rather than a table of special cases, and makes the generic types nearly free: they are the existing mechanics with the ecosystem extras switched off.
Families and the primary package
Packages group into two families by language prefix. Release-level work — version bump, changelog, build, publish — loops over every package of a family. Component-level work — base package and its constraints,
pyproject.tomlresolution,mrs.developer.json, lockfiles — uses the family's primary package, markedprimary = trueor, failing that, the first one in the file.settings.backendandsettings.frontendstill answer, with each family's primary package, so project release hooks andsettings dumpare unaffected.This makes #87 a first-class case
A repository that releases a single Python package living at its own root — repoplone itself, cookieplone — is now expressible without pretending to be a "backend" with no counterpart:
Breaking-ish changes, and how they are handled
Everything here keeps specification 1 files working unchanged. The two user-facing renames are gated on the spec version:
release_backend/release_frontendstill canonical; new names also acceptedrelease_python/release_node; an old id errors, naming the replacementBackend/Frontend, unchangedsectionversions currentlabelsBackend/Frontend, unchanged--start-stepaccepts both spellings whatever the file says: typing the old name is muscle memory, not a configuration error.Migrating is the deliberate moment a project's changelog headings change.
settings migratereports it rather than quietly pinning the old behaviour into the new file.repoplone settings migrateRewrites a specification 1 file in place, through tomlkit so comments and formatting survive, validating the result against the specification 2 schema before writing.
--dry-runprints instead.One case it cannot preserve: TOML stores a comment introducing the next table inside the preceding one, so a comment between
[frontend.package]and the following table goes with the removed component. tomlkit cannot place an unkeyed item at a chosen position, and appending it puts the comment below whatever it introduced — worse than losing it. Those comments are reported instead, so the author can restore them where they meant.JSON Schemas
Both specifications are described by JSON Schemas, shipped in
repoplone/schemas/and published through pytest-jsonschema asrepository-v1andrepository-v2.typeacts as a discriminator, so per-type key validity falls out of the schema:plone_versionson anode-voltopackage is an error rather than a silently ignored key.They gate shape only. Duplicate package names, one-primary-per-family and paths existing on disk stay in Python, where the error messages can name the offending package.
Note for review:
repository-v2.jsonhere is one branch ahead of pytest-jsonschema 1.1.0, which still requiresbase_packagefornode-aurora. Two strictxfails record that, and a second test pins exactly how the copies differ so no other divergence can hide behind the expected one. A pytest-jsonschema PR plus release will turn those xfails into failures, which is the signal to drop them.Bugs found and fixed along the way
Each was surfaced by a new fixture, not by speculation:
CHANGELOG.md, under two headings — its own towncrier build and the aggregated entry both wrote that file. The aggregated entry now skips a package that owns the repository changelog.repository.composewas read as an attribute, so a file declaring[repository]without it raised a dynaconfAccessErrorrather than meaning "no compose files". A plain Python package ships none.<root>/frontendin one place, "walk two levels up" in four others. They agree forfrontend/packages/<name>and disagree elsewhere: forpath = "frontend"the second lands on the repository's parent, writingCHANGELOG.mdanddistribution.jsonoutside the repo. One helper now serves every call site.CHANGELOG.mdcopy ran for every node package; with more than one the last would win. It is now the primary package only.settingstest fixture never busted the cwd cache, so it could answer with whichever project ran before it.Restart safety
--start-step release_pythonre-runs a whole step, and a step now covers a whole family. A package already on PyPI or npm is skipped, so a restart resumes rather than failing on a duplicate upload. An unreachable registry answers "not published": being rejected by the registry is a better failure than silently skipping a package that was never released.Commits
Each leaves the suite green and can be read on its own.
ed2680014ab9ee989ce93108c7d770d37620d6205esettings migrate, command updates, README74a38804cac54anode-auroradefaults to@plone/auroraTesting
669 passed, 2 xfailed.
make lintanduvx mypy srcclean.Two fixture projects were added:
fake-multi-package, shaped likepas-plugins-identitywith two packages in each family, andfake-root-package, the single root-level Python package #87 needs.The proof that specification 1 is untouched is that the pre-existing tests asserting
Backend/Frontendlabels and headings were not edited.Out of scope