Format packs #3129

Is your feature request related to a problem? Please describe.

Applications for use within governments need country-specific formats: most countries have their own set of formats that are needed to validate information correctly. For example, the Netherlands has the BSN and the KvK-nummer, where Germany has the Steuer-ID and the Handelsregisternummer. On the other hand, some data types exist everywhere but differ per country, such as the notation of phone numbers or the format of postal codes.

Describe the solution you'd like

To make advanced validation of values possible, #3128 defines how custom validators can be added. To make formats that are commonly used within a country easier to use, we create country packs containing them, starting with the Netherlands and Germany.

We extend the table defined in #3128 with a source column, with the values core for core formats, pack:{name} for format packs (for country packs the name is usually the ISO country code, such as nl or de) and custom for formats an admin defines.

Core formats are read-only by definition and can only be updated with Nextcloud Tables, while an administrator can update custom and pack formats by installing a new version of a pack or by editing the pack manually.

When an imported format has a technical name that already exists with a different definition, the frontend asks whether to override or keep the existing format. The backend answers such an import with a 409 Conflict, and the next request carries a query parameter per format with the choice made.

When the definition of a format changes, the rule of #3128 applies: existing data remains readable, and operators can see the number of values that conflict with the new format. Removing a format is only possible after the operator has seen how many columns will break; those columns are then marked as broken (also as described in #3128).

A format that is used by a column travels with its definition in JSON Schema (x-tables-format-definition) and in manifests, so an import recreates missing formats.

The starter packs are included in the application as JSON files that an admin can import with a single click.

Pack files may reuse other formats with a {"sameAs": "<technical name>"} block. The import resolves such a block into a copy of the referenced rule, so stored definitions and exports stay self-contained. Every rule in a pack has a version, and Tables stores the imported version per rule. When the shipped version of a country pack contains a newer version of a rule, an "update available" message is shown.

Regexes in custom formats are checked by both PHP and JS when saved: the admin UI compiles them as a JS RegExp with the u flag, and the server checks them with preg_match. A regex is only saved when both accept it. Because imports through occ, JSON Schema or manifests have no browser, the server also refuses constructs that PCRE accepts but JavaScript does not, such as possessive quantifiers, atomic groups, \A/\z, (?P<…>), POSIX classes, inline flags and \x{…}; this applies to every regex, including those in packs. Browsers only check values up to a fixed length (at most 2048 characters for uri and url, but mostly limited to 255 characters), so that the browser cannot freeze, as JavaScript has no backtrack limit.

Pack JSON files carry an English label, which is translated with $l->t(): once a translation is added in l10n, the label appears translated.

Acceptance Criteria

• An admin adds a regex format and a JSON Logic format; both validate on the server and in the browser.
• The NL and DE starter packs import; nl.bsn, de.ust-idnr and iban columns validate.
• A table with a custom format exported and imported on a clean instance has that format there.
• A column whose format was deleted accepts writes and reports broken: true.
• Core formats cannot be edited or deleted; a non-admin cannot define, import or remove formats.
• Importing a pack with a conflicting definition shows the conflict in the preview; keep and replace both work per format.
• Changing a rule in use shows the failing count first; after saving, existing values stay readable and failing writes are refused.
• Removing a pack removes exactly the formats with its source.
• A catastrophic regex counts as a failure and does not hang the request.
• A regex that only one of the two dialects accepts (e.g. a++ or \A…\z) is refused when saved.
• A sameAs rule is stored as a copy of the referenced rule; removing the referenced format leaves the copy working.
• With a newer version of a rule in the shipped pack than the imported one, the admin settings show "update available", and the preview labels the conflict as an update.
• A value longer than the browser's check length is still validated, and rejected if invalid, by the server.
• With the user's language set to German or Dutch, shipped pack formats show translated labels; a custom format shows its label as entered.
• A pack imported with occ that contains a regex JavaScript cannot compile is refused, naming the format.

Describe alternatives you've considered

As an alternative, we considered code plugins instead of JSON Logic definitions, as well as shipping packs within Tables switched on by default.

We also considered shipping definitions only via manifests, and failing closed on unknown formats or dropping them.

We also considered formats travelling by name only, but using technical names makes formats more interchangeable, as display names will inevitably differ in capitalisation or spelling.

Furthermore, we considered always preferring a newer definition over an older one, preferring local definitions over incoming ones, or simply refusing an import if a format already exists.

Another consideration was to let application owners define formats scoped to their application, but this was rejected because formats should be widely shareable, with the technical name as their identifier. Making core formats editable was rejected for the same reason.

Changing a rule without any warning, or refusing the change while values fail, was rejected for data consistency: although it is rare, a format sometimes has to be updated, and existing data should not be affected by that.

A final consideration was an is_core flag, or recognising core formats by a technical name without dots (with dots for pack namespaces), but that would rule out custom formats.

Additional context

• Milestone M1: data foundation
• Sprint 1, 12 to 25 October: the data layer cluster (widens the row "the formats registry with the first twelve validators")
• Functionality research Nextcloud Tables – OpenRegister ("a way for people to import formats (and corresponding validators) into their installation")
• JSON Logic
• BZSt notice on the Wirtschafts-Identifikationsnummer, BStBl 2024 I p. 1269
• python-stdnum (test oracle for the NL and DE checks)

Starter packs

• Starter pack NL (link once published): nl.bsn, nl.rsin, nl.btw-id, nl.kvk, nl.postcode, nl.phone
• Starter pack DE (link once published): de.postcode, de.phone, de.steuer-id, de.ust-idnr, de.w-idnr, de.handelsregister
• 74 test vectors pass in PHP and JS (oracle python-stdnum 2.2; link once published)
• Every format carries an example that passes its own rule
• Assumption: de.w-idnr checks the USt-IdNr check digit on its 9 digits (BStBl 2024 I p. 1269 says only "entspricht im Aufbau der USt-IdNr")

Relevant files:

• the format table from Formats as subtypes, validated by format definitions #3128, plus a migration adding the source column
• the admin settings section
• an occ command for pack import and removal
• the starter pack JSON files in the app

Related issues

• Formats as subtypes, validated by format definitions #3128 (formats as subtypes)
• JSON Schema import/export #3130 (JSON Schema import/export)