Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
f72f2b6
feat(cmdb-import): expose the import mapping read-only to the admin
WilcoLouwerse Oct 12, 2026
0108ef0
fix(cmdb-import): answer the mapping's failure reason as one bounded …
WilcoLouwerse Oct 12, 2026
80dc551
docs(openspec): state the bound on the mapping's failure reason
WilcoLouwerse Oct 12, 2026
ecc9c8b
fix(cmdb-import): expose the import's write rules in the mapping over…
WilcoLouwerse Oct 12, 2026
df28777
fix(cmdb-import): keep create-only defaults out of createOnlyFields
WilcoLouwerse Oct 12, 2026
6ecb274
fix(cmdb-import): keep a mapped field with a create default in create…
WilcoLouwerse Oct 12, 2026
024122b
feat(cmdb-import): run the mapping packs stored in OpenRegister
WilcoLouwerse Oct 12, 2026
f538f29
fix(cmdb-import): limit edited packs to vetted fields and run only ha…
WilcoLouwerse Oct 12, 2026
c59713e
docs(cmdb-import): state the vetted-field rule where the loader appli…
WilcoLouwerse Oct 12, 2026
9b64b31
chore(release): bump the version so the CMDB pack seed step runs on u…
WilcoLouwerse Oct 12, 2026
e82c8d2
test(repair): read info.xml as bytes in the seed-step registration test
WilcoLouwerse Oct 13, 2026
d4d7f38
style(e2e): prettier on the CMDB import pack spec
WilcoLouwerse Oct 13, 2026
1fe71a1
chore(cmdb-import): start the TOPdesk import packs at version 1.0.0
WilcoLouwerse Oct 13, 2026
057abc0
feat(cmdb-import): let an edited pack map any plain field of its targ…
WilcoLouwerse Oct 13, 2026
3d8b789
fix(cmdb-import): view the mapping without writing, and never overwri…
WilcoLouwerse Oct 13, 2026
8f07929
fix(cmdb-import): translate both mapping endpoint errors with literal…
WilcoLouwerse Oct 13, 2026
3c81e23
docs(cmdb-import): describe only what this part ships, and what editi…
WilcoLouwerse Oct 13, 2026
e2081bb
fix(cmdb-import): type the import's pack callback as never null
WilcoLouwerse Oct 13, 2026
c797020
fix(cmdb-import): refuse pack mappings of fields other subsystems own
WilcoLouwerse Oct 13, 2026
81b1bc7
fix(cmdb-import): show the mapping the next import runs, and keep a p…
WilcoLouwerse Oct 13, 2026
f7421aa
test(cmdb-import): pin every write method and opening marker of the s…
WilcoLouwerse Oct 13, 2026
6111cb8
docs(cmdb-import): align spec, design and docs with the pack rules an…
WilcoLouwerse Oct 13, 2026
94f976e
test(cmdb-import): put the deleted business-owner pack back when the …
WilcoLouwerse Oct 13, 2026
1004ee7
test(cmdb-import): expect updatePending in the mapping endpoint's pac…
WilcoLouwerse Oct 13, 2026
0a349a8
fix(cmdb-import): fail closed when a register fragment cannot be read
WilcoLouwerse Oct 13, 2026
afaa085
refactor(cmdb-import): read the register's field rules in their own c…
WilcoLouwerse Oct 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@
sheets becomes or updates a module, its vendor, a usage for the municipality
and a contact person for its owner; a repeat import matches on APPID, so it
updates instead of duplicating. See `docs/features/cmdb-import.md`.
- CMDB import: on an OpenRegister with the migration-pack editor
(openregister#4602), the import runs the mapping packs kept in
OpenRegister's migration pack store, where an administrator edits them.
The `SeedCmdbImportPacks` repair step stores them on install and upgrade;
a pack nobody edited follows newer releases, an edited one is kept.
Viewing the mapping writes nothing. An older OpenRegister keeps the packs
that ship with stackiq.

### Changed
- EOL feed: the register and schema slugs the feature reads from are corrected.
Expand Down
12 changes: 11 additions & 1 deletion appinfo/info.xml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Vrij en open source onder de EUPL-licentie.

**Ondersteuning:** Voor ondersteuning, neem contact op via support@conduction.nl. Voor een Service Level Agreement (SLA), neem contact op via sales@conduction.nl.
]]></description>
<version>0.2.5-unstable.20261008120000</version>
<version>0.2.6-unstable.20261012120000</version>
Comment thread
WilcoLouwerse marked this conversation as resolved.
<licence>EUPL-1.2</licence>
<author mail="info@conduction.nl" homepage="https://www.conduction.nl/">Conduction</author>
<namespace>Stackiq</namespace>
Expand Down Expand Up @@ -250,6 +250,14 @@ Vrij en open source onder de EUPL-licentie.
instead of failing.
-->
<step>OCA\Stackiq\Repair\RenameDutchCatalogValues</step>
<!--
Stores the CMDB import's sheet packs in OpenRegister's migration-pack
store, where an administrator edits them; the import reads them from
there. Creates a missing pack and replaces one nobody edited with this
release's shipped pack; an edited pack is kept and reported. Never
throws, and does nothing without OpenRegister's store.
-->
<step>OCA\Stackiq\Repair\SeedCmdbImportPacks</step>
</post-migration>
<!--
FRESH INSTALL. Nextcloud does NOT run post-migration on a first install:
Expand Down Expand Up @@ -324,6 +332,8 @@ Vrij en open source onder de EUPL-licentie.
exact stored value so an admin's own register is never touched. -->
<step>OCA\Stackiq\Repair\RepointEolSyncConfig</step>
<step>OCA\Stackiq\Repair\InitializeSettings</step>
<!-- A fresh install seeds the CMDB import packs too; see <post-migration> above. -->
<step>OCA\Stackiq\Repair\SeedCmdbImportPacks</step>
</install>
</repair-steps>

Expand Down
3 changes: 3 additions & 0 deletions appinfo/routes.php
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,9 @@
// @spec openspec/changes/cmdb-export-import/tasks.md#task-8
['name' => 'cmdbImport#import', 'url' => '/api/cmdb-import', 'verb' => 'POST'],
['name' => 'cmdbImport#cancel', 'url' => '/api/cmdb-import/{operationId}/cancel', 'verb' => 'POST'],
// The mapping the CMDB import uses, read-only — same posture as the import (Nextcloud admins only, CSRF).
// @spec openspec/changes/cmdb-import-mapping-view/specs/cmdb-export-import/spec.md#requirement-the-admin-settings-shall-show-the-mapping-the-import-uses-req-cmdb-020
['name' => 'settings#getCmdbImportMapping', 'url' => '/api/settings/cmdb-import/mapping', 'verb' => 'GET'],

// User Groups management routes
['name' => 'settings#getGenericUserGroups', 'url' => '/api/settings/user-groups/generic', 'verb' => 'GET'],
Expand Down
136 changes: 104 additions & 32 deletions docs/features/cmdb-import.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,10 @@ date stored there, including one set by hand.

### Columns and where they go

This is the mapping as it ships. On an OpenRegister with the migration-pack
editor an administrator can change it there (see
[Viewing and editing the mapping](#viewing-and-editing-the-mapping)).

| Column | Goes to | Rule |
|---|---|---|
| APPID | module external number, and the match key | required, see [Repeat imports](#repeat-imports) |
Expand Down Expand Up @@ -294,7 +298,7 @@ and the section shows the reason and the error code.
| `FIELD_INVALID` | A form field of the request has a value the import does not accept, for example an `updateExisting` or `publish` that is neither `true` nor `false`. The message names the field. | Not reachable from the section; reported for API callers. |
| `UPLOAD_FAILED` | The file reached the server but could not be stored there. | Try again; the Nextcloud log has the details. |
| `MISSING_RECORDS_UNSUPPORTED` | The request asked to mark or remove records missing from the export. Only keeping them is supported. | Not reachable from the section; reported for API callers. |
| `MAPPING_UNAVAILABLE` | OpenRegister's mapping engine is missing, or one of the mapping files is invalid. | Update OpenRegister. If you changed a mapping file, check it against the Nextcloud log. |
| `MAPPING_UNAVAILABLE` | OpenRegister's mapping engine or pack store is missing, or a mapping pack is invalid or breaks a rule of the import. The message names the pack when one is at fault. | Update or enable OpenRegister. If you changed a pack, correct it in OpenRegister; see [Viewing and editing the mapping](#viewing-and-editing-the-mapping). |
| `READER_UNAVAILABLE` | The Excel reader that ships with OpenRegister cannot be loaded. | Make sure OpenRegister is installed and enabled. |
| `NOT_CONFIGURED` | The stackiq register or its schemas cannot be found. | Run **Auto Configure** at the top of the stackiq admin settings. |
| `SCHEMA_OUTDATED` | A stackiq schema lacks a property the import recognises records by, for example `externalKey` on the module schema, or the module schema is older than version 0.3.8: it has no `applicationType`, or `externalKey` lacks the rule that only an administrator may change it. Importing anyway would create every application again, or leave the import key open to every editor. The message names the schema. | Press **Force Update** at the top of the stackiq admin settings to import the register configuration again. |
Expand Down Expand Up @@ -331,35 +335,103 @@ every import:
The section's help text shows the defaults; when the server refuses a file,
the message shows the limit the server applied. A larger file also has to
pass PHP's `upload_max_filesize` and `post_max_size` and the web server's
request size limit. Like the mapping files, the profile is part of the app:
a change made on the server is overwritten by the next app update.

## Adjusting the mapping

The mapping from columns to fields is not in code. It is a set of JSON files
in `lib/Settings/cmdb-import/`, executed by OpenRegister's mapping engine:

| File | What it maps |
request size limit. The profile is part of the app: a change made on the
server is overwritten by the next app update.

## Viewing and editing the mapping

**What you need.** Editing the mapping needs an OpenRegister with the
migration-pack editor, whose pack writes are admin-only and need a CSRF
token (added in [openregister#4602](https://github.com/ConductionNL/openregister/pull/4602);
the first OpenRegister release that contains it). On an older OpenRegister
nothing is stored in OpenRegister and there is nothing to edit: the import
runs the packs that ship with stackiq, and the import result says so.
Changing those shipped files on the server does not last, because the next
app update replaces them. Once the packs have been stored in OpenRegister,
the import does not fall back to the shipped copies: when OpenRegister's
pack store is gone, it refuses with `MAPPING_UNAVAILABLE` until OpenRegister
is back.

The mapping from export columns to stackiq fields is not in code. It is a
set of mapping packs that OpenRegister's mapping engine runs, one per kind of
record the import writes. With the editor, four of them are kept in
OpenRegister, where you can view and change them:

| Pack (in OpenRegister) | What it maps |
|---|---|
| `topdesk-profile.json` | the sheets, the constant each sheet adds to its rows (`Beheer`) and the columns it is known to lack, the match column, the required, date and id columns, the placeholder values that mean empty, the limits, and which pack is used for which target |
| `topdesk-module.json` | a row to the module (hosting model and BBN lookups) |
| `topdesk-manufacturer.json` | "Vendor" to the Supplier organisation |
| `topdesk-municipality.json` | a typed municipality name to a new organisation |
| `topdesk-usage.json` | a row to the usage (status and TIME lookups, dates, annotation) |
| `topdesk-business-owner.json` | the owner columns |

Each pack has a list of `fieldMappings`, one per column: `source` (the column
name in the export), `target` (the field), optionally `required`, and a
`transform` such as `trim`, `date` or a `lookup` with a `map` of export values
to stored values. For example, to accept a new "Applicatiesoort" value, add
it to the `map` of the hosting-model lookup in `topdesk-module.json`:

```json
"Cloud": ["SaaS"]
```

To accept a new "Applicatie Status" value, add it to the `map` of the status
lookup in `topdesk-usage.json`. The packs are checked by OpenRegister when an import
starts; an invalid pack stops the import with `MAPPING_UNAVAILABLE` before
any row is read. A mapping file changed on the server is overwritten by the
next app update, so propose lasting changes to the app itself.
| `stackiq-topdesk-module` | a row to the application (hosting model and BBN lookups, dates) |
| `stackiq-topdesk-manufacturer` | "Vendor" to the Supplier organisation |
| `stackiq-topdesk-usage` | a row to the usage (status and TIME lookups, phase-out date, internal note) |
| `stackiq-topdesk-business-owner` | the owner columns |

When OpenRegister has the editor, stackiq puts these four packs into
OpenRegister when it is installed or updated, and an import stores any that
is missing. From then on the import runs what is saved in OpenRegister. The
mapping endpoint (`GET /index.php/apps/stackiq/api/settings/cmdb-import/mapping`,
Nextcloud admins only) answers, per pack, where the import takes it from
(OpenRegister, or the copy that ships with stackiq), its version and the
link to its editor in OpenRegister. Reading it changes nothing: it does not
store or update a pack.

Each pack has a list of field mappings, one per column: the column name in
the export, the field it goes to, whether it is required, and a
transformation such as trim, a date, or a lookup table of export values to
stored values. For example, to accept a new "Applicatie Status" value, add it
to the lookup table of the status mapping in `stackiq-topdesk-usage`. To
accept a new "Applicatiesoort" value as a hosting model, add it to the lookup
table of the hosting-model mapping in `stackiq-topdesk-module`, for example
`Cloud` → `SaaS`. When a pack maps a column with a date transformation, the
import reads that column as an Excel date.

**What the import refuses.** The import writes past OpenRegister's access
rules, so it checks every pack before it reads a row. It refuses a pack, with
`MAPPING_UNAVAILABLE` and the name and id of the pack, when:

- OpenRegister finds the pack invalid;
- the application name, the APPID (to the source number), the vendor name or
the owner name is no longer mapped, or no longer marked required;
- a mapping or a default writes a field the pack may not write. A pack may
write any plain field of the record it writes (the application, the
organisation or the usage), as the register defines it; the owner pack
writes the name and the role only. Refused are the fields the import sets
itself (the import key, the provider, the consumer, the application of a
usage, the business owner, the type of an organisation: a Supplier stays a
Supplier), every link to another record, files, the publication dates, the
fields the register hides from forms or calculates (the service desk
exchange's keys and stamps, the suggested TIME classification) and the
installed version the service desk owns, and the fields only an
administrator or the catalogue sets, such as an organisation's status and
who registered it. The status and registration
defaults of the vendor and municipality packs stay as shipped.

The mapping endpoint answers the same reason, with the link to the pack's
editor in `details.editUrl`, so you can correct it there and try again.

**What a pack does not change.** Which sheets are read, the APPID and
"Applicatie Naam" columns, the values that mean empty (`NB`), the constant
each sheet adds (`Beheer geregeld: ja` or `nee`), the limits, and which fields
are written only on create or never on an existing record are part of the app
(`lib/Settings/cmdb-import/topdesk-profile.json`) and cannot be changed. The
municipality is mapped from the name you choose for the import, not from the
export; that pack is not stored in OpenRegister.

**Reading more columns.** The import reads only the columns the mapping
names. When you map a new column in a pack, the import reads that column from
then on.

**One mapping per Nextcloud.** The packs in OpenRegister apply to every
import on this Nextcloud, whatever the municipality.

**App updates.** A pack you have not changed follows the app: when an update
ships a newer version, the stored pack is replaced by it (the mapping
endpoint already shows the new pack, with `updatePending` true, before the
next import runs). A pack you changed
is never overwritten, also not when you save it while an import is updating
it. When an update ships a newer version of a pack you changed, the mapping
endpoint reports it (`newerShipped`); compare the two and copy what you need
into your pack in OpenRegister.

**Back to the shipped pack.** Delete the pack in OpenRegister. The next
import, or the next app update, puts the shipped pack back. Do not rename a
pack: the import finds it by its id (such as `stackiq-topdesk-usage`), and
refuses to run while a pack it stored has another id.
8 changes: 6 additions & 2 deletions l10n/en.js
Original file line number Diff line number Diff line change
Expand Up @@ -1046,7 +1046,10 @@ OC.L10N.register(
"The file is larger than the maximum of %s MB.": "The file is larger than the maximum of %s MB.",
"The file is not an Excel workbook (.xlsx).": "The file is not an Excel workbook (.xlsx).",
"The import failed. The details are in the Nextcloud log.": "The import failed. The details are in the Nextcloud log.",
"The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.": "The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.",
"The import mapping cannot run: OpenRegister is missing, or a mapping pack is invalid or unavailable.": "The import mapping cannot run: OpenRegister is missing, or a mapping pack is invalid or unavailable.",
"The import mapping cannot run: the OpenRegister migration pack %s is invalid or cannot be read. The details are in the Nextcloud log.": "The import mapping cannot run: the OpenRegister migration pack %s is invalid or cannot be read. The details are in the Nextcloud log.",
"The import mapping cannot be shown: OpenRegister is missing, or a mapping pack is invalid or unavailable.": "The import mapping cannot be shown: OpenRegister is missing, or a mapping pack is invalid or unavailable.",
"OpenRegister's migration pack store is not available, so this import used the mapping that ships with stackiq.": "OpenRegister's migration pack store is not available, so this import used the mapping that ships with stackiq.",
"The workbook has neither of the sheets %s.": "The workbook has neither of the sheets %s.",
"Column \"%1$s\": %2$s": "Column \"%1$s\": %2$s",
"Optional column \"%s\" not found": "Optional column \"%s\" not found",
Expand Down Expand Up @@ -1132,7 +1135,8 @@ OC.L10N.register(
"The workbook holds more than {count} different texts, the most the import reads.": "The workbook holds more than {count} different texts, the most the import reads.",
"Together, the cells of the workbook reference more than {size} of shared text, the most the import reads.": "Together, the cells of the workbook reference more than {size} of shared text, the most the import reads.",
"In development since": "In development since",
"In use since": "In use since"
"In use since": "In use since",
"The import mapping could not be read. The details are in the Nextcloud log.": "The import mapping could not be read. The details are in the Nextcloud log."
},
"nplurals=2; plural=(n != 1);"
)
8 changes: 6 additions & 2 deletions l10n/en.json
Original file line number Diff line number Diff line change
Expand Up @@ -1045,7 +1045,10 @@
"The file is larger than the maximum of %s MB.": "The file is larger than the maximum of %s MB.",
"The file is not an Excel workbook (.xlsx).": "The file is not an Excel workbook (.xlsx).",
"The import failed. The details are in the Nextcloud log.": "The import failed. The details are in the Nextcloud log.",
"The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.": "The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.",
"The import mapping cannot run: OpenRegister is missing, or a mapping pack is invalid or unavailable.": "The import mapping cannot run: OpenRegister is missing, or a mapping pack is invalid or unavailable.",
"The import mapping cannot run: the OpenRegister migration pack %s is invalid or cannot be read. The details are in the Nextcloud log.": "The import mapping cannot run: the OpenRegister migration pack %s is invalid or cannot be read. The details are in the Nextcloud log.",
"The import mapping cannot be shown: OpenRegister is missing, or a mapping pack is invalid or unavailable.": "The import mapping cannot be shown: OpenRegister is missing, or a mapping pack is invalid or unavailable.",
"OpenRegister's migration pack store is not available, so this import used the mapping that ships with stackiq.": "OpenRegister's migration pack store is not available, so this import used the mapping that ships with stackiq.",
"The workbook has neither of the sheets %s.": "The workbook has neither of the sheets %s.",
"Column \"%1$s\": %2$s": "Column \"%1$s\": %2$s",
"Optional column \"%s\" not found": "Optional column \"%s\" not found",
Expand Down Expand Up @@ -1131,6 +1134,7 @@
"The workbook holds more than {count} different texts, the most the import reads.": "The workbook holds more than {count} different texts, the most the import reads.",
"Together, the cells of the workbook reference more than {size} of shared text, the most the import reads.": "Together, the cells of the workbook reference more than {size} of shared text, the most the import reads.",
"In development since": "In development since",
"In use since": "In use since"
"In use since": "In use since",
"The import mapping could not be read. The details are in the Nextcloud log.": "The import mapping could not be read. The details are in the Nextcloud log."
}
}
Loading
Loading