WIP: Unified annotation versioning

[fm3]

URL of deployed dev instance (used for testing):

• https://___.webknossos.xyz

Steps to test:

• abc

TODOs:

• Mechanism to Revert Editable Mappings
  • Iterators for SegmentToAgglomerate, AgglomerateToGraph
  • How to encode Reverted chunks, agglomerates? → Single Zero-Byte?
    • Check a single zero byte is not a valid proto message
  • Iterate over current version, fetch old version, rewrite
  • Integrate this in Updater or flush Updater before this happens
  • Test
  • Cleanup: Generic Reversion-Aware Iterator? Build on top of VersionedFossilDbIterator?
  • Save perf by skipping fetching content when only version is needed in iterator (different from volume, because here we don’t need to update the segment index)
    • that might need new fossildb api (ListKeysWithVersions)
  • Make the iterator async?
• Annotation proto object
• Annotation-wide update actions
  • add layer
    • becomes update action rather than route
    • after applying updates, a summary is sent to wk if postgres-cached properties change
      • could this lead to annotationId lookups before postgres knows about the new layer?
    • frontend reloads
  • delete layer
  • update layer metadata
  • update annotation metadata (name+description)
  • should “duplicate” also copy update actions? This is currently the case only for editable mappings. Needed for its merge function
  • iron out reversion folds + layer deletions in merge + duplicate?
• Test sandbox annotation
• Unified versioning over layers
  • Route
  • Store Updates
  • Create Annotation
  • Updates need Layer Identifier
  • Apply Updates
    • updates that mutate annotation object
    • updates that mutate tracing objects
    • updates that mutate other stuff
      • volume buckets
      • proofreading
    • Special updates
      • AddSegmentIndex
      • ImportVolumedata
      • makeMappingEditable
      • Merge?
      • RevertToVersion
        • Revert volume buckets
        • Revert skeletons
        • Revert annotation-level properties
        • Revert proofreading fields
        • Handle gone layers (either in previous or target version)
        • What if two of those come in the same update batch?
        • split updates batches by RevertToVersion? What are the intermediate versions?
  • Version assertions
  • lazy apply for volumes and editable mappings?
    • can volume data still be written directly? can there be conflicts?
    • or ditch lazy apply completely? (might be ok with distributed skeletons etc)
    • When to materialize which layer?
      • store materialized layer only sometimes? count its update actions?
  • Ensure no parallel update applying on the same object (async cache?)
    • But parallel update applying should also not be a problem, except for perf (was it only a problem because version wasn’t specified when loading volume buckets during revert?)
    • Perf: Reduce cache mem overhead by removing older versions when newer are requested
    • Perf: When newer is requested, get old one from cache, then apply updates?
  • [ ]
• Search for // TODO
• Frontend
  • Linearized update actions
  • Use new update action route
  • update actions now need actionTracingId
  • updateTracing was renamed → updateVolumeTracing/updateSkeletonTracing
  • Version Restore View; See: WIP: Unified annotation versioning #7917 (comment)
    • For volume actions show which volume layer was edited (if the layer was deleted, show "unknown layer" / layer id)
  • Send latest description + name with update, next to segment/tree stats?
  • enforce revert actions to come in separate update group (has its own version number)
  • same for add layer / delete layer / initialize editable mapping
  • why is newestVersion sometimes called with emptystring annotationId?
• Migration
  • Linearized Updates
  • changed update actions
    • several actions were renamed, e.g. updateTracing was renamed → updateVolumeTracing/updateSkeletonTracing
    • update actions now need actionTracingId
  • Find annotation ids
  • Can we run a first part in the background?
    • on a second run, (re)do only annotations that in postgres have a modified date newer than when first run started
  • editable mapping update actions and arrays used to be stored by mappingName, now annotationId
  • while we’re migrating everything, could we also do Fix or Remove Morton Order in Volume Data Fossil Keys #3546 ?
    • how does it interact with ND data?

Issues:

• fixes Unified Annotation Versioning #7839
• fixes Pass UserToken implicitly around tracingstore + datastore #8029

---

(Please delete unneeded items, merge only when none are left open)

• Updated changelog
• Updated migration guide if applicable
• Updated documentation if applicable
• Adapted wk-libs python client if relevant API parts change
• Removed dev-only changes like prints and application.conf edits
• Considered common edge cases
• Needs datastore update after deployment

[fm3]

The next part of the frontend could be added now: The Restore Older Version flow. The version list view should no longer feature tabs per layer, but instead only one global GET tracings/annotation/:annotationId/updateActionLog route exists with a single list. (I currently adapted the frotnend so that each tab uses this new route)

Also, the revertToVersion update action is no longer layer specific (I guess this may already be covered by philipps changes).

It is required that the revertToVersion update action is the only action in its update group (so that it gets its own version number). Not sure if the frontend already does that.

Since Philipp is out of office, maybe @MichaelBuessemeyer could have a look at this after #6613 is done?

Note that this branch opens fossildb with different column families. I run rm -rf fossildb/data; git co fossildb/data/.gitignore when switching to and from this branch to avoid errors. If you need your fossildb data, you can of course create a backup first.

[MichaelBuessemeyer]

Since Philipp is out of office, maybe @MichaelBuessemeyer could have a look at this after #6613 is done?

Sure :)