Frijya and Frigg Adapters for Migrating Collections, AdminSets and Works to Postgres or Fcrepo 6

[orangewolf]

Overview

First, apologies. The nature of the work has been that we've been upgrading Hyku and testing Hyku against this branch. As such, it's a larger PR than we might normally prefer.

Type of change (for release notes)

This PR includes two primary considerations:

1. Code changes that allow for a "lazy migration" of data (see discussion below).
2. Refactorings that allow for downstream applications to configure behavior instead of override entire files.

Further Discussion

The primary work introduces the Frigga and Freyja metadata adapter; these adapters are configured to:

• Read first from one storage location, and failing to find something read from another (e.g. storage_first.find || storage_second.find if you will)
• Write to the first storage location, ignoring the second storage location.

Through configuration and convention, the Frigga and Freyja adapters write similar looking Solr documents as their ActiveFedora based counterparts.

Frigga's adapter's first storage location is Postgres via Valkyrie, and it's second storage location is fcrepo via ActiveFedora.

Freyja's adapter's first storage location is fcrepo via Valkyrie, and it's second storage location is an fcrepo via ActiveFedora. Note, this is likely Fedora 6 and Fedora 4 respectively.

What this means is that these adapters provide a pathway for minimal downtime during a storage migration; something demonstrated by GBH's AMS project and presented on at Samvera Connect 2023 by @orangewolf.

Guidance for testing, such as acceptance criteria or new user interface behaviors:

There are no known new user behaviors. Instead we're relying on the test suite as well as downstream testing of Hyku migrations to help verify.

@samvera/hyrax-code-reviewers

[tpendragon]

I'd 👍 adding a composite query service like this in Valkyrie if you wanted.

[orangewolf]

seems like a good plan to get through it in Hyrax land and then port it. Since I'm modifying a valkyrie branch to make the shared examples flexible enough for this use case, it should slot over super easy if we get to that point.

[jeremyf]

The goal of double_combo approach is to minimize downtime by allowing for in-place migrations.

Note: In this scenario, I’m using Fedora4 as the name for the legacy persistence layer, and Postgres as a symbolic name for the metadata adapter’s persistence layer; we could just as easily use Fedora4 but for clarifying purposes I’m separating these concepts.

This is done by leveraging a copy on write (CoW) type strategy. Below are two examples:

Given I wrote Work A to Fedora4 via ActiveFedora
And wrote Work A to Solr via ActiveFedora
And did not write Work A to Postgres
When I use Valkyrie to find Work A
Then Valkyrie will first check Postgress for Work A
And Valkyrie will next check Fedora4 for Work A

From the above scenario:

Given I have found Work A via Valkyrie
When I use Valkyrie to save Work A
Then Valkyrie will write to Postgres
And Valkyrie will update the Solr document originally written by ActiveFedora

At the point of completing the write scenario, when we go to find “Work A” via Valkyrie, we’ll first check and find the record in Postgres.

Consider that the Fedora4 repository also has works B, C, D, and E. And all of those are part of Collection Alpha. When rendering the contents of the collection, we leverage the Solr document representations of each of the works. This is done by creating presenters. To avoid presentation drift, we would like for A, B, C, D, and E to have consistent presenters regardless of whether they were persisted via ActiveFedora or Valkyrie.

Put another way, the Presenter for Work A’s solr document written by ActiveFedora and the Presenter for Work A’s solr document written by Valkyrie should have the same methods and each of those methods used for rendering should return the same values.

To reduce duplication, it would be ideal for both of those presenters to in fact be the same presenter. It then follows that we would want to minimize branching logic between documents written via ActiveFedora and those written by Valkyrie; the greatest minimization would be for ActiveFedora and Valkyrie to generate logically equivalent solr documents. I say logically equivalent in that there are administrative benefits for tracking in Solr which documents have been valkyrized (see valkyrie_bsi).

This leads to the aspiration that the has_model_ssim should be stable between a document written via ActiveFedora and one written via Valkyrie.

The has_model_ssim is a foundational attribute in that presenters use this for switching logic and we use it for Solr querying logic. See Hyrax::SolrDocumentBehavior#hydra_model and it’s uses.

[dlpierce]

I think we're getting close. I feel like the changes in dassie to enable Freyja need an on/off env toggle.