Best way to deduplicate relationship tuples?

[huttotw]

We are planning to migrate our system to use Keto in production, so we will need to backfill every permission and group that we currently have modeled in our old system.

This will be a multi-step migration for us (lots of entities, large system), so we are not planning to have a "one-and-done" type migration. It will need to be run several times in order to capture all of our current permission relations. Furthermore, we are looking to handle the case where (for whatever reason) permissions get out of sync with our domain model.

We are looking at this as almost like a re-indexing problem. You try to update the index on write to the entity (and are mostly successful), but it may fail, a developer may forget to update the index, etc. but we are brought back in sync every night with a cron job 🎉 .

It looks like we can use the query relationship tuples API to list the subjects that we are interested in, then deduplicate records from there.

1. Is this approach reasonable? If not, any advice?

[zepatrik]

From a fist glance

we are looking to handle the case where (for whatever reason) permissions get out of sync with our domain model

try to update the index on write to the entity

are the key points here. I would try to run both systems in parallel for some time and use the old system for a "backup" of stuff not yet transferred.
One question of course is how to handle permission revocation in such a scenario, because you will have to ensure that that always happens in both systems, or none, to stay consistent.
Permission granting is a bit more forgiving there, because you can just or the two responses you get.

Alternatively maybe there is a way to also transfer all permissions for a user at once and somehow mark which permission backend is responsible. In such a case it might take slightly more time to update everything, so you might not want to do it on user interaction but probably as a background job. The key point here would be to enforce and ensure atomic operations so either everything works or is reverted. Therefore I also recommend using the gRPC API that allows you to transact multiple relation tuples.

Did that help you already?

[vinckr]

Hey @huttotw , just checking in if your problem still persists.
I will mark this as answered for now, but feel free to come back to the discussion any time.
Thanks a ton 🐝

[huttotw]

@vinckr yes, feel free to close.

We ended up migrating one entity at a time (to keep the work load reasonable) using the gRPC API and it seems to have worked well. We have been using Keto in production for about 7 months and haven't had any problems.

[vinckr]

That is good to hear.
Since you are in production, would you like to be featured as adopter on our README?
Feel free to reach out to me if that would be interesting to you.