Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Akka.Persistence.Query: add support for CurrentPersistenceIds #158

Open
Aaronontheweb opened this issue Aug 5, 2021 · 1 comment
Open

Comments

@Aaronontheweb
Copy link
Member

Is your feature request related to a problem? Please describe.
While adding some integration tests for our Akka.Cluster.Sharding persistence cleanup tool, I ran into a small problem on this PR: petabridge/Akka.Cluster.Sharding.RepairTool#7 - Akka.Persistence.Redis' IReadJournal does not support ICurrentPersistenceIds().

I know we removed it as part of #126 because we thought it'd be too expensive to implement in clustered redis scenarios, but I wonder if that's true.

Describe the solution you'd like
The "all events" and "events by tag" queries are out of the question - they're expensive because they can't effectively be cleaned up in a clustered environment without some kind of inter-node index and we're not database architects so we're not going to build one. Users can use a relational database for that.

But Akka.Persistence requires us to keep track of all used PersistentIds even after the entities have had all of their data purged - we're required to keep those records in perpetuity and therefore cleanup is not an issue.

So, I suppose we can try implementing the ICurrentPersistentIds and IPersistentIds queries in one of two ways:

  1. Scatter-gather to all of the nodes in the cluster each time we need to run one of those queries and just use the built-in key indicies we have now. That's O(n) where n = entity count.
  2. Create a special table that only really needs to get replicated to single entity and have it contain a hashset that has all of the entity ids and we can use a Redis channel to receive notifications when that entity is updated. I don't really like this idea because it's single point of failure.

Describe alternatives you've considered
We could also go on not doing this, but since we're taking a dependency on ICurrentPersistentIds for helping Cluster.Sharding users cleanup I'd like to do our best to support it.

@Aaronontheweb
Copy link
Member Author

Worth noting: the actual delete and purge routines inside https://github.com/petabridge/Akka.Cluster.Sharding.RepairTool will stick work fine with Redis. Just not the tools for querying the journal to find the shardRegions before deleting them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant