Skip to content

Stale subscriptions data cleanup for operational stability #1

@Envek

Description

@Envek

As this gem actually stores runtime data in Anycable's Redis instance it is comparatively easy to use all available space by stale data about subscriptions (if for some reason it weren't deleted on client unsubscription).

One possible variant is:

  1. Set expiration with some configurable timeout (with default with 1 day) for subscriptions (SUBSCRIPTION_PREFIX + subscription.id keys). Let it die. But events keys can't be expired as it will lead to broken subscriptions.
  2. On application (anycable RPC server) start subscribe to all expire events in Redis via Keyspace notifications (but in that case subscription_id → event_id mapping should be stored separately from subscription itself as it is impossible to retrieve contents of an expired key)
  3. On key expired event fired subscription_id should be removed from all events. Event keys from which the last element was removed will be deleted automatically by Redis.
  4. Additionally, it is possible to run some rake task every day or week and check every SUBSCRIPTION_PREFIX keys and check their staleness via OBJECT IDLETIME, remove their ids from every event and remove them altogether.

The problem of this approach is in Redis configuration that should be altered to let it work: Keyspace notifications are disabled by default and OBJECT command may be disabled by security concerns (as it is considered to be a service command for debug purposes). And there may be no privileges to change these settings.

Any other ideas?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions