Skip to content

Implementation Plan: Regularly export tallies data so that it can be cleaned from Redis #2401

Open

Description

Problem

Much of our tallies data is captured on a weekly or monthly basis. For example, provider occurrences in search:

https://github.com//WordPress/openverse/blob/f54c71c4b7d4be49ea94f0501b5089171695235c/api/api/utils/tallies.py#L38-L39

These also do not have a TTL and will live forever in our Redis.

Description

We need an implementation plan for the creation of a DAG (or something) to export data for non-active tally keys from production Redis, save it to a queryable location suitable for long-term storage, and then clean the keys from production Redis.

I've requested an IP for this because it probably deserves some careful forethought and whatever solution we take would require wiring up new Airflow providers or infrastructure that doesn't currently talk to each other.

Related to #420

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Type

No type

Projects

  • Status

    📋 Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions