Closed
Description
Description
The batched update DAGs have default timeouts that are quite long, mostly because we were unsure how they would shake out once we started running the popularity update regularly:
openverse/catalog/dags/database/batched_update/constants.py
Lines 11 to 13 in 0a741a6
We could have separate timeout values for the automated and manual batched update runs, but for the time being I think we can safely change the following values:
UPDATE_TIMEOUT
should be set to 30 days (rather than 3 months)DAGRUN_TIMEOUT
should be set toUPDATE_TIMEOUT + SELECT_TIMEOUT
(rather than explicitly setting the timeout)
Additional context
The longest batched_update
run we've had was just under 12 days, and that was the automated run for Flickr's popularity update.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Assignees
Labels
Type
Projects
Status
✅ Done