Description
openedon Jan 12, 2024
Problem
The auckland_museum
DAG is dated, so when we turn it on for the first time it will begin backfilling from its 'start_date'.
The start_date currently configured is fairly recent, but the Museum has records that date back much further.
Description
To backfill all the records from the Museum's collection, we should research an appropriate start_date to configure for the DAG.
However, we should also be very careful to note that this will start a continuously running backfill that may put us at risk of exceeding the Museum's state rate limits. We'll need to check those rate limits and increase the delay
for the ingester class enough to ensure that we do not approach them.
If it is determined that it would be safe to lower the delay
after the backfill is complete (because we'll only be running one day's ingestion at a time), make an issue to update the DAG again once the backfill is done.
Additional context
Note that the DAG should not be turned on until this has been fixed, or else it will be tricky to do the backup properly.
Metadata
Assignees
Labels
Type
Projects
Status
📋 Backlog