Skip to content

Adjust auckland_museum parameters to allow backfill #3660

Open

Description

Problem

The auckland_museum DAG is dated, so when we turn it on for the first time it will begin backfilling from its 'start_date'.

The start_date currently configured is fairly recent, but the Museum has records that date back much further.

Description

To backfill all the records from the Museum's collection, we should research an appropriate start_date to configure for the DAG.

However, we should also be very careful to note that this will start a continuously running backfill that may put us at risk of exceeding the Museum's state rate limits. We'll need to check those rate limits and increase the delay for the ingester class enough to ensure that we do not approach them.

If it is determined that it would be safe to lower the delay after the backfill is complete (because we'll only be running one day's ingestion at a time), make an issue to update the DAG again once the backfill is done.

Additional context

Note that the DAG should not be turned on until this has been fixed, or else it will be tricky to do the backup properly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    ✨ goal: improvementImprovement to an existing user-facing feature💻 aspect: codeConcerns the software code in the repository🟨 priority: mediumNot blocking but should be addressed soon🧱 stack: catalogRelated to the catalog and Airflow DAGs

    Type

    No type

    Projects

    • Status

      📋 Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions