Investigate Jamendo inconsistent result count #3500
Labels
💻 aspect: code
Concerns the software code in the repository
🛠 goal: fix
Bug fix
🟩 priority: low
Low priority and doesn't need to be rushed
🧱 stack: catalog
Related to the catalog and Airflow DAGs
Description
The Jamendo DAG is non-dated, meaning each time it runs it should be ingesting all available data for the provider. Consequently, one would expect the number of records ingested to increase slightly on each run. This does not appear to be the case, or at least not consistently. Here are the record counts returned by the most recent production runs, from March to December 2023:
The three bolded entries, including the most recent DagRun (for November 2023), have an unexplained dramatic decrease in the number of records ingested.
We should investigate these dates closer to see if we can get a better understanding of what is happening.
Reproduction
Observe the record counts from recent production runs.
The text was updated successfully, but these errors were encountered: