Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide default headers including user-agent for all provider DAGs #3683

Open
stacimc opened this issue Jan 19, 2024 · 0 comments
Open

Provide default headers including user-agent for all provider DAGs #3683

stacimc opened this issue Jan 19, 2024 · 0 comments
Labels
💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs

Comments

@stacimc
Copy link
Contributor

stacimc commented Jan 19, 2024

Problem

Some provider APIs require a User-Agent header, like rawpixel (#3679). In rawpixel's case, we only discovered this when the DAG stopped working.

Description

We could set the User-Agent as a default header on all of our provider APIs by setting it in the ProviderDataIngester. Note that currently, headers is empty on the base class and subclasses totally overwrite it when they want to add headers. We would need to update the base class to provide default headers but still allow subclasses to extend them.

@stacimc stacimc added 🟩 priority: low Low priority and doesn't need to be rushed ✨ goal: improvement Improvement to an existing user-facing feature 💻 aspect: code Concerns the software code in the repository 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Jan 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs
Projects
Status: 📋 Backlog
Development

No branches or pull requests

1 participant