Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add low-complexity testing DAG #1721

Closed
1 task
zackkrida opened this issue Aug 11, 2021 · 3 comments · Fixed by WordPress/openverse-catalog#151
Closed
1 task

[Feature] Add low-complexity testing DAG #1721

zackkrida opened this issue Aug 11, 2021 · 3 comments · Fixed by WordPress/openverse-catalog#151
Assignees
Labels
💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟩 priority: low Low priority and doesn't need to be rushed 🚦 status: awaiting triage Has not been triaged & therefore, not ready for work

Comments

@zackkrida
Copy link
Member

zackkrida commented Aug 11, 2021

Problem

When testing airflow configuration many of the DAGs take a long time to run or have many 3rd party network calls. It would be nice to have a simple DAG that logs some values or something, for easy testing of Airflow configuration in production.

Something with multiple steps, maybe some fake latency to make the steps a few seconds long each, and just logs a bunch of output.

Description

Alternatives

Additional context

Implementation

  • 🙋 I would be interested in implementing this feature.
@zackkrida zackkrida added 🟩 priority: low Low priority and doesn't need to be rushed 🚦 status: awaiting triage Has not been triaged & therefore, not ready for work ✨ goal: improvement Improvement to an existing user-facing feature 💻 aspect: code Concerns the software code in the repository labels Aug 11, 2021
@zackkrida zackkrida changed the title [Feature] Add simple testing DAG [Feature] Add low-complexity testing DAG Aug 19, 2021
@sarayourfriend sarayourfriend self-assigned this Aug 20, 2021
@sarayourfriend
Copy link
Contributor

@zackkrida does the DAG need to actually return data for Airflow to process or just log stuff?

@zackkrida
Copy link
Member Author

i think just logging stuff is okay.

@obulat
Copy link
Contributor

obulat commented Aug 23, 2021

I don't know if it's useful at all here, but when a task returns a value, it can be used by the next task using xcom_pull. That's a way of sharing data between tasks. Also, anything printed out on the last line by a BashOperator becomes its return value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟩 priority: low Low priority and doesn't need to be rushed 🚦 status: awaiting triage Has not been triaged & therefore, not ready for work
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants