Skip to content

Airflow db cleanup dag update #8851

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
leahecole merged 12 commits into from
Jan 10, 2023
Merged

Airflow db cleanup dag update #8851

leahecole merged 12 commits into from
Jan 10, 2023

Conversation

@kubasieron
Copy link
Contributor

@kubasieron kubasieron commented Jan 3, 2023

Description

Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.

Checklist

@snippet-bot
Copy link

snippet-bot bot commented Jan 3, 2023 •
edited
Loading

No region tags are edited in this PR.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@product-auto-label product-auto-label bot added the samples Issues that are directly related to samples. label Jan 3, 2023
kosteev
kosteev reviewed Jan 3, 2023
View reviewed changes
@kubasieron kubasieron force-pushed the airflow-db-cleanup-dag-update branch from 18c88fb to 0201d43 Compare January 3, 2023 13:01
@leahecole
Copy link
Collaborator

leahecole commented Jan 3, 2023

Just to double check, we're no longer supporting the Airflow 1 version of this DAG and it's been removed from the docs too?

@kubasieron
Copy link
Contributor Author

kubasieron commented Jan 3, 2023

Just to double check, we're no longer supporting the Airflow 1 version of this DAG and it's been removed from the docs too?

We are supporting it, but now script in composer/workflows/airflow_db_cleanup.py works both for Airflow 1 and 2, so I feel like there is no need for script for Airflow 1 only, however I was not sure about it, so if you think differently, I'll create separate for Airflow 1 and 2.

@kubasieron kubasieron force-pushed the airflow-db-cleanup-dag-update branch from eedf8e4 to 8aad464 Compare January 5, 2023 11:12
@kubasieron kubasieron requested a review from a team as a code owner January 5, 2023 11:12
kosteev
kosteev approved these changes Jan 5, 2023
View reviewed changes
@leahecole leahecole added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 5, 2023
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 5, 2023
leahecole
leahecole requested changes Jan 5, 2023
View reviewed changes
@leahecole
Copy link
Collaborator

leahecole commented Jan 5, 2023

Requesting changes in addition to what Eugene pointed out

@kubasieron kubasieron force-pushed the airflow-db-cleanup-dag-update branch from e350601 to a6e359c Compare January 5, 2023 23:18
@kubasieron kubasieron requested review from leahecole and removed request for rafalbiegacz January 5, 2023 23:22
@kubasieron kubasieron force-pushed the airflow-db-cleanup-dag-update branch 2 times, most recently from abaa0e5 to a300410 Compare January 9, 2023 11:53
kosteev
kosteev approved these changes Jan 9, 2023
View reviewed changes
Copy link
Contributor

@kosteev kosteev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

composer/airflow_1_samples/airflow_db_cleanup.py Outdated
Copy link
Contributor

@kosteev kosteev Jan 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: redundant brackets (len("+composer")) -> len("+composer")

composer/workflows/airflow_db_cleanup.py Outdated
Copy link
Contributor

@kosteev kosteev Jan 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in "processf"

@leahecole leahecole assigned leahecole and unassigned dandhlee Jan 9, 2023
@kubasieron kubasieron force-pushed the airflow-db-cleanup-dag-update branch from a300410 to 9ce7432 Compare January 9, 2023 16:12
@leahecole leahecole added automerge Merge the pull request once unit tests and other checks pass. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jan 9, 2023
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 9, 2023
@leahecole leahecole added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed automerge Merge the pull request once unit tests and other checks pass. labels Jan 9, 2023
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 9, 2023
@leahecole
Copy link
Collaborator

leahecole commented Jan 10, 2023

I am admin merging because there have not been changes that would break lint since the last successful run, and we seem to be having kokoro problems

@leahecole leahecole merged commit f396dc1 into GoogleCloudPlatform:main Jan 10, 2023
telpirion pushed a commit that referenced this pull request Mar 13, 2023
@kubasieron @leahecole @telpirion
Airflow db cleanup dag update (#8851)
ef67a5b 
* Fix and adjust to Airflow 2.3 db cleanup DAG

* Remove cleanup Task and TaskSet tables

* removing duplicated airflow_1 file + some minor fixes

* fix lint errors

* small fix

* small fix  in deletion

* separate dag logic between airflow versions

* move session closing to finally part

* lint fixes

* import order fix

Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leahecole leahecole leahecole approved these changes

@michalmodras michalmodras michalmodras approved these changes

@rachael-ds rachael-ds Awaiting requested review from rachael-ds

+1 more reviewer

@kosteev kosteev kosteev approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@leahecole leahecole

Labels

samples Issues that are directly related to samples.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@kubasieron @leahecole @kosteev @michalmodras @kokoro-team @dandhlee