Skip to content

Conversation

@arvindp25
Copy link
Contributor

@arvindp25 arvindp25 commented Jun 21, 2025


Issues/feature request link: #51929
This pull request introduces new operator in google provider.
With help of new operator airflow user now can create google cloud log sinks via airflow.

### Checklist

- [x] Unit tests added
- [x] System test added
- [x] Documentation updated (docstrings + relevant .rst files if applicable)
- [x] PR description includes motivation and usage example
- [x] Follows Airflow’s [contribution guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg
Copy link

boring-cyborg bot commented Jun 21, 2025

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@boring-cyborg boring-cyborg bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Jun 21, 2025
@arvindp25 arvindp25 changed the title added support for Google cloud log sink management via google provider Added support for Google cloud log sink management via google provider Jun 21, 2025
@arvindp25 arvindp25 changed the title Added support for Google cloud log sink management via google provider feature: Added support for Google cloud log sink management via google provider Jun 21, 2025
Copy link
Contributor

@shahar1 shahar1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks promising overall! Small comment regarding validation of templated fields.
@VladaZakharova - I'll be happy for additional feedback :)

@arvindp25 arvindp25 requested a review from shahar1 June 22, 2025 15:31
@arvindp25 arvindp25 force-pushed the feature/add-cloud-logging-export-operator branch from 5889af3 to ee8bc9d Compare June 22, 2025 16:34
@VladaZakharova
Copy link
Contributor

Hi
can you please show system tests results with screenshots? thanks

@arvindp25
Copy link
Contributor Author

arvindp25 commented Jun 23, 2025

Hi can you please show system tests results with screenshots? thanks
@VladaZakharova
for system test, I have recorded video please fast forward as per your need.

video.mp4

further below I have added some additional screenshot from manual dag testing.

creating multiple log sink with different destination
sink_create_multiple

updating log sink
update_sink

list sink
list_sink

delete sinks
delete sink

Please let me know if i have to do something else as Proof of work.
thanks,

Copy link
Contributor

@shahar1 shahar1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, well done!
Waiting for @VladaZakharova 's approval to merge :)

@arvindp25 arvindp25 force-pushed the feature/add-cloud-logging-export-operator branch 2 times, most recently from 8455f95 to 83e661b Compare June 28, 2025 12:17
@arvindp25
Copy link
Contributor Author

arvindp25 commented Jun 28, 2025

System Test:-
video
https://github.com/user-attachments/assets/01fce6e2-cc4f-4d11-b8c7-b1d06b61a47e

unit test hook:-
image

unit test operator:-
image

manual dag testing:-
creating sink ✅

test_bq_sink and test_pubsub_sink got created.
image

update_sink ✅
my-airflow-test-sink updated successfully.
image

list sink ✅
total number of sink on project 5 can be matched in log output
image

delete_sink ✅
all three sink got deleted.
image

@potiuk potiuk force-pushed the feature/add-cloud-logging-export-operator branch from 83e661b to 2e67e19 Compare June 28, 2025 14:28
@arvindp25
Copy link
Contributor Author

@VladaZakharova I have made requested change. Could you please re-review?
change:-
1.moved connection logic to hook
2. used sink_config for configuration support both dict and protobuf obj
3. removed example_ from dag names

Added:-
protobuf support.
system test with protobuf object.

@arvindp25 arvindp25 requested a review from VladaZakharova June 28, 2025 16:37
@arvindp25 arvindp25 force-pushed the feature/add-cloud-logging-export-operator branch from b88400b to e8e6689 Compare June 29, 2025 10:59
Copy link
Contributor

@VladaZakharova VladaZakharova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, small comments left 😺

@arvindp25 arvindp25 requested a review from VladaZakharova June 30, 2025 14:01
@arvindp25
Copy link
Contributor Author

arvindp25 commented Jun 30, 2025

@VladaZakharova made requested change. Please check

@arvindp25
Copy link
Contributor Author

@VladaZakharova Just a friendly reminder to look at this.

Copy link
Contributor

@VladaZakharova VladaZakharova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job!

@potiuk potiuk merged commit bcebf83 into apache:main Jul 4, 2025
74 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants