Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SqlExecuteQueryOperator #25259

Closed
1 task done
eladkal opened this issue Jul 24, 2022 · 2 comments · Fixed by #25717
Closed
1 task done

Add SqlExecuteQueryOperator #25259

eladkal opened this issue Jul 24, 2022 · 2 comments · Fixed by #25717
Assignees
Labels
area:providers kind:task A task that needs to be completed as part of a larger issue provider:amazon-aws AWS/Amazon - related issues provider:common-sql provider:google Google (including GCP) related issues provider:microsoft-azure Azure-related issues provider:snowflake Issues related to Snowflake provider

Comments

@eladkal
Copy link
Contributor

eladkal commented Jul 24, 2022

Body

There has been previous attempt to add this operator in #10011
But now after #23971 is completed this should be much easier as DbApiHook has unified run method that works for all providers (with some configuration of parameters like split_statements etc..)

We can now explore again the option of SqlExecuteQueryOperator
There are two approaches that I can think of:

  1. Creating a BaseSqlExecuteQueryOperator that all Sql related operators (PostgresOperator, MySqlOperator, SnowflakeOperator) will inherit from.
  2. Deprecate the current individual operators in favor of SqlExecuteQueryOperator (just like Add SQLExecuteQueryOperator operator #10011 attempted to do)

Personally I prefer to try the 2nd approach first (It should be similar to what we do in SqlSensor or SqlBranchOperator the issue to resolve is how to handle the batch queries in some of the DBs and that is now should be possible thanks to #23971 in SqlSensor / SqlBranchOperator executing batch queries was never an issue as they are expected to execute only 1 SQL statement )

FYI @potiuk @kazanzhy

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.
@eladkal eladkal added provider:google Google (including GCP) related issues provider:microsoft-azure Azure-related issues provider:amazon-aws AWS/Amazon - related issues area:providers kind:task A task that needs to be completed as part of a larger issue provider:snowflake Issues related to Snowflake provider provider:DBs provider:common-sql labels Jul 24, 2022
@kazanzhy
Copy link
Contributor

Great idea, I like SqlSensor and use it in my pipelines.
If @mik-laj doesn't have time I could pick it up

@eladkal
Copy link
Contributor Author

eladkal commented Jul 24, 2022

Feel free to pick it @kazanzhy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers kind:task A task that needs to be completed as part of a larger issue provider:amazon-aws AWS/Amazon - related issues provider:common-sql provider:google Google (including GCP) related issues provider:microsoft-azure Azure-related issues provider:snowflake Issues related to Snowflake provider
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants