SDK - Using Airflow ops in Pipelines

[Ark-kun]

This CL adds support for converting arbitrary Airflow operators to Pipelines components so that it can be used as a pipeline task.

from airflow.operators.bash_operator import BashOperator
bash_op = kfp.components.create_component_from_airflow_op(BashOperator)

@pipeline()
def my_pipeline():
  bash_task = bash_op('echo "Hello from $SHELL"', xcom_push=True)

---

This change is 

[hongye-sun]

Thanks for the PR. This is really great that KFP users can directly use Airflow operators in their pipeline.

[Ark-kun]

/test kubeflow-pipeline-sample-test

[animeshsingh]

@Ark-kun is this only for bash operator? Other operators being supported as well? Also can this be extended to take an overall Airflow DAG, and execute as one of the 'steps' in KFP, assuming an Airflow deployment exists and KFP is made aware of it?

[animeshsingh]

@paveldournov you have a viewpoint on the above question? cc @vicaire @jlewi

[Ark-kun]

@Ark-kun is this only for bash operator? Other operators being supported as well?

This feature supports all Airflow operators.

can this be extended to take an overall Airflow DAG, and execute as one of the 'steps' in KFP

It could. How do you envision the API for AirFlow DAGs?

assuming an Airflow deployment exists and KFP is made aware of it?

This might be unnecessary

[Ark-kun]

/test kubeflow-pipeline-e2e-test

[Ark-kun]

/cc @hongye-sun I've added documentation for this functionality.

[hongye-sun]

/lgtm

[animeshsingh]

It could. How do you envision the API for AirFlow DAGs?

Two modes

1. maybe a component can take Airflow DAG as an input, and calls Airflow deployment
2. Pipeline DSL can compile to airflow DAG?

[hongye-sun]

/lgtm

[Ark-kun]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• sdk/OWNERS [Ark-kun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[NieRonghua]

@Ark-kun is this only for bash operator? Other operators being supported as well?

This feature supports all Airflow operators.

Hi, I’m an Airflow user, now i want to create components by using create_component_from_airflow_op(SimpleHttpOperator), but i do not konw how to define the airflow‘s conn such as msyql conn or http conn, can u give me an example? thx