Using Source and Destination as parameters

airflow_task = AirbyteTriggerSyncOperator(
    task_id='sync_money_json',
    airbyte_conn_id='airbyte_conn_example',
    source_name='Money',
    dest_name='JSON destination'
)

Using ConnectionId directly

airflow_task = AirbyteTriggerSyncOperator(
    task_id='sync_money_json',
    airbyte_conn_id='airbyte_conn_example',
    connection_id='5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6',
)

I used the name of the source_name and destination_name because I find them more convenient and readable.
At the time of connectionId I just managed to get this info from the URL in the connection page, is that correct?
Analyzing the two examples above, a task in Airflow requires having a task_id, this variable can be given a more friendly name than using source or connectionId does not have much difference.

PROS:
Using the direct connectionId makes integration with Airflow much easier because I'll directly call the endpoint that triggers the job.

CONS:
Imagine that the job has a problem, and it is necessary to access Airbyte UI for debugging. Having the source_name and destination_name information ends up having a logical path to find out my connection. However, if I only use the connectionId information in Airflow, I may have a problem finding the connection.
(Of course, you can enter the URL /connection and paste the connectionId, but it's not the coolest thing to do, IMHO)

The AirbyteTriggerSyncOperator must live within the Airflow project as a Provider package. People can install it easily when set up Airflow as an extra-package requirement apache-airflow[airbyte].
Airbyte and Airflow are evolving very fast. A repository could be created with the sample project and a CI to be executed weekly/monthly to ensure that the integration continues correctly.

Support interactions with Airflow #836

Description

michel-tricotopenedon Nov 6, 2020

Tell us about the problem you're trying to solve

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

michel-tricot
openedon Nov 6, 2020