Add operator to invoke Azure-Synapse pipeline

[ambika-garg]

This PR adds a custom operator AzureSynapsePipelineOperator to the Microsoft provider. This operator simplifies the execution of Azure Synapse pipelines, allowing Airflow users to trigger Azure Synapse Pipelines directly from their DAGs..

Usage example:

import datetime
from airflow import DAG
from airflow.providers.microsoft.azure.operators.synapse import AzureSynapsePipelineOperator

with DAG(
    dag_id="synapse_analytics_dag",
    schedule_interval=None,
    start_date=datetime.datetime(2021, 1, 1),
) as dag:
    run_synapse_pipeline = AzureSynapseRunPipelineOperator(
        azure_synapse_conn_id="azure_synapse_connection",
        task_id="run_synapse_pipeline",
        pipeline_name="synapse_pipeline_name",
        azure_synapse_workspace_dev_endpoint="azure_synapse_workspace_dev_endpoint"
    )

    run_synapse_pipeline
```

[Taragolis]

Seems like you use Windows. As I could see there is LF (common posix line ending character) changes to CRLF (windows).

You need to configure your git client: https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings?platform=windows

In additional it might required to make additional changes into your editor, Airflow uses .editorconfig configuration, so if your editor support it (e.g. PyCharm support it out of the box), you could set it up by follow instructions: https://editorconfig.org/

In addition it might required to run Static Checks against your changes, see Instruction

[ambika-garg]

Hi, after configuring git client to handle line endings, I am unable to start Breeze, it is exists with the error 127.
Refer to screenshot.

[eladkal]

What is the motivation for a new file?
can't the hook be in the existed synapse.py?

[ambika-garg]

My motivation is to keep code modular and easy to maintain. The synapse.py file deals with SparkClient and its functions, where as the synapse_pipeline.py file deals with ArtifactsClient and its functions. So both classes have different dependencies.
Although I had 2 options:

1. Include every function of AzureSynapsePipelineHook in AzureSynapseHook class, But that would add on lots of conditional statements, difficulty in dependency management and separate naming conventions for functions with same functionality but different parameters and return values like(get_conn()) function, further leading to confusions when code is extended.
2. Make Separate classes for each type of service, where each class has its own set of dependencies and responsibility. Also, this approach provides consistent naming convention for functions within each class, improving code readability.

I went with 2nd approach. That's why I made a new file. Also, I think making new hook class with separate dependencies is easier way to code extension .

Please let me know your opinion on this, since you have diverse background on development.

[eladkal]

In Airflow we use mixed governance model for providers which means that the company behind the provider have shared responsibility over it. Sadly in case of Microsoft the Azure team is not involved (Unlike Google and Aws) as a result we have no way of knowing what is Azure stand point of this subject.
In that case my point of view is to "fallback" to the common practice of how we do things in other providers - you can see for example the effort we did in Amazon/Aws #20139 to have organized files where the motivation is ease of find.

So in the case in front of us and given the information you shared I still think we should use synapse.py.

I welcome Azure team to be more involved in the project should they decide to.

[ambika-garg]

I went through the #20139 , understood the general protocols followed in Airflow. So, I'll be moving AzureSynapsePipelineHook class in synapse.py. Also, I'll discuss internally if Azure team can involve more in Airflow project.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.