Visual TOM Airflow integration

 

Disclaimer

No Support and No Warranty are provided by Absyss SAS for this project and related material. The use of this project's files is at your own risk.

Absyss SAS assumes no liability for damage caused by the usage of any of the files offered here via this Github repository.

Consultings days can be requested to help for the implementation.

Prerequisites

• Visual TOM 7.1.2 or greater
• "Python.exe (Windows Agent) or python3 (Linux Agent) must be accessible via the submitting user's PATH." (CR 50 if absent)

Instructions

Create/drop a directory named vtom-airflow in $ABM_BIN

This directory should contain the following files: runDag_requests_full.py (Script executed by the batch queue) (CR 55 if missing) config.template.py, rename in config.py (Script to be modified before the first execution)

• base_url → URL used to access the Airflow API server
• authType → "basic" or "GCP-serviceaccount"
• serviceaccountFile → JSON file to be modified for authentication if authType = GCP-serviceaccount
• username → Fill in if authType = basic
• password → Fill in if authType = basic

tom_sumbit.airflow => place in $TOM_ADMIN and create the Linux queue "queue_airflow"

submit_queue_airflow.bat => place in $ABM_BIN and create the Windows queue "queue_airflow"

From the interface:

• Create the queue: queue_airflow
• Create the Airflow Processing Template with a string field named "DAG Name" and set the default queue to "queue_airflow"

Available Actions

The Visual TOM Airflow connector provides a single action for triggering an Airflow DAG (Directed Acyclic Graph) and checking the execution status.

Each Visual TOM Job execution corresponds to one unique Airflow DAG run, with synchronous monitoring until completion.

DAG Triggering

Description

This triggering action includes the follwing features:

• verifying the existence of the target Airflow DAG,
• triggering a new DAG run,
• monitoring the execution in real time,
• analysis the status of all DAG tasks,
• return an exit code usable by Visual TOM.

The execution is blocking: the Visual TOM Job remains running until the Airflow DAG execution is fully completed.

Detailed Behavior

1. DAG Validation

   • Calls the Airflow API to verify that the DAG exists.
   • If the DAG does not exist, the Visual TOM Job fails immediately.

2. DAG Trigger

   • Creates a new dagRun using the Airflow API.
   • Each Visual TOM execution always triggers a new DAG run.
   • An optional configuration (conf) can be passed to the DAG.

3. Execution Monitoring

   • Periodically queries the Airflow API to retrieve:
     • the global DAG state (queued, running, success, failed),
     • the state of each individual task.
   • Task states are displayed in Visual TOM logs.

4. Final Evaluation

   • At the end of the execution:
     • if the DAG succeeds and all tasks succeed, the Visual TOM Job is successful,
     • if the DAG completes but at least one task fails, the Visual TOM Job fails,
     • in case of technical errors (API, authentication, timeout), the Visual TOM Job fails.

Visual TOM Return Codes

Return code	Meaning
0	DAG completed successfully and all tasks succeeded
2	DAG completed but at least one task failed
1	Technical error (DAG not found, API error, authentication issue, timeout, interruption)

These return codes allow Visual TOM to properly manage dependencies, retries, and alerts.

Input Parameters

• DAG Name (required)
  Identifier of the Airflow DAG to trigger. To be filled in the Visual TOM connector.

• DAG Configuration (optional)
  The DAG is triggered using the Airflow REST API. Optional parameters can be passed to the DAG using the conf payload of the API call and are accessible in Airflow via dag_run.conf.

Important Notes

• The connector supports the following authentication methods:
  • Basic Authentication
  • GCP Service Account.
• Monitoring relies exclusively on the Airflow REST APIs.
• Existing DAGs are never modified by the connector.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details

Code of Conduct

Absyss SAS has adopted the Contributor Covenant as its Code of Conduct, and we expect project participants to adhere to it. Please read the full text so that you can understand what actions will and will not be tolerated.