Skip to content

PR to add the Architectural Decision Record (ADR) for Apache Airflow to the Unity Docs repository #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

PR to add the Architectural Decision Record (ADR) for Apache Airflow to the Unity Docs repository #10

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6b29898
Adding ADR for integrating Apache Airflow
Nov 10, 2023
ed29555
Update science-processing-service-apache-airflow-integration.md
LucaCinquini Nov 10, 2023
94b3d0a
Update science-processing-service-apache-airflow-integration.md
LucaCinquini Nov 10, 2023
596324d
Update science-processing-service-apache-airflow-integration.md
LucaCinquini Nov 10, 2023
5b0d991
Update science-processing-service-apache-airflow-integration.md
LucaCinquini Nov 10, 2023
4f5b2e1
Update science-processing-service-apache-airflow-integration.md
LucaCinquini Nov 10, 2023
acc2938
Update science-processing-service-apache-airflow-integration.md
LucaCinquini Nov 10, 2023
6c500d0
Update science-processing-service-apache-airflow-integration.md
LucaCinquini Nov 10, 2023
34b68a7
Update science-processing-service-apache-airflow-integration.md
LucaCinquini Nov 10, 2023
17195d3
Update science-processing-service-apache-airflow-integration.md
LucaCinquini Nov 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Update science-processing-service-apache-airflow-integration.md
  • Loading branch information
@LucaCinquini
LucaCinquini authored Nov 10, 2023
commit 17195d3f9ff78813f6d84dba072ef80a7a232c67
5 changes: 4 additions & 1 deletion ...tural-decision-records/science-processing-service-apache-airflow-integration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

### **Context**

In recent years, Apache Airflow has emerged as one of the leading open source orchestration engines for scalable jobs processing. Additionally, it is gaining attention and traction at JPL across several projects in Earth and Planetary sciences. We are proposig to integrate the Airflow architecture in the Unity model, as such:
In recent years, Apache Airflow has emerged as one of the leading open source orchestration engines for scalable jobs processing. Additionally, it is gaining attention and traction at JPL across several projects in Earth and Planetary sciences. We are proposig to integrate the Airflow architecture in the Unity model, as such (see diagram below):

* The core components of Airflow (Web Server, Scheduler, Database) will compose the front-end EMS layer (which provides orchestration and monitoring across multiple back-ends)
* The Airflow Operators will be used to submit workloads to multiple pluggable ADES back-ends (Celery Workers, EKS, ECS, etc.)
Expand All @@ -18,6 +18,9 @@ Additionally, Unity may decide to provide Airflow extensions as follows:
* An OGC WPS-T interface to allow clients to submit job requests that conform to this API specificiation
* An Airflow HySDS Operator to allow projects to execute workloads on the HySDS system
* An Airflow WPS-T Operator to allow projects to submit requests to any WPS-T compliant back-end

<img width="982" alt="Screenshot 2023-11-10 at 09 38 20" src="https://github.com/LucaCinquini/unity-docs/assets/637059/5ada4434-cc05-4153-9b35-ca6c38a2d5df">


### Alternatives

Expand Down