An Exascale Computing Project team.
The goal of this repo is to develop a Software Development Kit (SDK) for workflows at exascale. This will integrate the Parsl, Flux, RADICAL-Pilot, and Swift/T workflow systems, plus other tools, into a consistent set of "ExaWorks SDK Component Packages". Applications teams will be able to easily draw from these packages as needed, and develop and share workflows built on the various systems.
Pure Python library for describing and managing parallel computation on one to many nodes. It contains abstractions to support various parallel and distributed workloads: from bag-of-tasks to dataflow, short to long duration tasks, single core through to multi-node.
Flux is a workload management system (à la Slurm, PBS, LSF), with single-user and multi-user (a.k.a. system instance) modes.
RADICAL-Cybertools is an abstractions-based suite of middleware building blocks that are architected for scalable, interoperable and sustainable approaches to support science on a range of high-performance and distributed computing systems.
Swift/T is an MPI-based workflow language and runtime system. It runs in a one big job model, with internal automatic task parallelization and data movement, enhanced by workflow-level compiler optimizations.
The objectives of this SDK are to:
-
Make it easy to install any or all of the workflow systems on the same underlying system.
-
Develop various levels of interoperabilty among the workflow systems.
-
Support Enterprise and HPC systems.
The SDK maintains a CI/CD system for its supported component packages. The CI/CD tools will be portable to a variety of CI/CD services. For now, we use GitHub Actions CI workflows.
Each Dockerfile image relies on several packages from the ExaWorks SDK Base (CentOS 7), particularly the C tools and OpenMPI. Other Base images will be supported, including those based on CentOS 8.
Most workflow system dependencies are in the Base Dockerfile, but some customizations may be made by the individual systems.
The SDK Base images can be built with a simple docker build command, using the directory base/ as the build context.
First, build the Base image (centos7|centos8) with:
$ cd docker/base/ $ docker build --tag <TAG> --file centos7/Dockerfile .
Then, build the desired workflow system FROM the desired TAG. For example:
$ cd ../flux/ $ docker build --tag <TAG> .
Other ExaWorks resources including our public Slack channels are available through the ExaWorks Home Page.
You are welcome to provide feedback via the ExaWorks SDK GitHub Discussion.
Please report technical issues to the ExaWorks SDK GitHub Issue Tracker.