Formicary is a distributed, cloud-native orchestration engine for executing complex workflows, data pipelines, and CI/CD jobs.
Formicary uses a declarative YAML-based approach to define jobs as a Directed Acyclic Graph (DAG) of tasks. It's built on a robust leader-follower architecture (Queen and Ants) that efficiently distributes work to executors running on Docker, Kubernetes, or as simple shell commands.
It is designed for use cases that require complex dependency management, parallel execution, and sophisticated error handling.
- Declarative Workflows: Define complex jobs with multiple tasks and dependencies in simple YAML files.
- Extensible Executors: Natively supports Kubernetes, Docker, Shell, HTTP/REST, and Websockets.
- Advanced Flow Control: Sophisticated retry logic, conditional execution (
on_exit_code), error handling, and optional tasks. - Parallel & Concurrent Execution: Fork jobs to run in parallel and join the results, with concurrency limits.
- Powerful Templating: Use Go templates in your job definitions for dynamic workflows.
- Built-in Caching: Speed up jobs with caching for dependencies (e.g.,
node_modules,vendor,.m2). - Robust Artifacts Management: Persist task outputs to an S3-compatible object store for use in later stages or for download.
- Security First: RBAC, encrypted secrets, and JWT-based API authentication.
- Real-time & Observability: Stream logs in real-time, get Prometheus metrics, and view detailed job statistics.
The easiest way to get started with Formicary is using docker-compose.
-
Clone the repository:
git clone https://github.com/bhatti/formicary.git cd formicary -
Prepare configuration: Define configuration file similar to
config/.formicary.yaml -
Run Formicary: This command starts the Queen server, a local Ant worker, Redis, and MinIO.
docker-compose up
-
Explore!
- Open the Formicary Dashboard at http://localhost:7777.
- Follow the Quick Start Guide to run your first job.
- Introduction
- Getting Started
- Guides & Tutorials
- Reference
- Community
- See how Formicary compares to tools like Airflow, Jenkins, GitLab CI, and GitHub Actions.
- Building a distributed orchestration and graph processing system
- Building Resilient, Interactive Playbooks with an Orchestration Engine
- Task Scheduling Algorithms in Distributed Orchestration Systems
Formicary is licensed under the GNU AGPLv3 License.