Skip to content

README_en.md

Latest commit

 

History

History
455 lines (348 loc) · 23.6 KB

README_en.md

File metadata and controls

455 lines (348 loc) · 23.6 KB
Raw

languages os stars forks pypi PyPI Downloads CodeFactor

awesome-cpp HelloGithub

Chinese Readme | English Readme | deepwiki

CGraph Readme

CGraph is a cross-platform Directed Acyclic Graph framework based on pure C++ without any 3rd-party dependencies.

You, with it, can build your own operators simply, and describe any running schedules as you need, such as dependence, parallelling, aggregation, conditional and so on. Python APIs are also supported to build your pipeline.

Tutorials and contact information are shown as follows. Please get in touch with us for free if you need more about this repository.

1. Introduction

The Chinese name of CGraph is "Se丶Tu". It is a cross-platform graph flow execution framework without any third-party dependencies. Through the underlying scheduling of GPipeline, it provides eDAG scheduling features including sequential execution of dependent elements, concurrent execution of independent elements, pause, resume, and timeout settings.

Users only need to inherit from the GNode class, implement the run() method in the subclass, and set dependencies as needed to execute tasks as a graph or as a pipeline. Users can also configure different GGroups containing multiple nodes to control conditional judgment, loop, and concurrent execution logic by themselves.

CGraph Skeleton

This project is written with only the C++11 standard library and has no third-party dependencies. It is compatible with MacOS, Linux, Windows, and Android. It supports local compilation and secondary development, and also provides a Python version: pycgraph. For compilation and installation, please refer to CGraph Compile Guide

For detailed feature introductions and usage, please refer to the articles on Chunel's Blog. Related videos are continuously updated on Bilibili. Welcome to watch and discuss:

2. Getting Started Demo

C++ Version

#include "CGraph.h"

using namespace CGraph;

class MyNode1 : public GNode {
public:
    CStatus run() override {
        printf("[%s], sleep for 1 second ...\n", this->getName().c_str());
        CGRAPH_SLEEP_SECOND(1)
        return CStatus();
    }
};

class MyNode2 : public GNode {
public:
    CStatus run() override {
        printf("[%s], sleep for 2 second ...\n", this->getName().c_str());
        CGRAPH_SLEEP_SECOND(2)
        return CStatus();
    }
};


int main() {
    /* Create a pipeline for configuring and executing graph flow information */
    GPipelinePtr pipeline = GPipelineFactory::create();
    GElementPtr a, b, c, d = nullptr;

    /* Register dependencies between nodes */
    pipeline->registerGElement<MyNode1>(&a, {}, "nodeA");
    pipeline->registerGElement<MyNode2>(&b, {a}, "nodeB");
    pipeline->registerGElement<MyNode1>(&c, {a}, "nodeC");
    pipeline->registerGElement<MyNode2>(&d, {b, c}, "nodeD");

    /* Execute the graph flow framework */
    pipeline->process();

    /* Clear all resources in the pipeline */
    GPipelineFactory::remove(pipeline);

    return 0;
}

CGraph Demo
As shown above, when the graph structure is executed, node a runs first. After node a finishes, nodes b and c run in parallel. After both b and c finish, node d runs.

Python Version

import time
from datetime import datetime

from pycgraph import GNode, GPipeline, CStatus


class MyNode1(GNode):
    def run(self):
        print("[{0}] {1}, enter MyNode1 run function. Sleep for 1 second ... ".format(datetime.now(), self.getName()))
        time.sleep(1)
        return CStatus()

class MyNode2(GNode):
    def run(self):
        print("[{0}] {1}, enter MyNode2 run function. Sleep for 2 second ... ".format(datetime.now(), self.getName()))
        time.sleep(2)
        return CStatus()


if __name__ == '__main__':
    pipeline = GPipeline()
    a, b, c, d = MyNode1(), MyNode2(), MyNode1(), MyNode2()

    pipeline.registerGElement(a, set(), "nodeA")
    pipeline.registerGElement(b, {a}, "nodeB")
    pipeline.registerGElement(c, {a}, "nodeC")
    pipeline.registerGElement(d, {b, c}, "nodeD")

    pipeline.process()

Other Versions

  • CsCGraph : A CSharp native, CGraph-API-liked DAG project
  • JaCGraph : A Java native, CGraph-API-liked DAG project
  • GoCGraph : A Go native, CGraph-API-liked DAG project
  • CGraph-lite : A one-header-only, CGraph-API-liked DAG project, lite version by C++

3. Recommended Reading

4. Related Projects

Star History Chart

Appendix 1. Version Information

[2021.05.04 - v1.0.0 - Chunel]

  • Provide graph execution and support parallel computation of independent nodes

[2021.05.09 - v1.1.0 - Chunel]

  • Optimize concurrency during graph execution

[2021.05.18 - v1.1.1 - Chunel]

  • Add node name and session information

[2021.05.23 - v1.2.0 - Chunel]

  • Provide single-node loop execution

[2021.05.29 - v1.3.0 - Chunel]

  • Provide cluster and region division and loop execution
  • Provide tutorial content with multiple usage examples

[2021.06.14 - v1.4.0 - Chunel]

  • Provide the param passing mechanism
  • Provide the group feature, with multi-node modules uniformly inheriting from the group module
  • Add support for Linux

[2021.06.20 - v1.4.1 - Chunel]

  • Provide the condition feature
  • Add support for Windows

[2021.06.24 - v1.5.0 - Chunel]

  • Provide the pipeline factory creation method
  • Update tutorial content

[2021.07.07 - v1.5.1 - Chunel]

  • Optimize the thread pool and implement task stealing

[2021.07.11 - v1.5.2 - Chunel]

  • Optimize the thread pool and implement automatic thread-count adjustment

[2021.07.31 - v1.5.3 - Chunel]

  • Optimize the thread pool, implement batch task retrieval, and optimize task stealing

[2021.08.29 - v1.6.0 - Chunel]

  • Provide multi-pipeline support and optimize underlying logic
  • Update tutorial content

[2021.09.19 - v1.6.1 - Chunel]

  • Provide the Lru operator, Trie operator, and template node, and optimize underlying logic
  • Update tutorial content

[2021.09.29 - v1.7.0 - Chunel]

  • Provide the aspect feature for horizontally extending node or group capabilities
  • Update tutorial content

[2021.10.07 - v1.7.1 - Chunel]

  • Optimize aspect implementation logic, provide aspect parameters, and support batch aspect addition
  • Update tutorial content

[2021.11.01 - v1.8.0 - Chunel]

  • Provide the adapter feature and the singleton adapter feature
  • Optimize pipeline execution logic
  • Update tutorial content

[2021.12.18 - v1.8.1 - Chunel]

  • Optimize return-value CStatus information

[2022.01.02 - v1.8.2 - Chunel]

  • Provide automatic exit on node execution timeout and the task group feature
  • Provide thread pool configuration parameter setters

[2022.01.23 - v1.8.3 - Chunel]

  • Provide the function adapter to implement functional programming
  • Provide thread-priority scheduling and CPU-binding execution
  • Update tutorial content

[2022.01.31 - v1.8.4 - Chunel]

  • Provide asynchronous execution for node

[2022.02.03 - v1.8.5 - Chunel]

  • Provide the daemon feature for periodically executing tasks outside the graph flow
  • Update tutorial content

[2022.04.03 - v1.8.6 - Chunel]

  • Provide the DistanceCalculator operator for computing arbitrary data types and arbitrary distance types
  • Update tutorial content

[2022.04.05 - v2.0.0 - Chunel]

  • Provide the domain feature and the Ann domain abstraction model, beginning support for specific professional domains
  • Provide the hold execution mechanism
  • Update tutorial content

[2022.05.01 - v2.0.1 - Chunel]

  • Optimize the pipeline registration mechanism and support custom sequential execution of the init method
  • Provide a one-click build script

[2022.05.29 - v2.1.0 - Chunel]

  • Provide an element parameter writing method
  • Provide support for C++14
  • Update tutorial content

[2022.10.03 - v2.1.1 - Chunel]

  • Provide a task-priority mechanism in the thread pool
  • Optimize group execution logic

[2022.11.03 - v2.2.0 - Chunel]

  • Provide the message feature, mainly for data transfer between different pipelines
  • Update tutorial content

[2022.12.24 - v2.2.1 - Chunel]

  • Provide the TemplateNode feature to optimize parameter passing
  • Update tutorial content

[2022.12.25 - v2.2.2 - yeshenyong]

  • Optimize graph execution logic

[2022.12.30 - v2.2.3 - Chunel]

  • Provide message publish-subscribe support
  • Provide execution engine switching

[2023.01.21 - v2.3.0 - Chunel]

  • Provide the event feature
  • Provide the CGraph Intro.xmind file to introduce the overall CGraph logic through a mind map

[2023.01.25 - v2.3.1 - Chunel]

  • Provide support for C++11. Thanks to MirrorYuChen for the related solution

[2023.02.10 - v2.3.2 - Chunel]

  • Optimize the scheduling strategy and provide a scheduling parameter configuration interface
  • Provide the English version of readme.md

[2023.02.12 - v2.3.3 - yeshenyong, Chunel]

  • Provide graphviz visualization for graph display
  • Provide parameter chain tracing

[2023.02.22 - v2.3.4 - Chunel]

  • Optimize the scheduling mechanism on Windows
  • Optimize the param mechanism and the event mechanism

[2023.03.25 - v2.4.0 - woodx, Chunel]

  • Provide a runnable Docker environment and the Dockerfile for building it
  • Provide the pipeline scheduling resource control mechanism
  • Optimize scheduling performance

[2023.05.05 - v2.4.1 - Chunel]

  • Provide thread-binding execution
  • Provide a method to get the maximum parallelism of a pipeline. Thanks to Hanano-Yuuki for the related solution
  • Provide asynchronous pipeline execution and exit during execution

[2023.06.17 - v2.4.2 - Chunel]

  • Provide the MultiCondition feature
  • Provide pause and resume execution for pipeline

[2023.07.12 - v2.4.3 - Chunel]

  • Optimize CStatus and add exception location information

[2023.09.05 - v2.5.0 - Chunel]

  • Provide the perf feature for pipeline performance analysis
  • Provide the timeout mechanism for element
  • Provide the some feature to optimize asynchronous execution of pipeline

[2023.09.15 - v2.5.1 - Chunel]

  • Provide the fence feature
  • Provide the coordinator feature

[2023.11.06 - v2.5.2 - Chunel]

  • Optimize the message feature, allow configuring how to handle blocked writes, and reduce memory copy count
  • Add example content with simple implementations for different industries
  • Optimize scheduling performance

[2023.11.15 - v2.5.3 - Chunel]

  • Provide proto definition files
  • Add the mutable feature and provide dependency registration syntax sugar

[2024.01.05 - v2.5.4 - Chunel]

  • Provide test content, including performance and functional test cases
  • Optimize the event mechanism and support asynchronous waiting

[2024.07.18 - v2.6.0 - PaPaPig-Melody, Chunel]

  • Provide topological execution for pipeline
  • Provide a method to determine whether dependencies exist between elements
  • Provide Bazel build support
  • Optimize the perf feature

[2024.09.17 - v2.6.1 - Chunel]

  • Provide static execution for pipeline and a micro-task mechanism based on static execution
  • Provide pipeline pruning to remove duplicate dependencies between elements
  • Provide a method for element to remove dependencies
  • Optimize the event mechanism so asynchronous events can wait for completion
  • Release the CGraph-lite project, providing simple DAG construction and parameter passing. Its APIs are fully compatible and can be seamlessly switched to this project

[2024.11.16 - v2.6.2 - Chunel]

  • Optimize parameter mutex mechanism and retrieval performance
  • Fix abnormal waiting in auxiliary threads
  • Update tutorial content

[2025.03.16 - v3.0.0 - Chunel]

  • Provide the Python version
  • Provide the stage feature for synchronized running between elements
  • Update tutorial content

[2025.05.04 - v3.1.0 - Chunel]

  • Provide the full-featured Python version
  • Provide Python and C++ hybrid programming
  • Provide Python packaging and support installation with pip3 install PyCGraph

[2025.06.15 - v3.1.1 - Chunel]

  • Provide C# and Java versions
  • Provide the CODE_OF_CONDUCT.md document

[2025.09.06 - v3.1.2 - Chunel]

  • Provide the Go version
  • Optimize the message feature
  • Optimize the aspect feature

[2025.11.08 - v3.2.2 - Chunel]

  • Provide local save and load features
  • Rename PyCGraph to pycgraph and support installation with pip3 install pycgraph

[2026.03.24 - v3.2.3 - Chunel]

  • Optimize the message feature
  • Optimize stability

[2026.05.10 - v3.2.4 - Chunel]

  • Optimize the pycgraph feature
Appendix 2. Thanks

  • Thanks to the recommendation from awesome-cpp, we all know, it is the most authoritative recommendation list for cpp project in the world
  • Thanks to the recommendation from Taskflow Group: awesome-parallel-computing, and we always treat taskflow as a role model
  • Thanks to the recommendation from awesome-workflow-engines
  • Thanks to all developers in CONTRIBUTORS for their contributions to the project
  • Thanks to all friends who have provided suggestions and advice for the CGraph project. They are not listed one by one here. Everyone is always welcome to join and build together
Appendix 3. Contact