[CANN] Fix deadlock during parallel execution

[abeclowicz]

Motivation and Context

During parallel execution (ORT_PARALLEL), graph partitioning and compilation may be distributed across different worker threads. However, the GraphEngine function aclgrphBuildFinalize is thread-sensitive and must be executed by the exact same thread that initialized the engine (aclgrphBuildInitialize) - otherwise, this cross-thread mismatch results in a deadlock.

Description

• The initialization and finalization calls are now executed inside a dedicated background thread (g_ge_thread), synchronized via std::promise and std::future to ensure proper execution flow.
• Any exceptions thrown inside this background thread are safely caught and rethrown in the parent thread to ensure proper error propagation.

Reproduce the Issue

Below is the simplified C++ reproduction code. The inference() function is the C/C++ sample taken directly from the ONNX Runtime documentation, with ORT_PARALLEL mode enabled.

#include <sys/wait.h>
#include <thread>
#include <unistd.h>
#include <vector>

extern void inference();

void run(int num_proc, int num_threads) {
    std::vector<pid_t> child_pids;
    child_pids.reserve(num_proc);

    for (int i = 0; i < num_proc; ++i) {
        auto pid = fork();

        if (pid == 0) {
            std::vector<std::thread> threads;
            threads.reserve(num_threads);

            for (int j = 0; j < num_threads; ++j) {
                threads.emplace_back(std::thread([] {
                    // The program deadlocks inside here
                    // during aclgrphBuildFinalize
                    inference();
                }));
            }

            for (auto& thread : threads) {
                thread.join();
            }

            exit(0);
        }
        else if (pid > 0) {
            child_pods.push_back(pid);
        }
    }

    for (auto pid : child_pids) {
        int status;
        waitpid(pid, &status, 0);
    }
}

int main() {
    run(100, 10); // stress test, deadlock occurs even with run(1, 1) or during a direct inference() call
    return 0;
}

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR addresses a CANN GE GraphEngine deadlock in ORT_PARALLEL by ensuring ge::aclgrphBuildInitialize and ge::aclgrphBuildFinalize execute on the same dedicated thread, and synchronizing init/finalize with std::promise/std::future.

Changes:

• Introduce a dedicated background thread to run aclgrphBuildInitialize and later aclgrphBuildFinalize in the same thread context.
• Add promise/future synchronization to block callers until initialization completes and to trigger finalization during provider shutdown.
• Update CANN provider shutdown to signal the finalize path and join the background thread.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
onnxruntime/core/providers/cann/cann_graph.cc	Adds GE init/finalize background thread plus init/final synchronization primitives.
onnxruntime/core/providers/cann/cann_execution_provider.cc	Signals GE finalize via promise and joins the background thread during DeleteRegistry().

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.