Skip to content

aurora

Jump to bottom
bonachea edited this page Apr 8, 2026 · 10 revisions

Using UPC++ on ALCF Aurora

ALCF Aurora

This document is a continuous work-in-progress, intended to provide up-to-date information on a public install maintained by (or in collaboration with) the UPC++ team. However, systems are constantly changing. So, please report any errors or omissions in the issue tracker.

Typically installs of UPC++ are maintained only for the current default versions of the system-provided environment modules such as for oneAPI.

This document is not a replacement for the documentation provided by the centers, and assumes general familiarity with the use of the system.

General

Stable installs are available through environment modules. A wrapper is used to transparently dispatch commands such as upcxx to an install appropriate to the currently loaded compiler (e.g. oneapi) environment module(s).

Environment Modules

In order to access the UPC++ installation on Aurora, one must run

$ module use /lus/flare/projects/CSC250STPM17_CNDA/aurora/modulefiles

to add a non-default directory to the MODULEPATH before the UPC++ environment modules will be accessible. We recommend inclusion of this command in one's shell startup files, such as $HOME/.login or $HOME/.bash_profile.

If not adding the command to one's shell startup files, the module use ... command will be required once per login shell or PBS job in which you need a upcxx environment module.

Environment modules provide three alternative configurations of the UPC++ library:

  • upcxx-ze-native This module includes experimental memory kinds support for Aurora's Intel GPUs, a UPC++ feature that enables communication to/from GPU memory via upcxx::copy on upcxx::global_ptr<T, upcxx::ze_device>. When using this module, copy operations on ze_device memory leverage network-direct RDMA ("native" memory kinds). However, use of this capability is currently in an experimental/prototype status.
  • upcxx-ze-reference This module also includes experimental memory kinds support for Aurora's Intel GPUs. However, it uses a "reference" implementation in which copy operations on ze_device memory are staged through host memory and do not leverage network-direct RDMA. This will generally result in lower performance than use of upcxx-ze, but enables development of applications using ze_device under conditions where the experimental/prototype support for native memory kinds may not be usable.
  • upcxx This module omits support for constructing an active upcxx::device_allocator<upcxx::ze_device> object, resulting in a small potential speed-up for applications which do not require a "ZE-aware" build of UPC++.

By default each module above will select the latest recommended version of the UPC++ library. One can see the installed versions with a command like module avail upcxx and optionally explicitly select a particular version with a command of the form: module load upcxx/20XX.YY.ZZ.

On Aurora, the UPC++ environment modules select a default network of ofi. You can optionally specify this explicitly on the compile line with upcxx -network=ofi ....

Job launch

The upcxx-run utility provided with UPC++ is a relatively simple wrapper, which in the case of Aurora uses aprun. To have full control over process placement, thread pinning and GPU allocation, users are advised to consider launching their UPC++ applications directly with aprun. However, one should do so only with the upcxx or upcxx-ze environment module loaded to ensure the appropriate environment variable settings.

If you would normally have passed -shared-heap to upcxx-run, then it is particularly important that both UPCXX_SHARED_HEAP_SIZE and GASNET_MAX_SEGSIZE be set accordingly. The values of those and other potentially relevant environment variables set (or inherited) by upcxx-run can be listed by adding -show to your upcxx-run command (which will print useful information but not run anything). Additional information is available in the Advanced Job Launch chapter of the UPC++ v1.0 Programmer's Guide.

Single-node runs

On a system like Aurora, there are multiple complications related to launch of executables compiled for -network=smp such that no use of aprun (or simple wrappers around it) can provide a satisfactory solution in general. Therefore, we recommend that for single-node (shared memory) application runs on Aurora, one should compile for the default network (ofi). It is also acceptable to use -network=mpi, such as may be required for some hybrid applications (UPC++ and MPI in the same executable). However, note that in multi-node runs -network=mpi imposes a significant performance penalty.

Batch jobs

By default, PBS jobs (both batch and interactive) do not inherit the necessary settings from the submit-time environment, meaning both the module use ... and module load upcxx may be required in batch jobs which use upcxx-run. This is shown in the example below.

Interactive example:

aurora-uan-0010$ module use /lus/flare/projects/CSC250STPM17_CNDA/aurora/modulefiles

aurora-uan-0010$ module load upcxx

aurora-uan-0010$ upcxx --version
UPC++ version 2025.10.0  / gex-2025.8.0-0-ge3628f258
Citing UPC++ in publication? Please see: https://upcxx.lbl.gov/publications
Copyright (c) 2025, The Regents of the University of California,
through Lawrence Berkeley National Laboratory.
https://upcxx.lbl.gov

Intel(R) oneAPI DPC++/C++ Compiler 2025.2.0 (2025.2.0.20250605)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/aurora/25.190.0/oneapi/compiler/2025.2/bin/compiler
Configuration file: /opt/aurora/25.190.0/oneapi/compiler/2025.2/bin/compiler/../icpx.cfg

aurora-uan-0010$ upcxx -O hello-world.cpp -o hello-world.x

aurora-uan-0010$ qsub -l walltime=10:00,filesystems=home -A ... -l select=2 -I

x4204c0s5b0n0$ module use /lus/flare/projects/CSC250STPM17_CNDA/aurora/modulefiles
x4204c0s5b0n0$ module load upcxx

x4204c0s5b0n0$ upcxx-run -n 4 -N 2 ./hello-world.x
Hello world from process 0 out of 4 processes
Hello world from process 1 out of 4 processes
Hello world from process 2 out of 4 processes
Hello world from process 3 out of 4 processes

CMake

A UPCXX CMake package is provided in the UPC++ install on Aurora, as described in README.md. Thus with the upcxx environment module loaded, CMake should "just work".

Known Issues

Correctness problems with intensive communication on HPE Slingshot-11

Currently, there are known issues with the vendor's communications software stack below UPC++ and GASNet-EX which may negatively impact certain communication-intensive UPC++ applications (e.g. those concurrently sending large numbers of RPCs to one or more processes).

Impacts observed have included crashes and hangs of correct UPC++ applications. Or course, either of those failure modes can be the result of other issues. If you believe your application is impacted, please follow the steps below.

  1. Try running your application on a system with a network other than Slingshot-11 (but not Slingshot-10 which has a similar, but distinct, issue). If the failures persist, then the problem is not the one described here. You should look for defects in your application, or for other defects in UPC++ or external software.
  2. If you have observed crashes, but not hangs, then try running your application with GASNET_OFI_RECEIVE_BUFF_SIZE=recv in the environment. This disables use of a feature linked to the known source of crashes, but may result in a small reduction in RPC performance.
  3. If you have observed hangs, then try running your application with all of the following environment variable settings:
    GASNET_OFI_RECEIVE_BUFF_SIZE=recv
    FI_OFI_RXM_RX_SIZE=8192
    FI_CXI_DEFAULT_CQ_SIZE=13107200
    FI_MR_CACHE_MONITOR=memhooks
    FI_CXI_RX_MATCH_MODE=software
    FI_CXI_REQ_BUF_MIN_POSTED=10
    FI_CXI_REQ_BUF_SIZE=25165824
    These settings will have negative impact on both performance and on memory use. However, in most cases they have been seen to be sufficient to eliminate the problem(s).

If none of the options above resolves crashes or hangs of your communication-intensive UPC++ application, you can seek assistance using the issue tracker.

Information about UPC++ installs on other production systems

Please report any errors or omissions in the issue tracker.

Clone this wiki locally