Can't run executable with FluxExecutor using Parsl, but can run it directly with Flux #5558

christopherwharrop-noaa · 2023-11-13T23:40:34Z

christopherwharrop-noaa
Nov 13, 2023

Up to this point I've been using the Intel compilers and MPI for developing Parsl code that uses Flux via Parsl's FluxExecutor. However, I'm now trying to make it work with GNU compilers and OpenMPI. I'm using the same exact program and configuration, just swapping the compiler and MPI. Well.... Intel+IntelMPI works. But gfortran+OpenMPI fails in the MPI_Init(). I've looked at the FAQ entries for OpenMPI and read that documentation several times. I'm not sure where I'm going wrong. I'm also having a hard time figuring out how to add debug flags since the problem happens when using Parsl, and not when running with flux run.

First a bit of basic info. I am installing Flux via a conda environment:

name: chiltepin-env

channels:
  - conda-forge

dependencies:
  - python
  - pip
  - flux-core
  - flux-sched
  - openmpi-mpicc
  - openmpi-mpicxx
  - openmpi-mpifort
  - pip:
    - parsl[monitoring]

This results in these versions being installed:

(chiltepin) admin@slurmfrontend:~/test/src$ conda list | grep flux
flux-core                 0.52.0          py311lua54he729abb_0    conda-forge
flux-sched                0.28.0          py311h85c2a0a_0    conda-forge

The OpenMPI plugins are installed:

(chiltepin) admin@slurmfrontend:~/test/src$ ompi_info | grep flux
[slurmfrontend:00616] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)
[slurmfrontend:00616] mca_base_component_repository_open: unable to open mca_osc_ucx: libucp.so.0: cannot open shared object file: No such file or directory (ignored)
[slurmfrontend:00616] mca_base_component_repository_open: unable to open mca_pml_ucx: libucp.so.0: cannot open shared object file: No such file or directory (ignored)
                MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component v4.1.6)
              MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v4.1.6)

And the OpenMPI wrappers point to very recent GNU compilers

(chiltepin) admin@slurmfrontend:~/test/src$ mpif90 --version
GNU Fortran (conda-forge gcc 12.3.0-3) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Finally, Parsl is the newest available:

(chiltepin) admin@slurmfrontend:~/test/src$ ./parsl_hello.py 
Running Parsl version: 2023.10.23

The weird part is that if I take Parsl out of the equation, and just start up a Flux instance in my containerized Slurm cluster, it works. Sort of. I mean, it runs and produces the expected output. But it also spews a TON of warnings.

(chiltepin) admin@slurmfrontend:~/test/src$ srun --mpi=pmi2 -N3 --tasks-per-node=1 -c2 --pty /opt/miniconda3/envs/chiltepin/bin/flux start
(chiltepin) admin@slurmnode1:~/test/src$ flux run -n 6 ./mpi_hello.exe
[slurmnode2:00270] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: slurmnode2
--------------------------------------------------------------------------
[slurmnode3:00256] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)
[slurmnode3:00257] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.
not available due to restrictive ptrace settings.


The vader shared memory BTL will fall back on another single-copy
The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.
mechanism if one is available. This may result in lower performance.


  Local host: slurmnode3
  Local host: slurmnode3
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[slurmnode1:02527] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: slurmnode1
--------------------------------------------------------------------------
[slurmnode1:02528] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: slurmnode1
--------------------------------------------------------------------------
[slurmnode2:00268] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: slurmnode2
--------------------------------------------------------------------------
Hello world from host slurmnode1, rank 0 out of 6
Hello world from host slurmnode1, rank 1 out of 6
Hello world from host slurmnode2, rank 2 out of 6
Hello world from host slurmnode2, rank 3 out of 6
Hello world from host slurmnode3, rank 4 out of 6
Hello world from host slurmnode3, rank 5 out of 6

I can issue flux commands directly and run my executable no problem with the caveat of all the warnings in the above output. But, whenever I try to run the executable as part of a Parsl Bash App using the FluxExecutor, I get output that looks like this:

(chiltepin) admin@slurmfrontend:~/test/src$ more parsl_flux_mpi_hello_compile.err
--> executable follows <--

    cd ./
    mpif90 -o mpi_hello.exe mpi_hello.f90
    ./mpi_hello.exe
    
--> end executable <--
[slurmnode2:00197] PMI_Init [pmix_flux.c:389:flux_init]: Operation failed
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  pmix init failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_init failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "Error" (-1) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[slurmnode2:00197] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

Answered by christopherwharrop-noaa

Nov 14, 2023

Ok. flux config builtin pmi_library_path confirms your guess. And, well, setting export FLUX_PMI_LIBRARY_PATH=/opt/miniconda3/envs/chiltepin/lib/flux/libpmi.so fixes it! I guess we still don't have an explanation yet as to why FLUX_PMI_LIBRARY_PATH isn't set. But, after setting that in my Parsl App before running flux pmi --method=libpmi:$FLUX_PMI_LIBRARY_PATH barrier it works. I added the same export command to the Parsl app that runs my hello MPI executable and it also works now, too on multiple tasks across multiple nodes. So, yay!

def pmi_barrier(parsl_resource_specification={}):
    return '''                                                                                            …

View full answer

garlick · 2023-11-14T01:24:53Z

garlick
Nov 14, 2023
Maintainer

Well, on the warnings (the easier part):

try running sudo sysctl kernel.yama.ptrace_scope=0 across your compute nodes to fix vader
seems like the openib BTL is missing some prerequisite. The openmpi people will probably tell you to use UCX instead. But if you just want to paper over that one I think there is a "OMPI_MCA_btl_" prefixed environment variable that can be set to tell openmpi to just use tcp,vader,self.

This error when running under parsl

PMI_Init [pmix_flux.c:389:flux_init]: Operation failed

is the critical one to solve I guess? This is an early failure in the openmpi flux pmix MCA plugin. I think it will fail with that error if the environment variables PMI_FD, PMI_RANK, and PMI_SIZE are not getting through to the mpi task (those are set by flux).

We've recently had other issues (#5460) with those flux plugins in openmpi but I believe they manifest as a hang with UCX rather than this. The solution to that one is add the flux-pmix project into the setup and run with the -o pmi=pmix run option. But you probably don't want to go down that road just yet. tcp should work just fine with the current plugins AFAIK.

0 replies

garlick · 2023-11-14T01:45:43Z

garlick
Nov 14, 2023
Maintainer

Actually there isn't anything really different in the way that openmpi is bootstrapping here vs intel MPI. Both would have dlopened the flux libpmi.so and then used the PMI-1 wire protocol over the PMI_FD to do its handshake. I'm not sure why it is suddenly failing.

As a sanity check you could try running flux pmi barrier instead of the MPI program and just verify that it works from parsl. That's just a standalone test of the PMI environment. But again, I can't see why it wouldn't work if intel mpi was working before.

0 replies

garlick · 2023-11-14T02:32:24Z

garlick
Nov 14, 2023
Maintainer

Ah from #5079 (also regarding a conda install) OMPI_MCA_btl=self,tcp should take care of the warnings.

0 replies

christopherwharrop-noaa · 2023-11-14T15:57:28Z

christopherwharrop-noaa
Nov 14, 2023
Author

Thank you so much for all of this great information. I have done two things thus far:

I added export OMPI_MCA_btl=self,tcp to .bashrc and .bash_profile so that it is set when using my containerized Slurm cluster.
I added a Parsl task to run flux pmi barrier to check whether basic PMI is working.

The first one did remove a lot of warnings so that when I now run my executable without using Parsl via flux run I get clean output:

(chiltepin) admin@slurmfrontend:~/test/src$ srun -N3 --pty --mpi=pmi2 --tasks-per-node=1 -c2 /opt/miniconda3/envs/chiltepin/bin/flux start
(chiltepin) admin@slurmnode1:~/test/src$ flux run -n 6 ./mpi_hello.exe 
Hello world from host slurmnode1, rank 0 out of 6
Hello world from host slurmnode1, rank 1 out of 6
Hello world from host slurmnode2, rank 2 out of 6
Hello world from host slurmnode2, rank 3 out of 6
Hello world from host slurmnode3, rank 4 out of 6
Hello world from host slurmnode3, rank 5 out of 6
(chiltepin) admin@slurmnode1:~/test/src$

I have also verified that PMI is working from Parsl using this Bash App:

# Test PMI                                                                                                                                                                                                                                                                     
@bash_app
def pmi_barrier(parsl_resource_specification={}):
    return '''                                                                                                                                                                                                                                                                 
    flux pmi barrier > parsl_flux_pmi_barrier.txt                                                                                                                                                                                                                              
    '''

Called like so:

# Check the Flux pmi status                                                                                                                                                                                                                                                    
p = pmi_barrier(parsl_resource_specification={"num_tasks": 6, "num_nodes": 3}).result()

And it's output is:

(base) christopher.w.harrop@harrop-lt src % cat parsl_flux_pmi_barrier.txt 
ƒjL6Vg3: completed pmi barrier on 6 tasks in 0.001s.
(base) christopher.w.harrop@harrop-lt src %

My Parsl Apps that run flux resource list and flux pmi barrier and mpif90 -o mpi_hello.exe mpi_hello.f90 all work as expected without any issues. But, the app that runs the executable is still failing with the following (I ran using only one task to clarify the output):

(chiltepin) admin@slurmfrontend:~/test/src$ more parsl_flux_mpi_hello_run.err
--> executable follows <--

    # Flux requires I_MPI_PMI_LIBRARY to be unset because it provides its own PMI
    cd ./
    ./mpi_hello.exe
    
--> end executable <--
[slurmnode3:00526] PMI_Init [pmix_flux.c:389:flux_init]: Operation failed
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  pmix init failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_init failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "Error" (-1) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[slurmnode3:00526] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

9 replies

garlick Nov 14, 2023
Maintainer

Just noticed that the PMI_Init() error is calling out a line number, here:

https://github.com/open-mpi/ompi/blob/3a1cd5ff34e2d5c62ff0554faa6a6f5eaf18324b/opal/mca/pmix/flux/pmix_flux.c#L388-L392

That calls an internal PMI_Init() function:

https://github.com/open-mpi/ompi/blob/3a1cd5ff34e2d5c62ff0554faa6a6f5eaf18324b/opal/mca/pmix/flux/pmix_flux.c#L137C1-L149C2

So it would fail if

FLUX_PMI_LIBRARY_PATH is not set
dlopen of the library pointed to by the above fails
the aforementioned issues with the environment, already ruled out by the flux pmi barrier test

You could try changing the flux pmi test a little bit to make it more like the above scenario:

bash -c "flux pmi --method=libpmi:\$FLUX_PMI_LIBRARY_PATH barrier"

Edit: the quoting might get out of hand when embedded in python script. I'd just make a shell script

#!/bin/sh
flux pmi --method=libpmi:$FLUX_PMI_LIBRARY_PATH barrier

and run that from parsl.

christopherwharrop-noaa Nov 14, 2023
Author

I put a env command in the App to print out the Flux variables and it's clear that FLUX_PMI_LIBRARY_PATH is not set:

@bash_app
def pmi_barrier(parsl_resource_specification={}):
    return '''                                                                                                                                                                                                                                                                 
    env | grep -i flux > parsl_flux_pmi_barrier.txt                                                                                                                                                                                                                            
    flux pmi barrier >> parsl_flux_pmi_barrier.txt                                                                                                                                                                                                                             
    '''

(chiltepin) admin@slurmfrontend:~/test/src$ cat parsl_flux_pmi_barrier.txt
FLUX_TASK_RANK=0
OMPI_MCA_btl_vader_backing_directory=/tmp/flux-OhIEW3/jobtmp-0-ƒnYtwBV
FLUX_KVS_NAMESPACE=job-29896998912
FLUX_TERMINUS_SESSION=0
FLUX_JOB_NNODES=1
FLUX_JOB_SIZE=1
FLUX_JOB_TMPDIR=/tmp/flux-OhIEW3/jobtmp-0-ƒnYtwBV
FLUX_URI=local:///tmp/flux-OhIEW3/local-1
FLUX_JOB_ID=ƒnYtwBV
FLUX_TASK_LOCAL_ID=0
ƒnYtwBV: completed pmi barrier on 1 tasks in 0.000s.

I then tried the altered pmi barrier test and it failed, of course. So I attempted to set it manually, but I'm not sure whether it needs setting on every compute node, or just in the Parsl app. These appear to be my choices:

(chiltepin) admin@slurmfrontend:~$ find /opt/miniconda3/envs/chiltepin -name '*pmi*' | grep lib
/opt/miniconda3/envs/chiltepin/lib/pkgconfig/pmix.pc
/opt/miniconda3/envs/chiltepin/lib/pkgconfig/flux-pmi.pc
/opt/miniconda3/envs/chiltepin/lib/pmix
/opt/miniconda3/envs/chiltepin/lib/openmpi/mca_pmix_pmix3x.so
/opt/miniconda3/envs/chiltepin/lib/openmpi/mca_pmix_flux.so
/opt/miniconda3/envs/chiltepin/lib/openmpi/mca_pmix_isolated.so
/opt/miniconda3/envs/chiltepin/lib/openmpi/mca_ess_pmi.so
/opt/miniconda3/envs/chiltepin/lib/flux/libpmi.so.0
/opt/miniconda3/envs/chiltepin/lib/flux/libpmi.so
/opt/miniconda3/envs/chiltepin/lib/flux/libpmi2.so
/opt/miniconda3/envs/chiltepin/lib/flux/libpmi2.so.0
/opt/miniconda3/envs/chiltepin/lib/flux/libpmi.so.0.0.0
/opt/miniconda3/envs/chiltepin/lib/flux/libpmi2.so.0.0.0
(chiltepin) admin@slurmfrontend:~$

I tried /opt/miniconda3/envs/chiltepin/lib/openmpi/mca_pmix_flux.so and /opt/miniconda3/envs/chiltepin/lib/flux/libpmi2.so but I'm not entirely sure I did it correctly. In both cases I get something that looks like:

(chiltepin) admin@slurmfrontend:~/test/src$ more flux_998b9281-ef9d-4cf7-a39d-4ee95fa460ad/1_stderr.txt 
2023-11-14 18:32:31,121 [INFO] Input : /home/admin/test/src/flux_998b9281-ef9d-4cf7-a39d-4ee95fa460ad/1_in.pkl
2023-11-14 18:32:31,121 [INFO] Output : /home/admin/test/src/flux_998b9281-ef9d-4cf7-a39d-4ee95fa460ad/1_out.pkl
2023-11-14 18:32:31,125 [INFO] Read input pickle file
flux-pmi: /opt/miniconda3/envs/chiltepin/lib/flux/libpmi2.so:  missing required PMI_* symbols
2023-11-14 18:32:31,139 [INFO] Finished execution
(chiltepin) admin@slurmfrontend:~/test/src$

garlick Nov 14, 2023
Maintainer

This one I think:

/opt/miniconda3/envs/chiltepin/lib/flux/libpmi.so

I wonder why it is not getting set. You can also ask flux for the value with flux config builtin pmi_library_path

christopherwharrop-noaa Nov 14, 2023
Author

Ok. flux config builtin pmi_library_path confirms your guess. And, well, setting export FLUX_PMI_LIBRARY_PATH=/opt/miniconda3/envs/chiltepin/lib/flux/libpmi.so fixes it! I guess we still don't have an explanation yet as to why FLUX_PMI_LIBRARY_PATH isn't set. But, after setting that in my Parsl App before running flux pmi --method=libpmi:$FLUX_PMI_LIBRARY_PATH barrier it works. I added the same export command to the Parsl app that runs my hello MPI executable and it also works now, too on multiple tasks across multiple nodes. So, yay!

def pmi_barrier(parsl_resource_specification={}):
    return '''                                                                                                                                                                                                                                                                 
    env | grep -i flux > parsl_flux_pmi_barrier.txt                                                                                                                                                                                                                            
    env | grep -i pmi >> parsl_flux_pmi_barrier.txt                                                                                                                                                                                                                            
    export FLUX_PMI_LIBRARY_PATH=/opt/miniconda3/envs/chiltepin/lib/flux/libpmi.so                                                                                                                                                                                             
    flux pmi --method=libpmi:$FLUX_PMI_LIBRARY_PATH barrier >> parsl_flux_pmi_barrier.txt                                                                                                                                                                                      
    '''

# Run the hello MPI program with Intel                                                                                                                                                                                                                                         
@bash_app
def mpi_hello(dirpath, stdout=None, stderr=None, app="./mpi_hello.exe", parsl_resource_specification={}):
    return '''                                                                                                                                                                                                                                                                 
    cd {}                                                                                                                                                                                                                                                                      
    export FLUX_PMI_LIBRARY_PATH=/opt/miniconda3/envs/chiltepin/lib/flux/libpmi.so                                                                                                                                                                                             
    {}                                                                                                                                                                                                                                                                         
    '''.format(dirpath, app)

Gives:

(chiltepin) admin@slurmfrontend:~/test/src$ more parsl_flux_pmi_barrier.txt
FLUX_TASK_RANK=5
OMPI_MCA_btl_vader_backing_directory=/tmp/flux-6ziP6O/jobtmp-2-ƒjgrKcs
FLUX_KVS_NAMESPACE=job-28017950720
FLUX_TERMINUS_SESSION=0
FLUX_JOB_NNODES=3
FLUX_JOB_SIZE=6
FLUX_JOB_TMPDIR=/tmp/flux-6ziP6O/jobtmp-2-ƒjgrKcs
FLUX_URI=local:///tmp/flux-6ziP6O/local-2
FLUX_JOB_ID=ƒjgrKcs
FLUX_TASK_LOCAL_ID=1
PMI_SIZE=6
PMI_FD=17
PMI_RANK=4
PMI_SIZE=6
PMI_FD=21
PMI_RANK=5
ƒjgrKcs: completed pmi barrier on 6 tasks in 0.002s.

(chiltepin) admin@slurmfrontend:~/test/src$ more parsl_flux_mpi_hello_run.out
Hello world from host slurmnode1, rank 0 out of 6
Hello world from host slurmnode1, rank 1 out of 6
Hello world from host slurmnode2, rank 2 out of 6
Hello world from host slurmnode2, rank 3 out of 6
Hello world from host slurmnode3, rank 4 out of 6
Hello world from host slurmnode3, rank 5 out of 6

Answer selected by christopherwharrop-noaa

christopherwharrop-noaa Nov 14, 2023
Author

Thank you very much for taking the time to help me with this! I know your time is valuable and I'm very grateful for your assistance.

garlick Nov 14, 2023
Maintainer

I'm glad that worked, and thanks for providing the detailed output - that really helps.

I just realized maybe FLUX_PMI_LIBRARY_PATH is not being set because we set that in the flux(1) command driver and your parsl launches are using that python API. Bug! However, I think it was already fixed in fux-core-0.56.0 by #5535, as luck would have it.

christopherwharrop-noaa Nov 14, 2023
Author

Aha! I'm happy to use the latest except that it's not yet available via Conda. I'll have to weigh the pros/cons of installation via Spack vs Conda and see what works best. I like the simplicity of Conda and it seems more straightforward when also needing to manage pip installed packages like Parsl.

garlick Nov 14, 2023
Maintainer

I'm not sure who is keeping conda packages up to date though. I don't think we are.

christopherwharrop-noaa · 2023-11-14T16:05:51Z

christopherwharrop-noaa
Nov 14, 2023
Author

I have been doing this inside a containerized Slurm cluster because it's easier to control the environment. I'm going to try reconfiguring for Slurm on our on-prem system and try running directly there without the container to verify whether the behavior is reproducible there.

0 replies

christopherwharrop-noaa · 2023-11-14T16:32:19Z

christopherwharrop-noaa
Nov 14, 2023
Author

In case it helps (more information is usually better), this is my Parsl config for the FluxExecutor. My containerized Slurm cluster has 3 nodes with 2 cores per node.

# Update to import config for your machine                                                                                                                                                                                                                                     
config = Config(
    executors=[
        FluxExecutor(
            label="flux",
            # Start Flux with srun and tell it there are 40 cores per node                                                                                                                                                                                                     
            launch_cmd='srun --mpi=pmi2 --tasks-per-node=1 -c2 ' + FluxExecutor.DEFAULT_LAUNCH_CMD,
            provider=SlurmProvider(
                channel=LocalChannel(),
                nodes_per_block=3,
                init_blocks=1,
                partition='slurmpar',
                account='',
                walltime='00:10:00',
                launcher=SimpleLauncher(),
                worker_init='''                                                                                                                                                                                                                                                
''',
            ),
        )
    ],
)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't run executable with FluxExecutor using Parsl, but can run it directly with Flux #5558

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Can't run executable with FluxExecutor using Parsl, but can run it directly with Flux #5558

christopherwharrop-noaa Nov 13, 2023

Replies: 6 comments · 9 replies

garlick Nov 14, 2023 Maintainer

garlick Nov 14, 2023 Maintainer

garlick Nov 14, 2023 Maintainer

christopherwharrop-noaa Nov 14, 2023 Author

garlick Nov 14, 2023 Maintainer

christopherwharrop-noaa Nov 14, 2023 Author

garlick Nov 14, 2023 Maintainer

christopherwharrop-noaa Nov 14, 2023 Author

christopherwharrop-noaa Nov 14, 2023 Author

garlick Nov 14, 2023 Maintainer

christopherwharrop-noaa Nov 14, 2023 Author

garlick Nov 14, 2023 Maintainer

christopherwharrop-noaa Nov 14, 2023 Author

christopherwharrop-noaa Nov 14, 2023 Author

christopherwharrop-noaa
Nov 13, 2023

Replies: 6 comments 9 replies

garlick
Nov 14, 2023
Maintainer

garlick
Nov 14, 2023
Maintainer

garlick
Nov 14, 2023
Maintainer

christopherwharrop-noaa
Nov 14, 2023
Author

garlick Nov 14, 2023
Maintainer

christopherwharrop-noaa Nov 14, 2023
Author

garlick Nov 14, 2023
Maintainer

christopherwharrop-noaa Nov 14, 2023
Author

christopherwharrop-noaa Nov 14, 2023
Author

garlick Nov 14, 2023
Maintainer

christopherwharrop-noaa Nov 14, 2023
Author

garlick Nov 14, 2023
Maintainer

christopherwharrop-noaa
Nov 14, 2023
Author

christopherwharrop-noaa
Nov 14, 2023
Author