accelerator framework/cuda: still not entirely fixed

[hppritcha]

I was checking out head of the v5.0.x branch in high expectations that it would work well on our nvidia + HPE SS11 (aka libfabric) system, but alas, if my application doesn't use cudA, yet is linked against a ompi v5.0.x with all the recent accelerator/cuda changes in place, and configured for CUDA support, things don't work right.

Hello, world, I am 1 of 2, (Open MPI v5.0.0rc9, package: Open MPI hpp@ch-fe1 Distribution, ident: 5.0.0rc9, repo rev: v5.0.0rc9-287-g5d87f3e6, Unreleased developer copy, 141)
Hello, world, I am 0 of 2, (Open MPI v5.0.0rc9, package: Open MPI hpp@ch-fe1 Distribution, ident: 5.0.0rc9, repo rev: v5.0.0rc9-287-g5d87f3e6, Unreleased developer copy, 141)
--------------------------------------------------------------------------
The call to cuEventDestory failed. This is a unrecoverable error and will
cause the program to abort.
  cuEventDestory return value:   709
Check the cuda.h file for what the return value means.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The call to cuEventDestory failed. This is a unrecoverable error and will
cause the program to abort.
  cuEventDestory return value:   709
Check the cuda.h file for what the return value means.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The call to cuEventDestory failed. This is a unrecoverable error and will
cause the program to abort.
  cuEventDestory return value:   709
Check the cuda.h file for what the return value means.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The call to cuEventDestory failed. This is a unrecoverable error and will
cause the program to abort.
  cuEventDestory return value:   709
Check the cuda.h file for what the return value means.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The call to cuEventDestory failed. This is a unrecoverable error and will
cause the program to abort.
  cuEventDestory return value:   709
Check the cuda.h file for what the return value means.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The call to cuEventDestory failed. This is a unrecoverable error and will
cause the program to abort.
  cuEventDestory return value:   709
Check the cuda.h file for what the return value means.

It looks like holes may have been plugged for OB1 (if i set the pml to use ob1 I don't see these messages), but such is not the case when using other PMLs apparently.

[wckzhang]

ACK, there shouldn't be any event create/destroy if cuda init hasn't been called, I fixed a bug in the ob1 path to fix this but must have missed it. It should be easy enough to figure it out.

[hppritcha]

if you'd like to fix go ahead and assign this to yourself. i've got some other things to fix elsewhere....

[wckzhang]

I wasn't able to reproduce this (Built 5.0.x head with cuda and application w/o cuda and ran using the OFI MTL) when I tried so Howard said he'd take a look at reproducing.

[hppritcha]

this problem vanishes if I don't use --enable-mca-dso configure option.

[hppritcha]

i'm fine with not using the --enable-mca-dso option. I think this problem is somewhat related to the problems that PR #10949 was seeking to address.

[rhc54]

Not being able to enable that option seems like a pretty drastic solution for the general community. The other PR has been sitting there waiting to be committed for 3 months now - can the community find some way to make that happen? Seems like there are a bunch of PRs stuck in that situation.

[jsquyres]

If something breaks when you use --enable-mca-dso, it usually means that there is a genuine bug -- perhaps in ordering of shutdown or somesuch.

[hppritcha]

well what's happening i believe is that the accelerator framework is being dlclosed prior to smcuda (part of btl framework).

[jsquyres]

well what's happening i believe is that the accelerator framework is being dlclosed prior to smcuda (part of btl framework).

That seems like a legit bug that should be fixed, right?

[hppritcha]

I think there may be something more complicated going on. it appears that on the system i'm using, that if i load a "cudatoolkit" module, this causes the call to the smcuda btl to initialize although the accelerator framework cuda component failed to initialized. The magic in the cudatoolkit module appears to be adding libcudart.so into the LD_LIBRARY_PATH.

Given this, i think this is some cuda support specific problem. I'm confused though why the problem only shows up when building with -enable-mca-dso configure option.