Retain the original path to Kokkos

[aprokop]

This PR is simply to facilitate the discussion of the best approach to provide transitive path to Kokkos when using ArborX.

The current approach was taken from Cabana.

Feel free to close if you think this is useless.

[masterleinad]

Since <PackageName>_ROOT and CMAKE_PREFIX_PATH take precedence I guess it is fine to use this as default value.

[dalg24]

Since <PackageName>_ROOT and CMAKE_PREFIX_PATH take precedence I guess it is fine to use this as default value.

Except that this would bypass the search procedure

[masterleinad]

Except that this would bypass the search procedure

Are you concerned about step 4 to step 9 as well?

[aprokop]

Here's my (admittedly incomplete) understanding of how things will work.

Per find_dependency() documentation:

It is designed to be used in a Package Configuration File (<PackageName>Config.cmake). find_dependency forwards the correct parameters for QUIET and REQUIRED which were passed to the original find_package() call. Any additional arguments specified are forwarded to find_package().

Thus, from this point on we will consider find_dependency(Kokkos) be the same as find_package(Kokkos REQUIRED QUIET), as those are additional options we used for finding Kokkos in the root CMake.

Per find_package() documentation, if only the following options are provided
(EXACT, QUIET, MODULE, REQUIRED, COMPONENTS, OPTIONAL_COMPONENTS, NO_POLICY_SCOPE), it starts in Module mode:

The command has two modes by which it searches for packages: “Module” mode and “Config” mode. The above signature selects Module mode. If no module is found the command falls back to Config mode, described below. This fall back is disabled if the MODULE option is given.

In Module mode, In Module mode, CMake searches for a file called Find.cmake.We know that there is noFindKokkos.cmake`. Thus, we know that we will fall back to Config mode.

Config mode search attempts to locate a configuration file provided by the package to be found. A cache entry called <PackageName>_DIR is created to hold the directory containing the file. <snip> The command searches for a file called <PackageName>Config.cmake or <lower-case-package-name>-config.cmake for each name specified. <snip> The search procedure is specified below. Once found, the configuration file is read and processed by CMake.

However:

If <PackageName>_DIR has been set to a directory not containing a configuration file CMake will ignore it and search from scratch.

So it sounds like indeed it shortcuts the search procedure.

[aprokop]

To further investigate it, lets see what happens when trying to build examples against ArborX installation:

$  cmake -DCMAKE_PREFIX_PATH="$HOME/local/opt/arborx-tmp;$HOME/local/opt/kokkos-check/" -DCMAKE_FIND_DEBUG_MODE=ON ..
<snip>
Checking prefix [/home/xap/local/opt/arborx-tmp/]
Checking file [/home/xap/local/opt/arborx-tmp/ArborXConfig.cmake]
Checking file [/home/xap/local/opt/arborx-tmp/arborx-config.cmake]
Checking file [/home/xap/local/opt/arborx-tmp/lib/cmake/ArborX/ArborXConfig.cmake]
Checking prefix [/home/xap/local/opt/arborx-tmp/]
Checking file [/home/xap/local/opt/arborx-tmp/KokkosConfig.cmake]    
Checking file [/home/xap/local/opt/arborx-tmp/kokkos-config.cmake]           
Checking prefix [/home/xap/local/opt/kokkos-check/]                  
Checking prefix [/home/xap/local/opt/spack/opt/spack/linux-ubuntu18.04-broadwell/gcc-7.4.0/boost-1.68.0-syfratmpslzh7vpultgho6h54mwh4g34/]
Checking file [/home/xap/local/opt/spack/opt/spack/linux-ubuntu18.04-broadwell/gcc-7.4.0/boost-1.68.0-syfratmpslzh7vpultgho6h54mwh4g34/KokkosConfig.cmake]
<continued>

We used a (non-existing) kokkos-check in CMAKE_PREFIX_DIR, and see it examined when looking for FindKokkos.cmake.

Now, lets see what happens when we have

if(NOT DEFINED Kokkos_DIR)                                                                                                                                                                                                                                                                                                                                                                 
  set(Kokkos_DIR /home/xap/local/opt/kokkos/lib/cmake/Kokkos)                                                                                                                                                                                                                                                                                                                              
endif()
find_dependency(Kokkos) 

in ArborXConfig.cmake:

$ VERBOSE=1 cmake -DCMAKE_PREFIX_PATH="$HOME/local/opt/arborx-tmp;$HOME/local/opt/kokkos-check/" -DCMAKE_FIND_DEBUG_MODE=ON ..
Checking prefix [/home/xap/local/opt/arborx-tmp/]
Checking file [/home/xap/local/opt/arborx-tmp/ArborXConfig.cmake]
Checking file [/home/xap/local/opt/arborx-tmp/arborx-config.cmake]
Checking file [/home/xap/local/opt/arborx-tmp/lib/cmake/ArborX/ArborXConfig.cmake]
Checking file [/home/xap/local/opt/kokkos/lib/cmake/Kokkos/KokkosConfig.cmake]
-- Enabled Kokkos devices: OPENMP;CUDA;SERIAL

We see that it does not check kokkos-check provided in CMAKE_PREFIX_PATH, or any other possible paths. Thus, providing Kokkos_DIR completely shortcuts the search.

Now, lets see what happens if we do this:

find_package(Kokkos QUIET)                                                                                                                                                                                                                                                                                                                                                                 
if(NOT Kokkos_FOUND)                                                                                                                                                                                                                                                                                                                                                                       
  set(Kokkos_DIR /home/xap/local/opt/kokkos/lib/cmake/Kokkos)                                                                                                                                                                                                                                                                                                                              
  find_dependency(Kokkos)                                                                                                                                                                                                                                                                                                                                                                  
endif()     

Note that the first line find_dependency() line was replaced find_package() as we don't want to propagate REQUIRED argument:

$ VERBOSE=1 cmake -DCMAKE_PREFIX_PATH="$HOME/local/opt/arborx-tmp;$HOME/local/opt/kokkos-check/" -DCMAKE_FIND_DEBUG_MODE=ON ..                                                                                                                                                                                                                                                                                                                                                                                  
Checking prefix [/home/xap/local/opt/arborx-tmp/]                            
Checking file [/home/xap/local/opt/arborx-tmp/ArborXConfig.cmake]                                                                                                                                  
Checking file [/home/xap/local/opt/arborx-tmp/arborx-config.cmake]                       
Checking file [/home/xap/local/opt/arborx-tmp/lib/cmake/ArborX/ArborXConfig.cmake]
Checking prefix [/home/xap/local/opt/arborx-tmp/]
Checking file [/home/xap/local/opt/arborx-tmp/KokkosConfig.cmake]
Checking file [/home/xap/local/opt/arborx-tmp/kokkos-config.cmake]
Checking prefix [/home/xap/local/opt/kokkos-check/]
Checking prefix [/home/xap/local/opt/spack/opt/spack/linux-ubuntu18.04-broadwell/gcc-7.4.0/boost-1.68.0-syfratmpslzh7vpultgho6h54mwh4g34/]
Checking file [/home/xap/local/opt/spack/opt/spack/linux-ubuntu18.04-broadwell/gcc-7.4.0/boost-1.68.0-syfratmpslzh7vpultgho6h54mwh4g34/KokkosConfig.cmake]
<snip>
Checking file [/usr/local/games/kokkos-config.cmake]                                                                                                                                                                                                                                                                                                                                       
Checking prefix [/snap/]                                                                                                                                                                                                                                                                                                                                                                   
Checking file [/snap/KokkosConfig.cmake]                                                                                                                                                                                                                                                                                                                                                   
Checking file [/snap/kokkos-config.cmake]                                                                                                                                                                                                                                                                                                                                                  
Checking prefix [/usr/X11R6/]                                                                                                                                                                                                                                                                                                                                                              
Checking prefix [/usr/pkg/]                                                                                                                                                                                                                                                                                                                                                                
Checking file [/home/xap/local/opt/kokkos/lib/cmake/Kokkos/KokkosConfig.cmake]

This time, indeed, all search paths are exhausted before fallback Kokkos_DIR is used. Also note, that instead of if(NOT DEFINED Kokkos_DIR) we used if(NOT Kokkos_FOUND), as Kokkos_DIR will always be set in cache after find_package().

[aprokop]

The new patch is safe from the behavior point of view. We exhaust all find_package(Kokkos) options before trying to search for it the second time. Thus, the ordering of options no longer plays any role. The only downside is no automatic argument propagation for the first find_package() call.