Confusion with duplicated module names

[xdelaruelle]

Here is a partial listing of my modules catalog:

--------------------------------------------- /cm/extra/ModuleFiles/Latest ---------------------------------------------
allinea/forge/6.0.3                  intel-cluster-runtime/intel64/3.7
cuda/7.5.18                          intel-mpi/64/5.0.3/048
gcc/4.8.5(default)                   intel-tbb-oss/intel64/43_20150424oss
gcc/4.9.3                            mkl/11.2
gcc/5.1.0                            mpc/1.0.3
gcc/5.3.0                            mpfr/3.1.3
gdb/7.9                              opencv/3.0.0
gdk/352.55                           perforce/2015.2
gmp/6.1.0                            pgi/16.4(default)
intel/15.0                           pgi/nightly
intel/compiler/64/15.0/2015.5.223    slurm/14.11.11
intel/mkl/64/11.2/2015.5.223         tcl/8.6.4
intel-cluster-checker/2.2.2

-------------------------------------------------- /cm/extra/Bundles ---------------------------------------------------
PrgEnv/GCC+MVAPICH2/2016-04-19   PrgEnv/Intel+OpenMPI/2016-03-21
PrgEnv/GCC+OpenMPI/2016-04-16    PrgEnv/PGI+MVAPICH2/2016-04-19
PrgEnv/Intel+MVAPICH2/2016-04-19 PrgEnv/PGI+OpenMPI/2016-04-16

---------------- /cm/extra/PrgEnv/GCC+MVAPICH2/2016-04-19 -----------------
cuda/7.5.18       gdk/352.55        hydra/3.2         slurm/14.11.11    zlib/1.2.8
gcc/4.9.3         hwloc/1.11.2      mvapich2/2.2b-GDR szip/2.1

---------------- /cm/extra/PrgEnv/Intel+OpenMPI/2016-03-21 ----------------
cube/4.3.4     gdk/352.55     intel/15.0     netcdf-f/4.4.3 score-p/2.0.1  zlib/1.2.8
cuda/7.5.18    hdf5/1.10.0    mkl/11.2       openmpi/1.10.2 slurm/14.11.11
fftw/3.3.4     hwloc/1.11.2   netcdf-c/4.4.0 pnetcdf/1.7.0  szip/2.1

The PrgEnv bundles are defined so that you can identify what components were built with what. For example, in PrgEnv/GCC+MVAPICH2/2016-04-19 the hydra and mvapich2 are built with the cuda and gcc level from the same bundle. Likewise, in the rgEnv/Intel+OpenMPI/2016-03-21 bundle, the openmpi/hdf5/netcdf are all built with the intel (compiler) and cuda from the same bundle.

There's some duplication here, the slurm, cuda, gcc & intel modules appear inside the Latest bucket as well as one or more of the PrgEnv bundles. I don't consider this a problem because the contents of the modules are identical, so I'm not concerned where the module is loaded from.

I'm running into a problem with this, however: I can load duplicates of a module because, even though the names are identical, their paths are not. Take this sequence of commands:

module load gcc/4.9.3
module load PrgEnv/GCC+MVAPICH2/2016-04-19 
module load gcc
module list

The "module list" gives the error message

ModuleCmd_List.c(146):FATAL:996: The environment variables LOADEDMODULES and _LMFILES_ have inconsistent lengths.

The _LMFILES_ shows the distinct paths (split here, for clarity)

/cm/extra/ModuleFiles/Latest/gcc/4.9.3
/cm/extra/Bundles/PrgEnv/GCC+MVAPICH2/2016-04-19
/cm/extra/PrgEnv/GCC+MVAPICH2/2016-04-19/gcc/4.9.3

while the LOADEDMODULES doesn't show the redundancy:

gcc/4.9.3
PrgEnv/GCC+MVAPICH2/2016-04-19

For my purposes, I'd prefer that the _LMFILES_ not be extended if the LOADEDMODULES is not extended as well. This would enforce the way I treat the modules are being identical. Can you point me to what code would need to be changed?

Reported by: cponder

[xdelaruelle]

We don't run into these problems as long as we stick to the protocol

module purge
module load PrgEnv/....
module load ....

which forces the "bundled" version of the module to be the one that gets loaded, so we don't run into these kinds of collisions. I may be able to hammer this into my users, but it'd be more expedient for me to fix the problem.

Original comment by: cponder

[xdelaruelle]

It looks like the relevant sections of code are

utility.c 2105-2193
cmdPath.c 1-410

Original comment by: cponder

[xdelaruelle]

This is a bit more complicated than not editing _LMFILES_ if LOADEDMODULES already has it listed. There's a similar mis-matching going on with the module swap command when it converts the name into the preferred path but can't find it:

--> module swap pgi pgi/nightly
--> explode : $LOADEDMODULES
slurm/14.11.11
cuda/7.5.18
PrgEnv/PGI+OpenMPI/2016-04-16
nvidia-only
--> explode : $_LMFILES_
/cm/extra/ModuleFiles/Latest/slurm/14.11.11
/cm/extra/ModuleFiles/Latest/cuda/7.5.18
/cm/extra/ModuleFiles/Latest/pgi/16.4
/cm/extra/Bundles/PrgEnv/PGI+OpenMPI/2016-04-16
/cm/extra/PrgEnv/PGI+OpenMPI/2016-04-16/pgi/nightly

so it's appending the full path at the end, and skipping the delete.
I think what should be done, is for modules to treat the LOADEDMODULES + LMFILES as a dictionary mapping, and use it to look up the fully-qualified path of the module it's deleting or swapping before it consults the module search-paths.
I think this would be a lot easier if the code had been written in Python...

Original comment by: cponder

[xdelaruelle]

v4 is not affected by this issue, as the module load gcc wont load a second gcc modulefile identical in module name.

No fix are excepted for the compatibility v3.2 version.

Xavier