Skip to content

Commit 01a136c

Browse files
henryiiihenryiii
committed
Adding a few useful items
1 parent ab13f27 commit 01a136c

File tree

7 files changed

+256
-2
lines changed

7 files changed

+256
-2
lines changed

‎README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
1-
# lmod_useful
2-
Useful scripts for setting up lmod
1+
# Lmod useful
2+
3+
Useful scripts for setting up Lmod. These are example files, as well as an environment capture script for Python/Plumbum.

‎docs/README.md

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
s is a guide to setting up Lmod (lua environment modules) on a CentOS system. I've used a similar procedure to set them up on a Mac, as well, so this is still a useful guide to the workings of Lmod if you use a different system; mostly paths will change. On a Mac, you'll want to install `Lmod` from the `science` tap in `brew`.
2+
3+
There are several good pages covering environment modules (TCL version), but not many that use the newer Lua syntax. This document aims to fill that roll.
4+
5+
# Installation
6+
7+
To install Lmod, you should add the epel repository and then do `yum install Lmod` to set it up. It will add an init script to `/etc/profile.d` to activate the `module` command automatically for all users. If you need to customize anything, like the paths included in the `MODULEPATH`, the scripts added here are `00-modulepath.*` and `z00_lmod.*`. You might find that removing the last entry in the modulepath init script (`&& export MODULEPATH=/etc/modulefiles:/usr/share/modulefiles`) is useful, as the `MODULEPATH` is better set by the profile script already sourced.
8+
9+
# Setting up the directories: Core
10+
11+
I'll be showing you how to set up the directories in `/usr/share/modulepath/`, since that is already prepared for us. We'll make two directories, `Core` and `Compiler`. `/usr/share/modulepath/Core` should be the only one in your module path; I perfer not to have `/usr/share/modulepath` in the `MODULEPATH`, as that will add clutter when looking at the available modules (and it is already available as `MODULEPATH_ROOT`).
12+
13+
Inside `Core`, you'll make directories for your compilers, such as `gcc`. Inside `gcc`, we'll set up the default 4.8 compiler by creating a file called `4.8.lua`. This is what it should look like:
14+
15+
```
16+
help([[
17+
This is the module file for the GCC compiler.
18+
]])
19+
20+
local version = "4.8"
21+
22+
whatis("Name: GCC compiler (system default)")
23+
whatis("Version: " .. version)
24+
whatis("Keywords: System, Compiler")
25+
whatis("URL: http://www.gnu.org/")
26+
whatis("Description: GNU compiler family")
27+
28+
family("compiler")
29+
30+
local prefix = "/usr/bin"
31+
32+
setenv("CC", pathJoin(prefix, "gcc"))
33+
setenv("CXX", pathJoin(prefix, "g++"))
34+
setenv("FC", pathJoin(prefix, "fc"))
35+
setenv("C77", pathJoin(prefix, "fc"))
36+
37+
local mroot = os.getenv("MODULEPATH_ROOT")
38+
local mdir = pathJoin(mroot, "Compiler", "gcc", version)
39+
prepend_path("MODULEPATH", mdir)
40+
```
41+
42+
This has a lot more than is needed, but provides an example of everything you need. This is in Lua syntax, so the items like the multiline string and the local statement for defining local variables might be a little unfamiliar to Python users, but otherwise it's a lot like Python.
43+
44+
The important features are:
45+
* The `family` command, which means only one module marked with this string can be loaded at a time.
46+
* The `setenv` commands, which set environment variables and forget the previous setting
47+
* Another option is `pushenv`, which remember the previous setting for unloading
48+
* Another similar command is `unsetenv`, which clears a variable on loading
49+
* The `prepend_path` command , which adds a path to the beginning of an environment variable, and removes it on unloading
50+
* There is also a matching `append_path` command
51+
* The fact that I'm changing `MODULEPATH` in a module is special to Lmod; it will also unload all modules in the added path when it is removed!
52+
53+
I've also chosen to use a prefix and a version to make this easy to change. The `joinPath` commmand is from one of the optional Lua libraries that Lmod includes automatically.
54+
55+
Feel free to add more compilers to `Core` if needed.
56+
57+
# Setting up the directories: Compiler
58+
59+
You'll notice that my compiler loaded it's matching directory in Compiler; that's where packages that are compiled with this compiler live. Matching names in multiple compiler directories will correctly swap if you swap compiliers! Inside a `/Compiler/gcc/4.8/` directory, add the modules that you want to use.
60+
61+
If you want the default module to be something other than the latest version number, make a `default.lua` symbolic link in the directory pointing to the file you want; if you want to alias a simpler version number, like 6 instead of 6.08.02, you can also make a similar symbolic link.
62+
63+
# Converting shell scripts
64+
65+
Many packages use a shell script to set up the environment. Lmod can capture those changes and write (most) of the `.lua` file for you. To run it, load the included `lmod` package and then run `sh_to_modulefile mysetupscript.sh -o mymodulefile.lua` and it will write a modulefile for you. You can use `--help` to see the options. If you like the prefix suggestion I used above, I have written my own version of this script that includes this, available as an example in a Plumbum branch [here](https://github.com/tomerfiliba/plumbum/blob/more_examples/examples/lmod_env.py). It also can save an environment to a file, then load and compare, creating the `.lua` file that describes the changes.
66+
67+
# Using modules
68+
69+
Some common commands you'll want are:
70+
* `module avail` Shows all currently loadable modules
71+
* `module load gcc` Prepares a compiler, makes the packages built with that compiler available
72+
* `module load package` Loads a package from the current compiler
73+
* `module list` Shows all the currently used modules
74+
* `module unload package` Unloads a package
75+
* `module save` Saves the current list of packages (can save to a name, or without a name is your personal default)
76+
* `module restore` Loads a (named or default) package list - note the system default is `system`
77+
* `module` Describes the available subcommands
78+
79+
> Note: a timesaver is available as the `ml` command; it is short for `module load`, but it works with any of the other module subcommands too. Without a package or subcommand it will list the in-use packages. It also supports unloading as `ml -package` too!
80+
81+
Created with Dillinger.io.

‎init/StdEnv.lua

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
load("gcc")

‎init/z00_StdEnv.csh

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
if ( ! $?__Init_Default_Modules ) then
2+
setenv __Init_Default_Modules 1
3+
if ( ! $?LMOD_SYSTEM_DEFAULT_MODULES ) then
4+
setenv LMOD_SYSTEM_DEFAULT_MODULES "StdEnv"
5+
endif
6+
module --initial_load restore
7+
else
8+
module refresh
9+
endif

‎init/z00_StdEnv.sh

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
if [ -z "$__Init_Default_Modules" ]; then
2+
export __Init_Default_Modules=1;
3+
4+
## ability to predefine elsewhere the default list
5+
LMOD_SYSTEM_DEFAULT_MODULES=${LMOD_SYSTEM_DEFAULT_MODULES:-"StdEnv"}
6+
export LMOD_SYSTEM_DEFAULT_MODULES
7+
module --initial_load restore
8+
else
9+
module refresh
10+
fi

‎modules/root.lua

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
help([[
2+
Loads the root environment
3+
]])
4+
5+
local version = "6.06.08"
6+
7+
whatis("loads the ROOT environment")
8+
9+
prefix = "/opt/root-" .. version
10+
11+
prepend_path("MANPATH", joinPath(prefix, "/man:"))
12+
prepend_path("PATH", joinPath(prefix, "/bin:"))
13+
14+
prepend_path("LD_LIBRARY_PATH", joinPath(prefix, "/lib"))
15+
prepend_path("DYLD_LIBRARY_PATH", joinPath(prefix, "/lib"))
16+
17+
prepend_path("CMAKE_PREFIX_PATH", prefix)
18+
prepend_path("PYTHONPATH", joinPath(prefix, "/lib"))
19+
20+
setenv("JUPYTER_PATH", joinPath(prefix, "/etc/notebook"))
21+
setenv("LIBPATH", joinPath(prefix, "/lib"))
22+
setenv("ROOTSYS", prefix)
23+
setenv("SHLIB_PATH", joinPath(prefix, "/lib"))
24+

‎scripts/lmod_env.py

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
#!/usr/bin/env python
2+
3+
'''
4+
This example cli program saves and loads an environment, noting the changes in a lmod (Lua Environment Modules) compatible format. It can also directly run and compare the changes made in a bash script. The ability to take a prefix is a set above lmod's built in script.
5+
6+
'''
7+
8+
from plumbum import local, cli, FG
9+
import json
10+
11+
def intersperse(lst, item):
12+
result = [item] * (len(lst) * 2 - 1)
13+
result[0::2] = lst
14+
return result
15+
16+
17+
def prefixify(value, prefix=None):
18+
'''
19+
Converts to a version that uses prefixes, if prefix given. Examples:
20+
21+
>>> prefixify('one two three ', ' ')
22+
'joinPath("one", prefix, "two", prefix, "three", prefix)'
23+
>>> prefixify(' ', ' ')
24+
'prefix'
25+
>>> prefixify('this', ' ')
26+
'"this"'
27+
'''
28+
29+
if prefix is None or prefix not in value:
30+
return '"{0}"'.format(value)
31+
if prefix == value:
32+
return 'prefix'
33+
vals = ['"{0}"'.format(v) for v in value.split(prefix)]
34+
vals = intersperse(vals, 'prefix')
35+
vals = [v for v in vals if v != '""']
36+
return 'joinPath({0})'.format(', '.join(vals))
37+
38+
def make_changes(orig, curr, prefix = None):
39+
'''
40+
>>> orig = dict(
41+
... same = 'hello',
42+
... gone = 'not here',
43+
... changed = 'anything',
44+
... prepended = 'base/string',
45+
... appended = 'base/string'
46+
... )
47+
>>> curr = dict(
48+
... same = 'hello',
49+
... added = 'wowsa',
50+
... changed = 'nothing',
51+
... prepended = 'new/base/string',
52+
... appended = 'base/string/more'
53+
... )
54+
>>> sorted(make_changes(orig, curr))
55+
['append_path("appended", "/more")', 'prepend_path("prepended", "new/")', 'pushenv("changed", "nothing")', 'setenv("added", "wowsa")', 'unsetenv("gone")']
56+
'''
57+
only_in_orig = set(orig) - set(curr)
58+
only_in_curr = set(curr) - set(orig)
59+
both = set(orig) & set(curr)
60+
61+
for name in only_in_orig:
62+
yield 'unsetenv("{0}")'.format(name)
63+
64+
for name in only_in_curr:
65+
yield 'setenv("{0}", {1})'.format(name, prefixify(curr[name], prefix))
66+
67+
for name in both:
68+
curr_val = curr[name]
69+
orig_val = orig[name]
70+
if curr_val != orig_val:
71+
if curr_val.startswith(orig_val):
72+
yield 'append_path("{0}", {1})'.format(name,
73+
prefixify(curr_val[len(orig_val):], prefix))
74+
elif curr_val.endswith(orig_val):
75+
yield 'prepend_path("{0}", {1})'.format(name,
76+
prefixify(curr_val[:-len(orig_val)], prefix))
77+
else:
78+
yield 'pushenv("{0}", {1})'.format(name, prefixify(curr_val, prefix))
79+
80+
def get_current_env(output):
81+
dic = {line.split('=',1)[0]:line.split('=',1)[1] for line in output.splitlines() if '=' in line}
82+
return dic
83+
84+
class LmodEnv(cli.Application):
85+
'Save then load an environment, noting the changes.'
86+
87+
prefix = cli.SwitchAttr(['-p', '--prefix'], help='A common prefix to factor out')
88+
89+
@cli.positional(cli.ExistingFile)
90+
def main(self, filename=None):
91+
if self.nested_command is not None:
92+
if self.prefix:
93+
self.nested_command.prefix = self.prefix
94+
return
95+
if filename is None:
96+
print("Error! please give a filename or a subcommand! See --help for details.")
97+
return 1
98+
orig = get_current_env(local['bash']('-c', 'env'))
99+
curr = get_current_env(local['bash']('-c', 'source '+filename+' && env'))
100+
if self.prefix:
101+
print('prefix = "{0}"\n'.format(self.prefix))
102+
print('\n'.join(sorted(make_changes(orig, curr, self.prefix))))
103+
104+
105+
@LmodEnv.subcommand("save")
106+
class Save(cli.Application):
107+
108+
@cli.positional(cli.NonexistentPath)
109+
def main(self, filename):
110+
with open(filename, 'w') as f:
111+
json.dump(dict(local.env), f)
112+
113+
@LmodEnv.subcommand("load")
114+
class Load(cli.Application):
115+
116+
prefix = cli.SwitchAttr(['-p', '--prefix'], help='A common prefix to factor out')
117+
118+
@cli.positional(cli.ExistingFile)
119+
def main(self, filename):
120+
with open(filename) as f:
121+
orig = json.load(f)
122+
curr = dict(local.env)
123+
if self.prefix:
124+
print('prefix = "{0}"\n'.format(self.prefix))
125+
print('\n'.join(sorted(make_changes(orig, curr, self.prefix))))
126+
127+
if __name__ == '__main__':
128+
LmodEnv()

0 commit comments

Comments
 (0)