-
Notifications
You must be signed in to change notification settings - Fork 10
Resources
fernanqv edited this page Mar 23, 2022
·
3 revisions
The configuration file resources.conf is used to describe computing resources. When you start WRF4G, resources.conf file is copied under ~/.drm4g/etc directory if it does not exist. The file can be edit directly or by executing wrf4g resource edit command.
WRF4G relies on DRM4G to manage computing resources. Further information about how to configure computing resources can be found in the DRM4G Documentation
The configuration resource file consists of sections, each led by a [section] header, followed by key = value entries. Lines beginning with # are ignored. Allowing sections are [DEFAULT] and [resource_name].
The DEFAULT section provides default values for all other resource sections.
The definition of each resource section has to begin with the line [resource_name] followed by key = value entries.
Configuration keys common to all resources:
-
enable: true or false in order to enable or disable a resource. -
communicatoror authentication type :-
local: The resource will be accessed directly. -
ssh: The resource will be accessed through ssh protocol.username: Name of the user that will be used to log on to the front-end.
-
-
frontend: Hostname or ip address of either the cluster or grid user interface you'll be connected to. The syntax is "host:port" and by default the port used is 22 -
private_key: Path to the identity file needed to log on to the front-end -
scratch: Directory used to store temporary files for jobs during their execution, by default, it is$HOME/.drm4g/jobs -
local_scratch: Job's working directory on the worker nodes, by default, it is$HOME/.wrf4g/jobs. -
lrmsor Local Resource Management System :-
pbs: TORQUE/PBS cluster. -
sge: Grid Engine cluster. -
slurm: SLURM cluster. -
slurm_res: [http://www.bsc.es/marenostrum-support-services/res RES(Red Española de Supercomputación)] resources. -
loadleveler: !LoadLeveler cluster. -
lsf: LSF cluster. -
fork: SHELL. -
cream: CREAM Compute Elements (CE).
-
-
queue: Queue available on the resource. If there are several queues, you have to use a "," as follows "queue = short,medium,long". -
max_jobs_in_queue: Max number of jobs in the queue. -
max_jobs_running: Max number of running jobs in the queue. -
parallel_env: It defines the parallel environments available for Grid Engine cluster. -
project: It specifies the project variable and is for TORQUE/PBS, Grid Engine and LSF clusters.
By default, WRF4G is going to use the local machine as fork lrms:
[localmachine]
enable = true
communicator = local
frontend = localhost
lrms = fork
max_jobs_running = 1TORQUE/PBS cluster, accessed through ssh protocol:
[meteo]
enable = true
communicator = ssh
username = user
frontend = mar.meteo.unican.es
private_key = ~/.ssh/id_rsa
lrms = pbs
queue = short, medium, long
max_jobs_running = 2, 10, 20
max_jobs_in_queue = 6, 20, 40ESR virtual organization, accessed through a grid user interface:
[esrVO]
enable = true
communicator = local
username = user
frontend = ui.meteo.unican.es
lrms = cream
vo = esr
bdii = bdii.grid.sara.nl:2170
myproxy_server = px.grid.sara.nl== Example usage ==
=== How to configure a TORQUE/PBS resource ===
In order to configure a TORQUE/PBS cluster accessed through ssh protocol, you should follow the next steps:
Configure the meteo resource. If you do not have a private_key file, you can generate one by executing [http://www.openbsd.org/openssh/manual.html ssh-keygen]. This command will generate a public key (~/.ssh/id_rsa.pub) that will be necessary later on.
[user@mycomputer~]$ wrf4g resource edit
[meteo]
enable = true
communicator = ssh
username = user
local_scratch = $TMPDIR
frontend = mar.meteo.unican.es
private_key = ~/.ssh/id_rsa
lrms = pbs
queue = short
max_jobs_running = 2
max_jobs_in_queue = 6
List and check if resource has been created successfully :
```shell
[user@mycomputer~]$ wrf4g resource list
RESOURCE STATE
meteo enabled- Configure identity's resource copying the public key (
~/.ssh/id_rsa.pub) toauthorized_keysfile on the remote frond-end, and adds the private key to the agent for the ssh authorization:
[user@mycomputer~]$ wrf4g id meteo init
--> Starting ssh-agent ...
--> Adding private key to ssh-agent ...
Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)
Lifetime set to 7 days
--> Copying public key on the remote frontend ...That's it! Now, you can summit experiments to meteo.
=== How to configure a EGI VO ===
For configuring a EGI VO such as ESR accessed through user grid interface, follow the below steps:
- Configure the
esrVOresource. If the grid user interface has definedLCG_GFAL_INFOSYSandMYPROXY_SERVERvariables, you do not have to indicatebdiiandmyproxy_serverkeys in your configuration:
[user@mycomputer~]$ wrf4g resource edit
[esrVO]
enable = true
communicator = local
username = user
frontend = ui.meteo.unican.es
lrms = cream
vo = esr
bdii = bdii.grid.sara.nl:2170
myproxy_server = px.grid.sara.nl
```1. List and check if resource has been created successfully :
```shell
[user@mycomputer~]$ wrf4g resource list
RESOURCE STATE
esrVO enabled
```1. List the CEs available on the `esr` VO:
```shell
[user@mycomputer~]$ wrf4g host list
HID ARCH JOBS(R/T) LRMS HOST
0 x86_64 0/0 cream-pbs esrVO::cream.afroditi.hellasgrid.gr
1 x86_64 0/0 cream-pbs esrVO::ce1.ipgp.fr
2 x86_64 0/0 cream-pbs esrVO::cr1.ipp.acad.bg
3 x86_64 0/0 cream-pbs esrVO::sbgce2.in2p3.fr
4 x86_64 0/0 cream-pbs esrVO::ce0.bordeaux.inra.fr
5 x86_64 0/0 cream-pbs esrVO::cce.ihep.ac.cn
6 x86_64 0/0 cream-pbs esrVO::ce02.ngcc.acad.bg
7 x86_64 0/0 cream-pbs esrVO::ce01.macc.unican.es
8 x86_64 0/0 cream-pbs esrVO::cygnus.grid.rug.nl
9 x86_64 0/0 cream-pbs esrVO::t2ce06.physics.ox.ac.uk
10 x86_64 0/0 cream-lsf esrVO::ce1.ts.infn.it
11 x86_64 0/0 cream-lsf esrVO::gridce1.pi.infn.it
12 x86_64 0/0 cream-lsf esrVO::gridce3.pi.infn.it
13 x86_64 0/0 cream-pbs esrVO::cream02.grid.uoi.gr
14 x86_64 0/0 cream-pbs esrVO::lapp-ce02.in2p3.fr
15 x86_64 0/0 cream-pbs esrVO::grid002.jet.efda.org
16 x86_64 0/0 cream-lsf esrVO::gridce4.pi.infn.it
17 x86_64 0/0 cream-lsf esrVO::gridce0.pi.infn.it
18 x86_64 0/0 cream-lsf esrVO::gridce2.pi.infn.it
19 x86_64 0/0 cream-pbs esrVO::t2ce06.physics.ox.ac.uk
20 x86_64 0/0 cream-pbs esrVO::grid0.fe.infn.it
21 x86_64 0/0 cream-pbs esrVO::ce0.m3pec.u-bordeaux1.fr
22 x86_64 0/0 cream-pbs esrVO::juk.nikhef.nl
23 x86_64 0/0 cream-pbs esrVO::gridce.ilc.cnr.it
24 x86_64 0/0 cream-lsf esrVO::cert-37.pd.infn.it
25 x86_64 0/0 cream-pbs esrVO::cream-ce-2.ba.infn.it
26 x86_64 0/0 cream-sge esrVO::cccreamceli09.in2p3.fr
27 x86_64 0/0 cream-sge esrVO::cccreamceli10.in2p3.fr
28 x86_64 0/0 cream-pbs esrVO::gazon.nikhef.nl
29 x86_64 0/0 cream-pbs esrVO::klomp.nikhef.nl
30 x86_64 0/0 cream-pbs esrVO::cream-ce-3.ba.infn.it
31 x86_64 0/0 cream-pbs esrVO::cream-ce-4.ba.infn.it
32 x86_64 0/0 cream-pbs esrVO::creamce.gina.sara.nl
33 x86_64 0/0 cream-lsf esrVO::prod-ce-01.pd.infn.it
34 x86_64 0/0 cream-pbs esrVO::creamce2.gina.sara.nl
35 x86_64 0/0 cream-pbs esrVO::creamce3.gina.sara.nl
36 x86_64 0/0 cream-slur esrVO::ce3.ui.savba.sk
37 x86_64 0/0 cream-pbs esrVO::glite-cream.scai.fraunhofer.de
38 x86_64 0/0 cream-pbs esrVO::cream-ce02.marie.hellasgrid.gr
39 x86_64 0/0 cream-pbs esrVO::cream-ce01.marie.hellasgrid.gr
40 x86_64 0/0 cream-pbs esrVO::fal-pygrid-44.lancs.ac.uk
41 x86_64 0/0 cream-pbs esrVO::hepgrid6.ph.liv.ac.uk
42 x86_64 0/0 cream-pbs esrVO::cream-ce01.ariagni.hellasgrid.gr
43 x86_64 0/0 cream-pbs esrVO::snf-189278.vm.okeanos.grnet.gr
44 x86_64 0/0 cream-pbs esrVO::snf-458754.vm.okeanos.grnet.gr
45 x86_64 0/0 cream-pbs esrVO::hepgrid5.ph.liv.ac.uk
46 x86_64 0/0 cream-pbs esrVO::cream01.kallisto.hellasgrid.gr
47 x86_64 0/0 cream-pbs esrVO::hepgrid10.ph.liv.ac.uk
48 x86_64 0/0 cream-pbs esrVO::hepgrid97.ph.liv.ac.uk
```1. Create an identity for 7 days:
```shell
[user@mycomputer~]$ wrf4g id esrVO init
--> Create a local proxy credential ...
Insert your Grid password:
Your identity: /DC=es/DC=irisgrid/O=unican/CN=user
Creating proxy ........................................................................................... Done
Proxy Verify OK
Your proxy is valid until: Thu Feb 26 21:37:19 2015
Your identity: /DC=es/DC=irisgrid/O=unican/CN=user
Creating proxy .................................................................... Done
Proxy Verify OK
A proxy valid for 168 hours (7.0 days) for user /DC=es/DC=irisgrid/O=unican/CN=user now exists on px.grid.sara.nl.
```1. Check the timeleft of your identity:
```shell
[user@mycomputer~]$ wrf4g id esrVO info
--> Grid credentials
subject : /DC=es/DC=irisgrid/O=unican/CN=user/CN=proxy/CN=proxy
issuer : /DC=es/DC=irisgrid/O=unican/CN=user/CN=proxy
identity : /DC=es/DC=irisgrid/O=unican/CN=user
type : full legacy globus proxy
strength : 2048 bits
path : /home/user/.wrf4g/security/px.grid.sara.nl
timeleft : 167:57:52 (7.0 days)That's it! Now, you can summit jobs to the esr VO. Keep in mind that you will have to renew your identity depending on the proxy-lifetime used.