-
Notifications
You must be signed in to change notification settings - Fork 10
Resources
fernanqv edited this page Mar 17, 2022
·
3 revisions
The configuration file resources.conf is used to describe computing resources. When you start WRF4G, resources.conf file is copied under ~/.wrf4g/etc directory if it does not exist. The file can be edit directly or by executing wrf4g resource edit command.
The configuration resource file consists of sections, each led by a [section] header, followed by key = value entries. Lines beginning with # are ignored. Allowing sections are [DEFAULT] and [resource_name].
The DEFAULT section provides default values for all other resource sections.
The definition of each resource section has to begin with the line [resource_name] followed by key = value entries.
Configuration keys common to all resources:
-
enable: true or false in order to enable or disable a resource. -
communicatoror authentication type :-
local: The resource will be accessed directly. -
ssh: The resource will be accessed through ssh protocol.username: Name of the user that will be used to log on to the front-end.
-
-
frontend: Hostname or ip address of either the cluster or grid user interface you'll be connected to. The syntax is "host:port" and by default the port used is 22 -
private_key: Path to the identity file needed to log on to the front-end -
scratch: Directory used to store temporary files for jobs during their execution, by default, it is$HOME/.drm4g/jobs -
local_scratch: Job's working directory on the worker nodes, by default, it is$HOME/.wrf4g/jobs. -
lrmsor Local Resource Management System :-
pbs: TORQUE/PBS cluster. -
sge: Grid Engine cluster. -
slurm: SLURM cluster. -
slurm_res: [http://www.bsc.es/marenostrum-support-services/res RES(Red Española de Supercomputación)] resources. -
loadleveler: !LoadLeveler cluster. -
lsf: LSF cluster. -
fork: SHELL. -
cream: CREAM Compute Elements (CE).
-
-
queue: Queue available on the resource. If there are several queues, you have to use a "," as follows "queue = short,medium,long". -
max_jobs_in_queue: Max number of jobs in the queue. -
max_jobs_running: Max number of running jobs in the queue. -
parallel_env: It defines the parallel environments available for Grid Engine cluster. -
project: It specifies the project variable and is for TORQUE/PBS, Grid Engine and LSF clusters.
By default, WRF4G is going to use the local machine as fork lrms:
[localmachine]
enable = true
communicator = local
frontend = localhost
lrms = fork
max_jobs_running = 1TORQUE/PBS cluster, accessed through ssh protocol:
[meteo]
enable = true
communicator = ssh
username = user
frontend = mar.meteo.unican.es
private_key = ~/.ssh/id_rsa
lrms = pbs
queue = short, medium, long
max_jobs_running = 2, 10, 20
max_jobs_in_queue = 6, 20, 40ESR virtual organization, accessed through a grid user interface:
[esrVO]
enable = true
communicator = local
username = user
frontend = ui.meteo.unican.es
lrms = cream
vo = esr
bdii = bdii.grid.sara.nl:2170
myproxy_server = px.grid.sara.nl== Example usage ==
=== How to configure a TORQUE/PBS resource ===
In order to configure a TORQUE/PBS cluster accessed through ssh protocol, you should follow the next steps:
Configure the meteo resource. If you do not have a private_key file, you can generate one by executing [http://www.openbsd.org/openssh/manual.html ssh-keygen]. This command will generate a public key (~/.ssh/id_rsa.pub) that will be necessary later on.
[user@mycomputer~]$ wrf4g resource edit
[meteo]
enable = true
communicator = ssh
username = user
local_scratch = $TMPDIR
frontend = mar.meteo.unican.es
private_key = ~/.ssh/id_rsa
lrms = pbs
queue = short
max_jobs_running = 2
max_jobs_in_queue = 6
List and check if resource has been created successfully :
```shell
[user@mycomputer~]$ wrf4g resource list
RESOURCE STATE
meteo enabled- Configure identity's resource copying the public key (
~/.ssh/id_rsa.pub) toauthorized_keysfile on the remote frond-end, and adds the private key to the agent for the ssh authorization:
[user@mycomputer~]$ wrf4g id meteo init
--> Starting ssh-agent ...
--> Adding private key to ssh-agent ...
Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)
Lifetime set to 7 days
--> Copying public key on the remote frontend ...That's it! Now, you can summit experiments to meteo.
=== How to configure a EGI VO ===
For configuring a EGI VO such as ESR accessed through user grid interface, follow the below steps:
- Configure the
esrVOresource. If the grid user interface has definedLCG_GFAL_INFOSYSandMYPROXY_SERVERvariables, you do not have to indicatebdiiandmyproxy_serverkeys in your configuration:
[user@mycomputer~]$ wrf4g resource edit
[esrVO]
enable = true
communicator = local
username = user
frontend = ui.meteo.unican.es
lrms = cream
vo = esr
bdii = bdii.grid.sara.nl:2170
myproxy_server = px.grid.sara.nl
```1. List and check if resource has been created successfully :
```shell
[user@mycomputer~]$ wrf4g resource list
RESOURCE STATE
esrVO enabled
```1. List the CEs available on the `esr` VO:
```shell
[user@mycomputer~]$ wrf4g host list
HID ARCH JOBS(R/T) LRMS HOST
0 x86_64 0/0 cream-pbs esrVO::cream.afroditi.hellasgrid.gr
1 x86_64 0/0 cream-pbs esrVO::ce1.ipgp.fr
2 x86_64 0/0 cream-pbs esrVO::cr1.ipp.acad.bg
3 x86_64 0/0 cream-pbs esrVO::sbgce2.in2p3.fr
4 x86_64 0/0 cream-pbs esrVO::ce0.bordeaux.inra.fr
5 x86_64 0/0 cream-pbs esrVO::cce.ihep.ac.cn
6 x86_64 0/0 cream-pbs esrVO::ce02.ngcc.acad.bg
7 x86_64 0/0 cream-pbs esrVO::ce01.macc.unican.es
8 x86_64 0/0 cream-pbs esrVO::cygnus.grid.rug.nl
9 x86_64 0/0 cream-pbs esrVO::t2ce06.physics.ox.ac.uk
10 x86_64 0/0 cream-lsf esrVO::ce1.ts.infn.it
11 x86_64 0/0 cream-lsf esrVO::gridce1.pi.infn.it
12 x86_64 0/0 cream-lsf esrVO::gridce3.pi.infn.it
13 x86_64 0/0 cream-pbs esrVO::cream02.grid.uoi.gr
14 x86_64 0/0 cream-pbs esrVO::lapp-ce02.in2p3.fr
15 x86_64 0/0 cream-pbs esrVO::grid002.jet.efda.org
16 x86_64 0/0 cream-lsf esrVO::gridce4.pi.infn.it
17 x86_64 0/0 cream-lsf esrVO::gridce0.pi.infn.it
18 x86_64 0/0 cream-lsf esrVO::gridce2.pi.infn.it
19 x86_64 0/0 cream-pbs esrVO::t2ce06.physics.ox.ac.uk
20 x86_64 0/0 cream-pbs esrVO::grid0.fe.infn.it
21 x86_64 0/0 cream-pbs esrVO::ce0.m3pec.u-bordeaux1.fr
22 x86_64 0/0 cream-pbs esrVO::juk.nikhef.nl
23 x86_64 0/0 cream-pbs esrVO::gridce.ilc.cnr.it
24 x86_64 0/0 cream-lsf esrVO::cert-37.pd.infn.it
25 x86_64 0/0 cream-pbs esrVO::cream-ce-2.ba.infn.it
26 x86_64 0/0 cream-sge esrVO::cccreamceli09.in2p3.fr
27 x86_64 0/0 cream-sge esrVO::cccreamceli10.in2p3.fr
28 x86_64 0/0 cream-pbs esrVO::gazon.nikhef.nl
29 x86_64 0/0 cream-pbs esrVO::klomp.nikhef.nl
30 x86_64 0/0 cream-pbs esrVO::cream-ce-3.ba.infn.it
31 x86_64 0/0 cream-pbs esrVO::cream-ce-4.ba.infn.it
32 x86_64 0/0 cream-pbs esrVO::creamce.gina.sara.nl
33 x86_64 0/0 cream-lsf esrVO::prod-ce-01.pd.infn.it
34 x86_64 0/0 cream-pbs esrVO::creamce2.gina.sara.nl
35 x86_64 0/0 cream-pbs esrVO::creamce3.gina.sara.nl
36 x86_64 0/0 cream-slur esrVO::ce3.ui.savba.sk
37 x86_64 0/0 cream-pbs esrVO::glite-cream.scai.fraunhofer.de
38 x86_64 0/0 cream-pbs esrVO::cream-ce02.marie.hellasgrid.gr
39 x86_64 0/0 cream-pbs esrVO::cream-ce01.marie.hellasgrid.gr
40 x86_64 0/0 cream-pbs esrVO::fal-pygrid-44.lancs.ac.uk
41 x86_64 0/0 cream-pbs esrVO::hepgrid6.ph.liv.ac.uk
42 x86_64 0/0 cream-pbs esrVO::cream-ce01.ariagni.hellasgrid.gr
43 x86_64 0/0 cream-pbs esrVO::snf-189278.vm.okeanos.grnet.gr
44 x86_64 0/0 cream-pbs esrVO::snf-458754.vm.okeanos.grnet.gr
45 x86_64 0/0 cream-pbs esrVO::hepgrid5.ph.liv.ac.uk
46 x86_64 0/0 cream-pbs esrVO::cream01.kallisto.hellasgrid.gr
47 x86_64 0/0 cream-pbs esrVO::hepgrid10.ph.liv.ac.uk
48 x86_64 0/0 cream-pbs esrVO::hepgrid97.ph.liv.ac.uk
```1. Create an identity for 7 days:
```shell
[user@mycomputer~]$ wrf4g id esrVO init
--> Create a local proxy credential ...
Insert your Grid password:
Your identity: /DC=es/DC=irisgrid/O=unican/CN=user
Creating proxy ........................................................................................... Done
Proxy Verify OK
Your proxy is valid until: Thu Feb 26 21:37:19 2015
Your identity: /DC=es/DC=irisgrid/O=unican/CN=user
Creating proxy .................................................................... Done
Proxy Verify OK
A proxy valid for 168 hours (7.0 days) for user /DC=es/DC=irisgrid/O=unican/CN=user now exists on px.grid.sara.nl.
```1. Check the timeleft of your identity:
```shell
[user@mycomputer~]$ wrf4g id esrVO info
--> Grid credentials
subject : /DC=es/DC=irisgrid/O=unican/CN=user/CN=proxy/CN=proxy
issuer : /DC=es/DC=irisgrid/O=unican/CN=user/CN=proxy
identity : /DC=es/DC=irisgrid/O=unican/CN=user
type : full legacy globus proxy
strength : 2048 bits
path : /home/user/.wrf4g/security/px.grid.sara.nl
timeleft : 167:57:52 (7.0 days)That's it! Now, you can summit jobs to the esr VO. Keep in mind that you will have to renew your identity depending on the proxy-lifetime used.