Skip to content

Commit f19e864

Browse files
authored
Merge pull request #3 from hufnagel/Theta
adding Theta specific files
2 parents 99f34cb + 2a21c83 commit f19e864

File tree

6 files changed

+207
-0
lines changed

6 files changed

+207
-0
lines changed

site_specific/Theta/README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# ALCF Theta
2+
3+
ALCF Theta is a KNL based HPC cluster at Argonne Labs
4+
5+
As an LCF it is very restrictive compared to a "standard" grid site. Mainly that means no outbound
6+
internet connectivity from the worker nodes. One can work around this by implementing gateway
7+
services at the edge of the cluster, i.e. the HPC worker node connects to the gateway, which itself
8+
has outbound internet connectivity. At Theta this also runs into technical limitations since the
9+
connection from the worker nodes to the gateway is routed through RSIP, which has a very small
10+
limit on number of connections (order 5 to 10 per node maximum).
11+
12+
At Theta we use:
13+
* site squid proxy maineted by ALCF Theta support
14+
* local node squid proxy that connects to the site squid
15+
* cvmfsexec to mount cvmfs in user space (using site squid proxy)
16+
* stageout wrapper, allowing xrdcp from worker nodes to FNAL dCache (through site squid proxy)
17+
18+
This directory contains:
19+
* customize.sh : configuration for local node squid proxy
20+
* default.local : cvmfsexec configuration
21+
* example_wrapper.sh : example node wrapper script setting up local squid and cvmfsexec
22+
* proxychains.conf : stageout wrapper configuration
23+
* proxychains.sh : stageout wrapper
24+
25+
List of needed external software:
26+
* frontier-squid : https://twiki.cern.ch/twiki/bin/view/Frontier/InstallSquid
27+
* cvmfsexec : https://github.com/cvmfs/cvmfsexec
28+
* proxychains-ng : https://github.com/rofl0r/proxychains-ng
29+
30+
**NB This repository is public, do not add any credential, password, or private information.**

site_specific/Theta/customize.sh

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/bin/bash
2+
#
3+
# Edit customize.sh as you wish to customize squid.conf.
4+
# It will not be overwritten by upgrades.
5+
# See customhelps.awk for information on predefined edit functions.
6+
# In order to test changes to this, run this to regenerate squid.conf:
7+
# /local/scratch/uscms//frontier-cache/utils/bin/fn-local-squid.sh
8+
# and to reload the changes into a running squid use
9+
# /local/scratch/uscms//frontier-cache/utils/bin/fn-local-squid.sh reload
10+
# Avoid single quotes in the awk source or you have to protect them from bash.
11+
#
12+
13+
awk --file `dirname $0`/customhelps.awk --source '{
14+
setoption("cache_peer", "theta-proxy.tmi.alcf.anl.gov parent 3128 0 no-query")
15+
setoption("acl NET_LOCAL src", "127.0.0.1/32")
16+
setoption("cache_mem", "128 MB")
17+
setoptionparameter("cache_dir", 3, "10000")
18+
print
19+
}'

site_specific/Theta/default.local

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
CVMFS_HTTP_PROXY="http://theta-proxy.tmi.alcf.anl.gov:3128"
2+
CVMFS_CACHE_BASE=/local/scratch/uscms/cvmfs-cache
3+
CVMFS_QUOTA_LIMIT=10000
4+
CMS_LOCAL_SITE=T3_US_ANL
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/bin/bash
2+
3+
# clean possible leftovers from previous jobs
4+
/usr/bin/fusermount -u /local/scratch/uscms/cvmfsexec/dist/cvmfs/config-osg.opensciencegrid.org >& /dev/null
5+
/usr/bin/fusermount -u /local/scratch/uscms/cvmfsexec/dist/cvmfs/cms.cern.ch >& /dev/null
6+
/usr/bin/fusermount -u /local/scratch/uscms/cvmfsexec/dist/cvmfs/unpacked.cern.ch >& /dev/null
7+
/usr/bin/fusermount -u /local/scratch/uscms/cvmfsexec/dist/cvmfs/oasis.opensciencegrid.org >& /dev/null
8+
rm -rfd /local/scratch/uscms >& /dev/null
9+
10+
# local squid
11+
mkdir -p /local/scratch/uscms
12+
cd /local/scratch/uscms
13+
tar xzf /projects/HEPCloud-FNAL/frontier-cache_local_scratch.tgz
14+
/local/scratch/uscms/frontier-cache/utils/bin/fn-local-squid.sh start
15+
16+
# cvmfs
17+
mkdir -p /local/scratch/uscms/cvmfs-cache
18+
cd /local/scratch/uscms
19+
tar xzf /projects/HEPCloud-FNAL/cvmfsexec_local_scratch.tgz
20+
21+
# unpriviliged singularity from cvmfs
22+
/local/scratch/uscms/cvmfsexec/cvmfsexec cms.cern.ch unpacked.cern.ch oasis.opensciencegrid.org -- /cvmfs/oasis.opensciencegrid.org/mis/singularity/bin/singularity exec --pid --ipc --contain --bind /etc/hosts --bind /projects/HighLumin --bind /projects/HEPCloud-FNAL --bind /cvmfs --home $HOME /cvmfs/unpacked.cern.ch/registry.hub.docker.com/cmssw/cms:rhel7 hostname
23+
24+
# locally installed singularity
25+
#/local/scratch/uscms/cvmfsexec/cvmfsexec cms.cern.ch unpacked.cern.ch -- singularity exec -u --pid --ipc --contain --bind /etc/hosts --bind /projects/HighLumin --bind /projects/HEPCloud-FNAL --bind /cvmfs --home $HOME /cvmfs/unpacked.cern.ch/registry.hub.docker.com/cmssw/cms:rhel7 hostname
26+
27+
/local/scratch/uscms/frontier-cache/utils/bin/fn-local-squid.sh stop
28+
29+
# clean up
30+
rm -rfd /local/scratch/uscms >& /dev/null
Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# proxychains.conf VER 4.x
2+
#
3+
# HTTP, SOCKS4a, SOCKS5 tunneling proxifier with DNS.
4+
5+
6+
# The option below identifies how the ProxyList is treated.
7+
# only one option should be uncommented at time,
8+
# otherwise the last appearing option will be accepted
9+
#
10+
#dynamic_chain
11+
#
12+
# Dynamic - Each connection will be done via chained proxies
13+
# all proxies chained in the order as they appear in the list
14+
# at least one proxy must be online to play in chain
15+
# (dead proxies are skipped)
16+
# otherwise EINTR is returned to the app
17+
#
18+
strict_chain
19+
#
20+
# Strict - Each connection will be done via chained proxies
21+
# all proxies chained in the order as they appear in the list
22+
# all proxies must be online to play in chain
23+
# otherwise EINTR is returned to the app
24+
#
25+
#round_robin_chain
26+
#
27+
# Round Robin - Each connection will be done via chained proxies
28+
# of chain_len length
29+
# all proxies chained in the order as they appear in the list
30+
# at least one proxy must be online to play in chain
31+
# (dead proxies are skipped).
32+
# the start of the current proxy chain is the proxy after the last
33+
# proxy in the previously invoked proxy chain.
34+
# if the end of the proxy chain is reached while looking for proxies
35+
# start at the beginning again.
36+
# otherwise EINTR is returned to the app
37+
# These semantics are not guaranteed in a multithreaded environment.
38+
#
39+
#random_chain
40+
#
41+
# Random - Each connection will be done via random proxy
42+
# (or proxy chain, see chain_len) from the list.
43+
# this option is good to test your IDS :)
44+
45+
# Make sense only if random_chain or round_robin_chain
46+
#chain_len = 2
47+
48+
# Quiet mode (no output from library)
49+
#quiet_mode
50+
51+
# Proxy DNS requests - no leak for DNS data
52+
proxy_dns
53+
54+
# set the class A subnet number to use for the internal remote DNS mapping
55+
# we use the reserved 224.x.x.x range by default,
56+
# if the proxified app does a DNS request, we will return an IP from that range.
57+
# on further accesses to this ip we will send the saved DNS name to the proxy.
58+
# in case some control-freak app checks the returned ip, and denies to
59+
# connect, you can use another subnet, e.g. 10.x.x.x or 127.x.x.x.
60+
# of course you should make sure that the proxified app does not need
61+
# *real* access to this subnet.
62+
# i.e. dont use the same subnet then in the localnet section
63+
#remote_dns_subnet 127
64+
#remote_dns_subnet 10
65+
remote_dns_subnet 224
66+
67+
# Some timeouts in milliseconds
68+
tcp_read_time_out 15000
69+
tcp_connect_time_out 8000
70+
71+
### Examples for localnet exclusion
72+
## localnet ranges will *not* use a proxy to connect.
73+
## Exclude connections to 192.168.1.0/24 with port 80
74+
# localnet 192.168.1.0:80/255.255.255.0
75+
76+
## Exclude connections to 192.168.100.0/24
77+
# localnet 192.168.100.0/255.255.255.0
78+
79+
## Exclude connections to ANYwhere with port 80
80+
# localnet 0.0.0.0:80/0.0.0.0
81+
82+
## RFC5735 Loopback address range
83+
## if you enable this, you have to make sure remote_dns_subnet is not 127
84+
## you'll need to enable it if you want to use an application that
85+
## connects to localhost.
86+
# localnet 127.0.0.0/255.0.0.0
87+
88+
## RFC1918 Private Address Ranges
89+
# localnet 10.0.0.0/255.0.0.0
90+
# localnet 172.16.0.0/255.240.0.0
91+
# localnet 192.168.0.0/255.255.0.0
92+
93+
# ProxyList format
94+
# type ip port [user pass]
95+
# (values separated by 'tab' or 'blank')
96+
#
97+
# only numeric ipv4 addresses are valid
98+
#
99+
#
100+
# Examples:
101+
#
102+
# socks5 192.168.67.78 1080 lamer secret
103+
# http 192.168.89.3 8080 justu hidden
104+
# socks4 192.168.1.49 1080
105+
# http 192.168.39.93 8080
106+
#
107+
#
108+
# proxy types: http, socks4, socks5
109+
# ( auth types supported: "basic"-http "user/pass"-socks )
110+
#
111+
[ProxyList]
112+
# add proxy here ...
113+
# meanwile
114+
# defaults set to "tor"
115+
#http 127.0.0.1 3128
116+
http 10.236.1.189 3128
117+
118+

site_specific/Theta/proxychains.sh

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/bin/bash
2+
3+
export X509_USER_PROXY=$JOBSTARTDIR/myproxy.pem
4+
export X509_CERT_DIR=/cvmfs/oasis.opensciencegrid.org/mis/certificates/
5+
6+
/projects/HEPCloud-FNAL/proxychains-ng-4.14/proxychains4 -f /projects/HEPCloud-FNAL/proxychains.conf $@

0 commit comments

Comments
 (0)