Deploying with the RAC Storage Container: failed to start containers: racnode1

[smbptnk]

Failed to start racnode1 with below error.
[root@ip-xxx-xxx-xxx-xxx ~]# docker start racnode1
Error response from daemon: error while mounting volume '/var/lib/docker/volumes/racstorage/_data': error while mounting volume with options: type='nfs' device=':/oradata' o='addr=192.168.17.25,rw,bg,hard,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0': connection refused
Error: failed to start containers: racnode1

Thanks in adv.

[psaini79]

Did you create RAC-Storage container? Please share following:

docker ps -a
docker volume ls
docker logs -f racnode-storage
docker network ls

[smbptnk]

I have created RAC Storage container. As you can see below

1. docker logs -f racnode-storage
   ASM_STORAGE_SIZE env variable is not defined! Assining 25GB instead of 50GB default
   Oracle user will be the owner for /oradata
   Checking Space
   Space check passed : /oradata has avilable space 44 and ASM storage set to 25
   Creating ASM Disks /oradata/asm_disk01.img of size 5
   /oradata/asm_disk01.img file already exist! Skipping file creation
   Creating ASM Disks /oradata/asm_disk02.img of size 5
   /oradata/asm_disk02.img file already exist! Skipping file creation
   Creating ASM Disks /oradata/asm_disk03.img of size 5
   /oradata/asm_disk03.img file already exist! Skipping file creation
   Creating ASM Disks /oradata/asm_disk04.img of size 5
   /oradata/asm_disk04.img file already exist! Skipping file creation
   Creating ASM Disks /oradata/asm_disk05.img of size 5
   /oradata/asm_disk05.img file already exist! Skipping file creation
   #################################################
   Starting NFS Server Setup
   #################################################
   Setting up /etc/exports
   /oradata *(rw,sync,no_wdelay,no_root_squash)
   Starting RPC Bind
   Exporting File System
   Starting RPC NFSD
   Starting RPC Mountd
   Checking NFS server
   ####################################################
   NFS Server is up and running
   Create NFS volume for /oradata/
   ####################################################
   0

2. docker ps -a
   CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
   3c775b01ba84 oracle/database-rac:12.2.0.1 "/usr/sbin/oracleinit" 7 minutes ago Created racnode1
   47f2ea2124ea oracle/rac-storage-server:12.2.0.1 "/bin/sh -c 'exec ..." 19 minutes ago Up 19 minutes racnode-storage
   0fa6bb1b9355 store/oracle/database-enterprise:12.2.0.1 "/bin/sh -c '/bin/..." 2 days ago Exited (137) 41 hours ago catcdb
   bc6a3587e503 store/oracle/database-enterprise:12.2.0.1 "/bin/sh -c '/bin/..." 2 days ago Exited (137) 41 hours ago mxpdb

3. docker volume ls
   DRIVER VOLUME NAME
   local 04cdc8b6424b293b404dc8835c24eb20ec5ec387852071210dbeedf5f1e4a34b
   local 0c6cd1bbdd263303a78db4dc3f20985170ac413c9c9ba024ccd52353d54d7134
   local 1e0dc3827019b455e4709f48424a815e561461a5106d96452ee739ef9ab9643b
   local 228e37af4c31af2481c91eb521094937c550f6fb2443f2531b46e6386cf5b73d
   local 472520202fd81906e9851cd27249df1976c756dceeb8d6612464281edf25ba21
   local 4ef7d45140e4ae04b27697651c0f35250cbd75155c5592441fc5f36d72646826
   local 6dc993c7612bf16c23bbce9fbcc6fc4de34c51c6ae301621d652ce4ea3e72c11
   local 721c75927058c747465be1dccd6bd7437b397f7ace5d255df946792ddfb4c442
   local 7b1dbbcd19092145069d06d86c380231bbfe9e75597deb048e2f337325a03951
   local b8f57998aaf80b1775473b7f02395119fdb4c6c958434f53a66dcde462aa926d
   local c4adcbceeab80a52724ab78a958338346437df5777d16d3378f400a0597f2521
   local d43e465f94026d5e0e524036dea2e27d62367ccb30fbe85c68f09c305131d803
   local racstorage

4. docker network ls
   NETWORK ID NAME DRIVER SCOPE
   47daa93faab0 bridge bridge local
   7f11e556dc51 host host local
   2a70667460e9 none null local
   b3d495bff13b rac_priv1_nw bridge local
   5f6e430f9fc0 rac_pub1_nw bridge local

On docker start racnode1
Got Error response from daemon: error while mounting volume '/var/lib/docker/volumes/racstorage/_data': error while mounting volume with options: type='nfs' device=':/oradata' o='addr=192.168.17.25,rw,bg,hard,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0': connection refused
Error: failed to start containers: racnode1

FYI:
[root@ip-1xx-xx-xx-xxx asm_vol]# ls -l /docker_volumes/asm_vol
total 26214400
-rw-r--r--. 1 54321 54321 5368709120 Aug 2 10:37 asm_disk01.img
-rw-r--r--. 1 54321 54321 5368709120 Aug 2 10:38 asm_disk02.img
-rw-r--r--. 1 54321 54321 5368709120 Aug 2 10:38 asm_disk03.img
-rw-r--r--. 1 54321 54321 5368709120 Aug 2 10:39 asm_disk04.img
-rw-r--r--. 1 54321 54321 5368709120 Aug 2 10:40 asm_disk05.img

[psaini79]

Though setup looks correct, I have a question why ASM disks are 25GB total? I think they should be by default 50GB. Did you change the size? if yes, please create ASM disks of size 50G.

Also, can you paste outpu of following from Docker host:
showmount -e <rac_storage_server_container_ip>

Execute following on docker host:
rpm -qa | grep nfs-utils

Did you install nfs server packages on docker host?
rpm -qa

[smbptnk]

I allowed custom allocation of 25GB instead of default 50 GB. I think it should be fine for my requirement. Evidently we do not see any error due to ASM disk allocation. Per your reference nfs-utils was missing, so I installed the rpm and tried restarting racnode1 , this time it passed the earlier error , now it is different error -
$ docker start racnode1
Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:258: applying cgroup configuration for process caused "failed to write 95000 to cpu.rt_runtime_us: write /sys/fs/cgroup/cpu,cpuacct/docker/f5b353f1026e525258705d503b16e413889a70aa5113f55a8038c9602566896d/cpu.rt_runtime_us: invalid argument""
Error: failed to start containers: racnode1

my env: AMI ID
RHEL-7.5_HVM_GA-20180322-x86_64-1-Hourly2-GP2 (ami-6871xxxx)

[psaini79]

Hi,

I would request you to look into following threads:

#837
#838

I also request you to check following thread if you are using overlay FS.
#839

Param

[psaini79]

Keep me posted if you see any issue.

[smbptnk]

Thread #838 helped. I can connect to racnode1 container
docker logs -f racnode1
Failures were encountered during execution of CVU verification request "stage -pre crsinst".
Welcome to Oracle Linux Server 7.5!
Set hostname to .
Failed to parse kernel command line, ignoring: No such file or directory
Failed to parse kernel command line, ignoring: No such file or directory
Failed to parse kernel command line, ignoring: No such file or directory
/usr/lib/systemd/system-generators/systemd-fstab-generator failed with error code 1.
Cannot add dependency job for unit display-manager.service, ignoring: Unit not found.
Couldn't determine result for ConditionKernelCommandLine=|rd.modules-load for systemd-modules-load.service, assuming failed: No such file or directory
Couldn't determine result for ConditionKernelCommandLine=|modules-load for systemd-modules-load.service, assuming failed: No such file or directory
Verifying OS Kernel Parameter: semopm ...FAILED
racnode1: PRVG-1205 : OS kernel parameter "semopm" does not have expected
current value on node "racnode1" [Expected = "100" ; Current = "32";
Configured = "100"].

Verifying OS Kernel Parameter: rmem_default ...FAILED
racnode1: PRVG-2044 : Command "/sbin/sysctl -a | grep
rmem_default[[:space:]]*=" failed on node "racnode1" and produced no
output.
racnode1: PRVF-7544 : Check cannot be performed for kernel parameter
"rmem_default" on node "racnode1"
racnode1: PRVG-1205 : OS kernel parameter "rmem_default" does not have expected
current value on node "racnode1" [Expected = "262144" ; Current =
"unknown"; Configured = "262144"].

Verifying OS Kernel Parameter: rmem_max ...FAILED
racnode1: PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_max[[:space:]]*="
failed on node "racnode1" and produced no output.
racnode1: PRVF-7544 : Check cannot be performed for kernel parameter "rmem_max"
on node "racnode1"
racnode1: PRVG-1205 : OS kernel parameter "rmem_max" does not have expected
current value on node "racnode1" [Expected = "4194304" ; Current =
"unknown"; Configured = "4194304"].

Verifying OS Kernel Parameter: wmem_default ...FAILED
racnode1: PRVG-2044 : Command "/sbin/sysctl -a | grep
wmem_default[[:space:]]*=" failed on node "racnode1" and produced no
output.
racnode1: PRVF-7544 : Check cannot be performed for kernel parameter
"wmem_default" on node "racnode1"
racnode1: PRVG-1205 : OS kernel parameter "wmem_default" does not have expected
current value on node "racnode1" [Expected = "262144" ; Current =
"unknown"; Configured = "262144"].

Verifying OS Kernel Parameter: wmem_max ...FAILED
racnode1: PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_max[[:space:]]*="
failed on node "racnode1" and produced no output.
racnode1: PRVF-7544 : Check cannot be performed for kernel parameter "wmem_max"
on node "racnode1"
racnode1: PRVG-1205 : OS kernel parameter "wmem_max" does not have expected
current value on node "racnode1" [Expected = "1048576" ; Current =
"unknown"; Configured = "1048576"].

Verifying OS Kernel Parameter: aio-max-nr ...FAILED
racnode1: PRVG-1205 : OS kernel parameter "aio-max-nr" does not have expected
current value on node "racnode1" [Expected = "1048576" ; Current =
"65536"; Configured = "1048576"].

racnode1: PRVG-10048 : Name "racnode1" was not resolved to an address of the
specified type by name servers o"127.0.0.11".

racnode1: PRVG-10048 : Name "racnode1" was not resolved to an address of the
specified type by name servers o"127.0.0.11".

PRVG-1101 : SCAN name "racnode-scan" failed to resolve

cluvfy.tar.gz

[psaini79]

OK. It seems some kernel parameters are not set in you env. Did you make sure as per the README.MD, you set the following at Docker Host level:

Please setup some kernel parameters at host level

[psaini79]

OK. It seems some kernel parameters are not set in you env. Did you make sure as per the README.MD, you set the following at Docker Host level:

fs.file-max = 6815744
net.core.rmem_max = 4194304
net.core.rmem_default = 262144
net.core.wmem_max = 1048576
net.core.wmem_default = 262144
net.core.rmem_default = 262144

sysctl -a
sysctl -p

However, I am wondering you are kernel.sem cvu messages as well. If you still see issue, please setup following at host level:

oracle-database-server-12cR2-preinstall setting for fs.file-max is 6815744

fs.file-max = 6815744

oracle-database-server-12cR2-preinstall setting for kernel.sem is '250 32000 100 128'

kernel.sem = 250 32000 100 128

oracle-database-server-12cR2-preinstall setting for kernel.shmmni is 4096

kernel.shmmni = 4096

oracle-database-server-12cR2-preinstall setting for kernel.shmall is 1073741824 on x86_64

kernel.shmall = 1073741824

oracle-database-server-12cR2-preinstall setting for kernel.shmmax is 4398046511104 on x86_64

kernel.shmmax = 4398046511104

oracle-database-server-12cR2-preinstall setting for kernel.panic_on_oops is 1 per Orabug 19212317

kernel.panic_on_oops = 1

oracle-database-server-12cR2-preinstall setting for net.core.rmem_default is 262144

net.core.rmem_default = 262144

oracle-database-server-12cR2-preinstall setting for net.core.rmem_max is 4194304

net.core.rmem_max = 4194304

oracle-database-server-12cR2-preinstall setting for net.core.wmem_default is 262144

net.core.wmem_default = 262144

oracle-database-server-12cR2-preinstall setting for net.core.wmem_max is 1048576

net.core.wmem_max = 1048576

oracle-database-server-12cR2-preinstall setting for net.ipv4.conf.all.rp_filter is 2

net.ipv4.conf.all.rp_filter = 2

oracle-database-server-12cR2-preinstall setting for net.ipv4.conf.default.rp_filter is 2

net.ipv4.conf.default.rp_filter = 2

oracle-database-server-12cR2-preinstall setting for fs.aio-max-nr is 1048576

fs.aio-max-nr = 1048576

oracle-database-server-12cR2-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500

net.ipv4.ip_local_port_range = 9000 65500

sysctl -a
sysctl -p

This is strange that you are getting kernel parameters as we have modified some kernel parameters at container level. I will look into it.

[smbptnk]

double checked for kernel parameters at host, all seems to be set properly.
Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying OS Kernel Parameter: semopm ...FAILED
racnode1: PRVG-1205 : OS kernel parameter "semopm" does not have expected
current value on node "racnode1" [Expected = "100" ; Current = "32";
Configured = "100"].

Verifying OS Kernel Parameter: rmem_default ...FAILED
racnode1: PRVG-2044 : Command "/sbin/sysctl -a | grep
rmem_default[[:space:]]*=" failed on node "racnode1" and produced no
output.
racnode1: PRVF-7544 : Check cannot be performed for kernel parameter
"rmem_default" on node "racnode1"
racnode1: PRVG-1205 : OS kernel parameter "rmem_default" does not have expected
current value on node "racnode1" [Expected = "262144" ; Current =
"unknown"; Configured = "262144"].

Verifying OS Kernel Parameter: rmem_max ...FAILED
racnode1: PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_max[[:space:]]*="
failed on node "racnode1" and produced no output.
racnode1: PRVF-7544 : Check cannot be performed for kernel parameter "rmem_max"
on node "racnode1"
racnode1: PRVG-1205 : OS kernel parameter "rmem_max" does not have expected
current value on node "racnode1" [Expected = "4194304" ; Current =
"unknown"; Configured = "4194304"].

Verifying OS Kernel Parameter: wmem_default ...FAILED
racnode1: PRVG-2044 : Command "/sbin/sysctl -a | grep
wmem_default[[:space:]]*=" failed on node "racnode1" and produced no
output.
racnode1: PRVF-7544 : Check cannot be performed for kernel parameter
"wmem_default" on node "racnode1"
racnode1: PRVG-1205 : OS kernel parameter "wmem_default" does not have expected
current value on node "racnode1" [Expected = "262144" ; Current =
"unknown"; Configured = "262144"].

Verifying OS Kernel Parameter: wmem_max ...FAILED
racnode1: PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_max[[:space:]]*="
failed on node "racnode1" and produced no output.
racnode1: PRVF-7544 : Check cannot be performed for kernel parameter "wmem_max"
on node "racnode1"
racnode1: PRVG-1205 : OS kernel parameter "wmem_max" does not have expected
current value on node "racnode1" [Expected = "1048576" ; Current =
"unknown"; Configured = "1048576"].

racnode1: PRVG-10048 : Name "racnode1" was not resolved to an address of the
specified type by name servers o"127.0.0.11".

racnode1: PRVG-10048 : Name "racnode1" was not resolved to an address of the
specified type by name servers o"127.0.0.11".

Attached racnode1 : sysctl.conf.gz

[psaini79]

1. DNS/NTP error is expected because DNS is not available but it can be ignored and it is instrumented in the installation scripts in RAC docker image. For testing prupose, you can use single SCAN and can ignore DNS but not recommended and supported in your application workload or production workload.
2. Kernel parameter errors are strange to me as I never faced them.

However, I have a question:

Are you running cvu manually? or these errors are thrown by our scripts shipped with DockerFiles?

Also, I again tested it on my env and it looks like following:

=====
Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: racnode1:/usr,racnode1:/var,racnode1:/etc,racnode1:/sbin,racnode1:/tmp,racnode1:/u01/app/grid ...PASSED
Verifying User Existence: grid ...
Verifying Users With Same UID: 54332 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Group Membership: asmadmin ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Port Availability for component "Oracle Remote Method Invocation (ORMI)" ...PASSED
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...PASSED
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...PASSED
Verifying Port Availability for component "Oracle Notification Service (ONS) Enterprise Manager support" ...PASSED
Verifying Port Availability for component "Oracle Database Listener" ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED
Verifying Node Connectivity ...
Verifying Hosts File ...PASSED
Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast check ...PASSED
Verifying Device Checks for ASM ...
Verifying ASM device sharedness check ...
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Shared Storage Accessibility:/asmdisks/asm_disk04.img,/asmdisks/asm_disk01.img,/asmdisks/asm_disk06.img,/asmdisks/asm_disk05.img,/asmdisks/asm_disk02.img,/asmdisks/asm_disk03.img ...PASSED
Verifying ASM device sharedness check ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying I/O scheduler ...
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying I/O scheduler ...PASSED
Verifying Network Time Protocol (NTP) ...
Verifying '/etc/ntp.conf' ...PASSED
Verifying '/var/run/ntpd.pid' ...PASSED
Verifying '/var/run/chronyd.pid' ...PASSED
Verifying Network Time Protocol (NTP) ...FAILED
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Time zone consistency ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying resolv.conf Integrity ...
Verifying (Linux) resolv.conf Integrity ...FAILED (PRVG-10048)
Verifying resolv.conf Integrity ...FAILED (PRVG-10048)
Verifying DNS/NIS name service ...
Verifying Name Service Switch Configuration File Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Single Client Access Name (SCAN) ...WARNING (PRVG-11368)
Verifying Domain Sockets ...PASSED
Verifying /boot mount ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying loopback network interface address ...PASSED
Verifying Oracle base: /u01/app/grid ...
Verifying '/u01/app/grid' ...PASSED
Verifying Oracle base: /u01/app/grid ...PASSED
Verifying User Equivalence ...PASSED
Verifying File system mount options for path /var ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED

Pre-check for cluster services setup was unsuccessful on all the nodes.

Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Swap Size ...FAILED
racnode1: PRVF-7573 : Sufficient swap size is not available on node "racnode1"
[Required = 16GB (1.6777216E7KB) ; Found = 8GB (8388604.0KB)]

Verifying Network Time Protocol (NTP) ...FAILED
Verifying resolv.conf Integrity ...FAILED
racnode1: PRVG-10048 : Name "racnode1" was not resolved to an address of the
specified type by name servers o"162.88.18.6".
racnode1: PRVG-10048 : Name "racnode1" was not resolved to an address of the
specified type by name servers o"162.88.2.6".
racnode1: Check for integrity of file "/etc/resolv.conf" failed

Verifying (Linux) resolv.conf Integrity ...FAILED
racnode1: PRVG-10048 : Name "racnode1" was not resolved to an address of the
specified type by name servers o"162.88.18.6".
racnode1: PRVG-10048 : Name "racnode1" was not resolved to an address of the
specified type by name servers o"162.88.2.6".

Verifying Single Client Access Name (SCAN) ...WARNING
racnode1: PRVG-11368 : A SCAN is recommended to resolve to "3" or more IP
addresses, but SCAN "racnode-scan.k8local.com" resolves to only
"192.168.16.53,192.168.16.54"

[psaini79]

I would request if you can paste the steps so that I can try to reproduce internally and will look into it.

[smbptnk]

I followed the steps as per README.MD
cvu errors are thrown by the scripts shipped with DockerFiles. I'm not running them manually.
My env is on AWS AMI ID
RHEL-7.5_HVM_GA-20180322-x86_64-1-Hourly2-GP2 (ami-6871a115)
No sure if its tested on this environment.