RAC/OracleConnectionManager - TNS-04011: Oracle Connection Manager instance not yet started.

[mariuspiritu]

Docker host: Linux docker-rac-buc 4.14.35-1902.7.3.1.el7uek.x86_64 #2 SMP Tue Nov 12 13:19:18 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

Docker version 17.06.2-ol, build 503b6df

racnode1 and racnode2 successfully started. i can connect to the database from within the racnode1 and racnode2 containers

Connection Manager apparently started successfully but is not connectable.

[root@docker-rac-buc abs]# docker logs -f racnode-cman
12-05-2019 05:57:15 UTC : : Creating /tmp/orod.log
12-05-2019 05:57:16 UTC : : Domain is defined to .
12-05-2019 05:57:16 UTC : : PORT is not defined. Setting PORT to '1521'
12-05-2019 05:57:16 UTC : : Public IP is set to 172.16.1.15
12-05-2019 05:57:16 UTC : : RAC Node PUBLIC Hostname is set to racnode-cman1
12-05-2019 05:57:16 UTC : : SCAN_NAME name is racnode-scan
12-05-2019 05:57:16 UTC : : SCAN_IP name is 172.16.1.70
12-05-2019 05:57:16 UTC : : Generating CMAN file
12-05-2019 05:57:16 UTC : : Copying CMAN file to /u01/app/oracle/product/12.2.0/client_1/network/admin
12-05-2019 05:57:16 UTC : : Starting CMAN

CMCTL for Linux: Version 12.2.0.1.0 - Production on 05-DEC-2019 05:57:17

Copyright (c) 1996, 2016, Oracle. All rights reserved.

Current instance CMAN_racnode-cman1.. is not yet started
Connecting to (ADDRESS=(PROTOCOL=TCP)(HOST=racnode-cman1)(PORT=1521))
TNS-04012: Unable to start Oracle Connection Manager instance.
12-05-2019 05:57:17 UTC : : Checking CMAN Status

CMCTL for Linux: Version 12.2.0.1.0 - Production on 05-DEC-2019 05:57:17

Copyright (c) 1996, 2016, Oracle. All rights reserved.

Current instance CMAN_racnode-cman1.. is not yet started
Connecting to (ADDRESS=(PROTOCOL=TCP)(HOST=racnode-cman1)(PORT=1521))
TNS-04011: Oracle Connection Manager instance not yet started.
12-05-2019 05:57:17 UTC : : cman started sucessfully
12-05-2019 05:57:17 UTC : : ################################################
12-05-2019 05:57:17 UTC : : CONNECTION MANAGER IS READY TO USE!
12-05-2019 05:57:17 UTC : : ################################################
12-05-2019 05:57:17 UTC : : cman started sucessfully

12-05-2019 05:57:16 UTC : : SCAN_IP name is 172.16.1.70
12-05-2019 05:57:16 UTC : : Generating CMAN file
12-05-2019 05:57:16 UTC : : Copying CMAN file to /u01/app/oracle/product/12.2.0/client_1/network/admin
12-05-2019 05:57:16 UTC : : Starting CMAN
12-05-2019 05:57:17 UTC : : Checking CMAN Status
12-05-2019 05:57:17 UTC : : cman started sucessfully
12-05-2019 05:57:17 UTC : : ################################################
12-05-2019 05:57:17 UTC : : CONNECTION MANAGER IS READY TO USE!
12-05-2019 05:57:17 UTC : : ################################################
12-05-2019 05:57:17 UTC : : cman started sucessfully

######################################################################
from racnode-cman container:

[oracle@racnode-cman1 ~]$ lsnrctl status

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 05-DEC-2019 06:49:00

Copyright (c) 1991, 2016, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=6171bf2e9848)(PORT=1521)))
TNS-12545: Connect failed because target host or object does not exist
TNS-12560: TNS:protocol adapter error
TNS-00515: Connect failed because target host or object does not exist
Linux Error: 2: No such file or directory
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused
[oracle@racnode-cman1 ~]$
[oracle@racnode-cman1 ~]$
[oracle@racnode-cman1 ~]$
[oracle@racnode-cman1 ~]$ cmctl

CMCTL for Linux: Version 12.2.0.1.0 - Production on 05-DEC-2019 06:50:02

Copyright (c) 1996, 2016, Oracle. All rights reserved.

Welcome to CMCTL, type "help" for information.

CMCTL> admin
Current instance CMAN_racnode-cman1.. is not yet started
Connections refer to (ADDRESS=(PROTOCOL=TCP)(HOST=racnode-cman1)(PORT=1521)).
The command completed successfully.
CMCTL:CMAN_racnode-cman1..> start
TNS-04012: Unable to start Oracle Connection Manager instance.
CMCTL:CMAN_racnode-cman1..>

#########################################################################
my cman.ora:

[oracle@racnode-cman1 ~]$ cat /u01/app/oracle/product/12.2.0/client_1/network/admin/cman.ora
######################################################################
#
# Copyright (c) 2001,2002, Oracle Corporation. All rights reserved.
#
# NAME
# cman.ora
#
# DESCRIPTION
# Sample CMAN configuration file that the user can modify for their
# own use.
#
# NOTES
# 1. Change to your fully qualified hostname
# 2. Change to the listening port number
# 3. Change and to your log and trace directories
#
# MODIFIED (MM/DD/YYYY)
# asankrut 10/05/2002 - Added Rule List Specifications
# asankrut 06/11/2002 - Modified to add new parameters; added comments.
# asankrut 12/31/2001 - Creation.
#
######################################################################

# CMAN Alias
CMAN_racnode-cman1.. =
(configuration=
# Listening address of the cman
(address=(protocol=tcp)(host=racnode-cman1..)(port=1521))
# Configuration parameters of this CMAN
(parameter_list =
# Need authentication for connection?
# Valid values: boolean values for on/off
(connection_statistics=yes)
(log_directory=/u01/app/oracle/product/12.2.0/client_1/network/log)
(log_level=USER)
(trace_directory=/u01/app/oracle/product/12.2.0/client_1/network/trace)
(trace_level=USER)
(valid_node_checking_registration=on)
(registration_invited_nodes=*)
)

# Rule list

# Rule Specification:
# src = Source of connection; '*' for 'ANY'
# dst = Destination of connection; '*' for 'ANY'
# srv = Service of connection; '*' for 'ANY'
# act = Action: 'accept', 'reject' or 'drop'

# Action List Specification:
# aut = aso_authentication_filter
# moct = outbound_connect_timeout
# mct = session_timeout
# mit = idle_timeout
# conn_stats = connect_statistics
(rule_list=
(rule=
(src=*)(dst=*)(srv=*)(act=accept)
(action_list=(aut=off)(moct=0)(mct=0)(mit=0)(conn_stats=on))
)
)
)

the listener.ora:

[oracle@racnode-cman1 ~]$ cat /u01/app/oracle/product/12.2.0/client_1/network/admin/listener.ora
# listener.ora Network Configuration File: /u01/app/oracle/product/12.2.0/client_1/network/admin/listener.ora
# Generated by Oracle configuration tools.

LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 6171bf2e9848)(PORT = 1521))
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
)
)

I have been struggling with this for a while, a little help would be much appreciated.
Thanks!

[psaini79]

@mariuspiritu

Let me understand the problem correctly, are you not able to connect to RAC DB? or CMAN process has some issue?

[psaini79]

If you are unable to connect to DB, please follow this thread #1425.

I am in the process of making the changes and will upload the code by early next week.

[mariuspiritu]

@mariuspiritu

Let me understand the problem correctly, are you not able to connect to RAC DB? or CMAN process has some issue?

CMAN is not connectable. cman and listener failed to start.

[psaini79]

@mariuspiritu

Please execute following steps on connection manager container and paste the output:

docker exec -i -t racnode-cman1 /bin/bash
cmctl
 administer CMAN_racnode-cman1.example.com
show status
show services

on Connection manager, you do not have standard listener but cman service must be up and running.

[mariuspiritu]

[root@localhost ~]# docker exec -i -t racnode-cman /bin/bash

[oracle@racnode-cman1 ~]$ cmctl

CMCTL for Linux: Version 12.2.0.1.0 - Production on 13-DEC-2019 09:24:36

Copyright (c) 1996, 2016, Oracle. All rights reserved.

Welcome to CMCTL, type "help" for information.

CMCTL> administer CMAN_racnode-cman1..
Current instance CMAN_racnode-cman1.. is not yet started
Connections refer to (ADDRESS=(PROTOCOL=TCP)(HOST=racnode-cman1)(PORT=1521)).
The command completed successfully.
CMCTL:CMAN_racnode-cman1..> show status
TNS-04011: Oracle Connection Manager instance not yet started.
CMCTL:CMAN_racnode-cman1..> show services
TNS-04011: Oracle Connection Manager instance not yet started.
CMCTL:CMAN_racnode-cman1..>

[psaini79]

@mariuspiritu

it seems cman container services are not started.
Please start cman listener manually and paste the output:

docker exec -i -t racnode-cman /bin/bash
cmctl
administer CMAN_racnode-cman1
start

[mariuspiritu]

[root@localhost ~]# docker exec -i -t racnode-cman /bin/bash
[oracle@racnode-cman1 ~]$
[oracle@racnode-cman1 ~]$ cmctl

CMCTL for Linux: Version 12.2.0.1.0 - Production on 16-DEC-2019 10:05:25

Copyright (c) 1996, 2016, Oracle. All rights reserved.

Welcome to CMCTL, type "help" for information.

CMCTL> administer CMAN_racnode-cman1..
Current instance CMAN_racnode-cman1.. is not yet started
Connections refer to (ADDRESS=(PROTOCOL=TCP)(HOST=racnode-cman1)(PORT=1521)).
The command completed successfully.
CMCTL:CMAN_racnode-cman1..> start
TNS-04012: Unable to start Oracle Connection Manager instance.
CMCTL:CMAN_racnode-cman1..>

[psaini79]

@mariuspiritu

It seems some problem in cman container as cman instance is not coming up. Please paste the output of cman logs.

[psaini79]

@mariuspiritu

No Response. I am closing this thread. If the issue still persist, please reopen the thread and share the required details.