WDT3: Error occurred in "OPSS Processing" phase execution

[zeusbaba]

We are using WDT for domain config of MW components (OAM, OHS, OSB, SOA, ...)
Currently, we are working on upgrading WDT from 1.9.20 to 3.2.5 (the latest version)

The flow we follow is simply;

• ensure rcu schemas are setup using oracle's rcu utility, each schema has a unique pwd
• domain config (createDomain) using WDT by feeding it with WDT templates

While applying this for OAM domain, I get an error at createDomain phase.
NB! this works fine when using WDT 1.9.20 and this error happens only when using the WDT 3.x.x

...
####<Oct 6, 2023 10:17:09 AM> <FINER> <DomainCreator> <__apply_base_domain_config> <> <RETURN>
####<Oct 6, 2023 10:17:09 AM> <INFO> <DomainCreator> <__extend_domain_with_select_template> <WLSDPLY-12205> <Writing base domain elhoam_domain to directory /u01/elhoam/domain/elhoam_domain>
####<Oct 6, 2023 10:17:09 AM> <FINER> <WlstHelper> <write_domain> <> <ENTRY /u01/elhoam/domain/elhoam_domain>
####<Oct 6, 2023 10:17:18 AM> <FINER> <WlstHelper> <write_domain> <> <THROW
oracle.weblogic.deploy.create.CreateException: WLSDPLY-00044: writeDomain(/u01/elhoam/domain/elhoam_domain) failed : Error writing domain:
64254: Error occurred in "OPSS Processing" phase execution
64254: Encountered error: oracle.security.opss.tools.lifecycle.LifecycleException: JDBC password for opss-audit-DBDS is missing for configuring security store to database.
64254: Check log for more detail.
null
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.python.core.PyReflectedConstructor.__call__(Unknown Source)
        at org.python.core.PyJavaInstance.__init__(Unknown Source)
        at org.python.core.PyJavaClass.__call__(PyJavaClass.java:891)
        at org.python.core.PyObject.__call__(Unknown Source)
        at wlsdeploy.exception.exception_helper$py.create_create_exception$4(/u01/binaries/elhoam/wdt/3.2.5/weblogic-deploy/lib/python/wlsdeploy/exception/exception_helper.py:102)
        at wlsdeploy.exception.exception_helper$py.call_function(/u01/binaries/elhoam/wdt/3.2.5/weblogic-deploy/lib/python/wlsdeploy/exception/exception_helper.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(PyFunction.java:193)
        at org.python.core.PyObject._callextra(Unknown Source)
        at wlsdeploy.exception.exception_helper$py.create_exception$2(/u01/binaries/elhoam/wdt/3.2.5/weblogic-deploy/lib/python/wlsdeploy/exception/exception_helper.py:74)
        at wlsdeploy.exception.exception_helper$py.call_function(/u01/binaries/elhoam/wdt/3.2.5/weblogic-deploy/lib/python/wlsdeploy/exception/exception_helper.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(PyFunction.java:193)
        at wlsdeploy.tool.util.wlst_helper$py.write_domain$38(/u01/binaries/elhoam/wdt/3.2.5/weblogic-deploy/lib/python/wlsdeploy/tool/util/wlst_helper.py:764)
        at wlsdeploy.tool.util.wlst_helper$py.call_function(/u01/binaries/elhoam/wdt/3.2.5/weblogic-deploy/lib/python/wlsdeploy/tool/util/wlst_helper.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(PyFunction.java:199)
        at org.python.core.PyMethod.__call__(Unknown Source)
        at org.python.core.PyObject.__call__(Unknown Source)
        at org.python.core.PyObject.invoke(Unknown Source)
 ...  
        at wlsdeploy.tool.create.domain_creator$py._DomainCreator__extend_domain_with_select_template$18(/u01/binaries/elhoam/wdt/3.2.5/weblogic-deploy/lib/python/wlsdeploy/tool/create/domain_creator.py:661)
        at wlsdeploy.tool.create.domain_creator$py.call_function(/u01/binaries/elhoam/wdt/3.2.5/weblogic-deploy/lib/python/wlsdeploy/tool/create/domain_creator.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
...
        at weblogic.management.scripting.WLST.main(WLST.java:186)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at weblogic.WLST.main(WLST.java:47)
Caused by: com.oracle.cie.domain.script.jython.WLSTException: Error writing domain:
64254: Error occurred in "OPSS Processing" phase execution
64254: Encountered error: oracle.security.opss.tools.lifecycle.LifecycleException: JDBC password for opss-audit-DBDS is missing for configuring security store to database.
64254: Check log for more detail.
null
        at com.oracle.cie.domain.script.jython.CommandExceptionHandler.handleException(CommandExceptionHandler.java:69)
        at com.oracle.cie.domain.script.jython.WLScriptContext.handleException(WLScriptContext.java:3085)
        at com.oracle.cie.domain.script.jython.WLScriptContext.runCmd(WLScriptContext.java:738)
        at sun.reflect.GeneratedMethodAccessor800.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)

If needed, I can also send the full createDomain.log

Please kindly advise, your feedback is much appreciated

Regards
Yilmaz

[robertpatrick]

@zeusbaba In order to support RCU schemas where every Data Source has a different password, you will need to create a sparse DataSource configuration that specifies the password for each RCU DataSource.

[jshum2479]

@zeusbaba We no longer use the shadow table to retrieve database connection/credentials information to set up the datasources. Since you are using different credentials for each datasource, you can create a sparse model and specify their passwords.

https://oracle.github.io/weblogic-deploy-tooling/userguide/database/connect-db/#default-template-data-source

[zeusbaba]

Thanks for feedback.
We already are using WDT template for datasources.
we split templates into context groups, e.g. JDBC, JMS, baseline, etc.

see example snippet from template for JDBCSystemResource

resources:
  JDBCSystemResource:
    'opss-data-source':
        Target: 'elhoam_cluster,AdminServer'
        JdbcResource:
            JDBCConnectionPoolParams:
                MaxCapacity: 25
                ShrinkFrequencySeconds: 900
                MinCapacity: 2
                StatementCacheSize: 10
            JDBCDataSourceParams:
                GlobalTransactionsProtocol: None
                JNDIName: jdbc/OpssDataSource
            JDBCDriverParams:
                URL: '....redacted...'
                PasswordEncrypted: '...redacted...'
                DriverName: oracle.jdbc.OracleDriver
                Properties:
                    user:
                        Value: ELHOAM_OPSS

we define all of those datasources in that template

'opss-data-source':  
'opss-audit-viewDS':
'opss-audit-DBDS':
oamDS:  
'mds-owsm':  
LocalSvcTblDataSource: 

[zeusbaba]

as I stated in the original post, the same setup with the same WDT templates works fine with v1.9.20 but fails with v3.2.5

[robertpatrick]

As Johnny mentioned, we changed our approach to handling RCU-related Data Sources to not use the STB table to populate that connection-related information (primarily due to RCU limitations for database connection URLs). The STB table approach was handling your specific use case in WDT 1.x.

The template approach works fine in our tests so clearly, there is something different between what you are doing and what we test. Can you provide a simplified set of steps and model files that will allow us to reproduce the issue?

[zeusbaba]

with 'template approach', do you mean that only datasources is sufficient?
don't we need to define RCUDbInfo: anymore?

btw, as you requested, i'm preparing templates to share, i need to clean out customer related data, sharing soon,

[zeusbaba]

here are the steps:

step1) create rcu schemas using oracle's rcu util

        rcu_components: [
          "IAU", "IAU_APPEND", "IAU_VIEWER",
          "OAM", "MDS", "OPSS",
          "STB", "WLS"
        ]

step2) configure OAM domain ( createDomain) using WDT templates
here is the snippet for JDBCSystemResource:
it contains each of datasources corresponding to rcu schemas

[robertpatrick]

@zeusbaba Thanks, we have managed to identify an issue so Johnny is working to try to understand what needs to be done to correct the problem. Once we identify a fix, it would be good if we could send you an installer to test to make sure that it resolves your issue prior to releasing a new version.

[jshum2479]

@zeusbaba I have created a fix for the issue, if you don't mind, can you test this out in your environment?

weblogic-deploy.zip

[zeusbaba]

thanks @robertpatrick & @jshum2479
Happy to hear that you / your team has found a solution so fast :)

sure, i'll do a quick round of testing now, i'll get back to you with results soon

[zeusbaba]

i have just run the same cycle by using the version you have sent WebLogic Deploy Tooling 3.2.6-SNAPSHOT
the error is still the same.

 1. WLSDPLY-12409: createDomain failed to create the domain: writeDomain(/u01/elhoam/domain/elhoam_domain) failed : Error writing domain:",
        "64254: Error occurred in \"OPSS Processing\" phase execution",
        "64254: Encountered error: oracle.security.opss.tools.lifecycle.LifecycleException: JDBC password for opss-audit-DBDS is missing for configuring security store to database.",

do we still need to define RCUDbInfo in WDT templates?

domainInfo:
    RCUDbInfo:
        rcu_prefix: 'ELHOAM'
        rcu_db_conn_string: 'elhdb-...'
        rcu_schema_password: '...'

[zeusbaba]

few lines from createDomain.log

####<Oct 10, 2023 5:04:06 PM> <FINER> <WlstHelper> <write_domain> <> <THROW
oracle.weblogic.deploy.create.CreateException: WLSDPLY-00044: writeDomain(/u01/elhoam/domain/elhoam_domain) failed : Error writing domain:
64254: Error occurred in "OPSS Processing" phase execution
64254: Encountered error: oracle.security.opss.tools.lifecycle.LifecycleException: JDBC password for opss-audit-DBDS is missing for configuring security store to database.
64254: Check log for more detail.
null
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.python.core.PyReflectedConstructor.__call__(Unknown Source)
        at org.python.core.PyJavaInstance.__init__(Unknown Source)
        at org.python.core.PyJavaClass.__call__(PyJavaClass.java:891)
        at org.python.core.PyObject.__call__(Unknown Source)
        at wlsdeploy.exception.exception_helper$py.create_create_exception$4(/u01/binaries/elhoam/wdt/fix_10okt23/weblogic-deploy/lib/python/wlsdeploy/exception/exception_helper.py:102)
        at wlsdeploy.exception.exception_helper$py.call_function(/u01/binaries/elhoam/wdt/fix_10okt23/weblogic-deploy/lib/python/wlsdeploy/exception/exception_helper.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(PyFunction.java:193)
        at org.python.core.PyObject._callextra(Unknown Source)
        at wlsdeploy.exception.exception_helper$py.create_exception$2(/u01/binaries/elhoam/wdt/fix_10okt23/weblogic-deploy/lib/python/wlsdeploy/exception/exception_helper.py:74)
        at wlsdeploy.exception.exception_helper$py.call_function(/u01/binaries/elhoam/wdt/fix_10okt23/weblogic-deploy/lib/python/wlsdeploy/exception/exception_helper.py)
...
Caused by: com.oracle.cie.domain.script.jython.WLSTException: Error writing domain:
64254: Error occurred in "OPSS Processing" phase execution
64254: Encountered error: oracle.security.opss.tools.lifecycle.LifecycleException: JDBC password for opss-audit-DBDS is missing for configuring security store to database.
64254: Check log for more detail.
null
        at com.oracle.cie.domain.script.jython.CommandExceptionHandler.handleException(CommandExceptionHandler.java:69)
        at com.oracle.cie.domain.script.jython.WLScriptContext.handleException(WLScriptContext.java:3085)
        at com.oracle.cie.domain.script.jython.WLScriptContext.runCmd(WLScriptContext.java:738)
        at sun.reflect.GeneratedMethodAccessor834.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.python.core.PyReflectedFunction.__call__(Unknown Source)
        at org.python.core.PyMethod.__call__(Unknown Source)
        at org.python.core.PyObject.__call__(Unknown Source)
        at org.python.core.PyInstance.invoke(Unknown Source)
        at org.python.pycode._pyx5.command$1(/tmp/WLSTOfflineIni4737292384558188252.py:19)
        at org.python.pycode._pyx5.call_function(/tmp/WLSTOfflineIni4737292384558188252.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(PyFunction.java:184)
        at org.python.pycode._pyx5.writeDomain$28(/tmp/WLSTOfflineIni4737292384558188252.py:111)
        at org.python.pycode._pyx5.call_function(/tmp/WLSTOfflineIni4737292384558188252.py)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(PyFunction.java:181)
        at wlsdeploy.tool.util.wlst_helper$py.write_domain$38(/u01/binaries/elhoam/wdt/fix_10okt23/weblogic-deploy/lib/python/wlsdeploy/tool/util/wlst_helper.py:762)
        ... 59 more