Add log rotation for NM .out/.log, enhance log rotation for introspec…

…tor .out, and document log rotation for WLS .out and .log.

docs-source/content/userguide/managing-domains/_index.md

@@ -13,6 +13,7 @@ description: "Important considerations for WebLogic domains in Kubernetes."
 * [About the Domain resource](#about-the-domain-resource)
 * [Managing life cycle operations](#managing-life-cycle-operations)
 * [Scaling clusters](#scaling-clusters)
+* [Log files](#log-files)
 
 #### Important considerations for WebLogic domains in Kubernetes
 
@@ -35,6 +36,7 @@ Be aware of the following important considerations for WebLogic domains running
 * _Log File Locations:_ The operator can automatically override WebLogic domain, server, and introspector log locations.
   This occurs if the Domain `logHomeEnabled` field is explicitly set to `true`, or if `logHomeEnabled` isn't set
   and `domainHomeSourceType` is set to `PersistentVolume`.  When overriding, the log location will be the location specified by the `logHome` setting.
+  For additional log file tuning information, see [Log files](#log-files).
 
 * _Listen Address Overrides:_  The operator will automatically override all WebLogic domain default,
   SSL, admin, or custom channel listen addresses (using situational configuration overrides).  These will become `domainUID` followed by a
@@ -132,3 +134,65 @@ The operator let's you initiate scaling of clusters in various ways:
 * [Using the operator's REST APIs]({{< relref "/userguide/managing-domains/domain-lifecycle/scaling#calling-the-operators-rest-scale-api" >}})
 * [Using WLDF policies]({{< relref "/userguide/managing-domains/domain-lifecycle/scaling#using-a-wldf-policy-rule-and-script-action-to-call-the-operators-rest-scale-api" >}})
 * [Using a Prometheus action]({{< relref "/userguide/managing-domains/domain-lifecycle/scaling#using-a-prometheus-alert-action-to-call-the-operators-rest-scale-api" >}})
+
+### Log files
+
+The operator can automatically override WebLogic domain, server, and introspector `.log` and `.out` locations.
+This occurs if the Domain `logHomeEnabled` field is explicitly set to `true`, or if `logHomeEnabled` isn't set
+and `domainHomeSourceType` is set to `PersistentVolume`.  When overriding, the log location will be the location specified by the `logHome` setting.
+
+If you want to fine tune the `.log` and `.out` rotation behavior for WebLogic domains and servers, then
+you can update the related `Log MBean` in your WebLogic configuration. Alternatively, for WebLogic
+servers, you can set corresponding system properties in `JAVA_OPTIONS`:
+
+- Here are some WLST off-line examples for creating and accessing commonly tuned Log MBeans:
+
+  ```bash
+  # domain log
+  cd('/')
+  create(dname,'Log')
+  cd('/Log/' + dname);
+
+  # configured server log for a server named 'sname'
+  cd('/Servers/' + sname)
+  create(sname, 'Log')
+  cd('/Servers/' + sname + '/Log/' + sname)
+
+  # templated (dynamic) server log for a template named 'tname'
+  cd('/ServerTemplates/' + tname)
+  create(tname,'Log')
+  cd('/ServerTemplates/' + tname + '/Log/' + tname)
+  ```
+
+- Here is WLST off-line for commonly tuned Log MBean attributes:
+
+  ```bash
+  # minimum log file size before rotation in kilobytes
+  set('FileMinSize', 1000)
+
+  # maximum number of rotated files
+  set('FileCount', 10)
+
+  # set to true to rotate file every time on startup (instead of append)
+  set('RotateLogOnStartup', 'true')
+  ```
+
+- Here are the defaults for commonly tuned Log MBean attributes:
+
+  | Log MBean Attribute | Production Mode Default | Development Mode Default |
+  | --------- | ----------------------- | ------------------------ |
+  | FileMinSize (in kilobytes) | 5000 | 500 |
+  | FileCount | 100 | 7 |
+  | RotateLogOnStartup | false | true |
+
+- For WebLogic server `.log` and `.out` files (including both dynamic and configured servers), you can alternatively
+set logging attributes using system properties that start with `weblogic.log.`
+and that end with the corresponding Log MBean attribute name. 
+
+  For example, you can include `-Dweblogic.log.FileMinSize=1000 -Dweblogic.log.FileCount=10 -Dweblogic.log.RotateLogOnStartup=true` in `domain.spec.serverPod.env.name.JAVA_OPTIONS` to set the behavior for all WebLogic servers in your domain. For information about setting `JAVA_OPTIONS`, see [Domain resource]({{< relref "/userguide/managing-domains/domain-resource/_index.md#jvm-memory-and-java-option-environment-variables" >}}).
+
+{{% notice warning %}}
+Kubernetes stores pod logs on each of its nodes, and, depending on the Kubernetes implementation, extra steps may be necessary to limit their disk space usage.
+For more information, see [Kubernetes Logging Architecture](https://kubernetes.io/docs/concepts/cluster-administration/logging/).
+{{% /notice %}}
+

operator/src/main/resources/scripts/introspectDomain.sh

@@ -38,8 +38,9 @@
 #        ORACLE_HOME = Oracle Install Home - defaults via utils.sh/exportInstallHomes
 #        MW_HOME     = MiddleWare Install Home - defaults to ${ORACLE_HOME}
 #        WL_HOME     = WebLogic Install Home - defaults to ${ORACLE_HOME}/wlserver
+#        INTROSPECTOR_LOG_FILE_MAX = Max number of log files to keep around (default 11).
 #
-#    - Transitively requires other env vars for startNodeManager.sh, wlst.sh,
+#    - Transitively requires other env vars for startNodeManager.sh, wlst.sh, modelInImage.sh,
 #      and introspectDomain.py (see these scripts to find out what else needs to be set).
 #
 
@@ -77,6 +78,7 @@ source ${SCRIPTPATH}/modelInImage.sh
 
 #
 # setup introspector log file
+#   keep max 11 total by default (delete oldest first)
 #
 
 traceDirs before LOG_HOME
@@ -87,17 +89,17 @@ if [ ! -z "${LOG_HOME}" ] && [ ! -d "${LOG_HOME}" ]; then
 fi
 
 ilog_dir="${LOG_HOME:-/tmp}"
-ilog_timestamp="$(date --utc '+%Y-%m-%d_%H.%M.%S')"
-ilog_prefix="${ilog_dir}/introspector_script"
-ilog_file="${ilog_prefix}_${ilog_timestamp}_.out"
-ilog_max=${MAX_INTROSPECTOR_LOG_FILES:-10}
+ilog_file="${ilog_dir}/introspector_script.out"
 
 if [ ! -d "${ilog_dir}" ]; then
   trace "Creating introspector log directory: '${ilog_dir}'"
   createFolder "${ilog_dir}"
 fi
 
-echo "" >> ${ilog_file}
+testLogFileRotate "${ilog_file}"
+[ $? -ne 0 ] && trace SEVERE "Error accessing '${ilog_dir}'. See previous log messages." && exit 1
+
+logFileRotate ${ilog_file} ${INTROSPECTOR_LOG_FILE_MAX:-11}
 
 #
 # main introspection function
@@ -109,14 +111,6 @@ function doIntrospect() {
 
   traceDirs after LOG_HOME
 
-  # keep only 10 total log files by default (delete oldest first)
-  for ilog_cur in \
-    $(ls -1r ${ilog_prefix}* 2>/dev/null | tail -n +$((ilog_max + 1)))
-  do
-    trace "Removing old introspector log file '${ilog_cur}'"
-    rm ${ilog_cur}
-  done
-
   # list potentially interesting env-vars and dirs before they're updated by export.*Homes
 
   traceEnv before

operator/src/main/resources/scripts/startNodeManager.sh

@@ -27,10 +27,14 @@
 #                          ${DOMAIN_UID}/${SERVER_NAME}_nodemanager.out
 #                       Default:
 #                          Use LOG_HOME.  If LOG_HOME not set, use NODEMGR_HOME.
+#   NODEMGR_LOG_FILE_MAX = max NM .log and .out files to keep around (default=11)
+#
 #   ADMIN_PORT_SECURE = "true" if the admin protocol is secure. Default is false
+#
 #   FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR = "true" if WebLogic server should fail to 
 #                       boot if situational configuration related errors are 
 #                       found. Default to "true" if unspecified.
+#
 #   NODEMGR_MEM_ARGS  = JVM mem args for starting the Node Manager instance
 #   NODEMGR_JAVA_OPTIONS  = Java options for starting the Node Manager instance
 #
@@ -172,46 +176,45 @@ fi
 # Create nodemanager.properties and nodemanager.domains files in NM home 
 #
 
-nm_props_file=${NODEMGR_HOME}/nodemanager.properties
 nm_domains_file=${NODEMGR_HOME}/nodemanager.domains
-
 cat <<EOF > ${nm_domains_file}
   ${domain_name}=${DOMAIN_HOME}
 EOF
+[ ! $? -eq 0 ] && trace SEVERE "Failed to create '${nm_domains_file}'." && exit 1
 
-  [ ! $? -eq 0 ] && trace SEVERE "Failed to create '${nm_domains_file}'." && exit 1
+nm_props_file=${NODEMGR_HOME}/nodemanager.properties
 
 cat <<EOF > ${nm_props_file}
   #Node manager properties
-  DomainsFile=${nm_domains_file}
-  LogLimit=0
-  DomainsDirRemoteSharingEnabled=true
-  PropertiesVersion=12.2.1
-  AuthenticationEnabled=false
   NodeManagerHome=${NODEMGR_HOME}
   JavaHome=${JAVA_HOME}
-  LogLevel=FINEST
+  DomainsFile=${nm_domains_file}
   DomainsFileEnabled=true
-  ListenAddress=127.0.0.1
+  DomainsDirRemoteSharingEnabled=true
   NativeVersionEnabled=true
+  PropertiesVersion=12.2.1
+  ListenAddress=127.0.0.1
   ListenPort=5556
-  LogToStderr=true
-  weblogic.StartScriptName=startWebLogic.sh
+  ListenBacklog=50
+  AuthenticationEnabled=false
   SecureListener=false
-  LogCount=1
-  QuitEnabled=false
-  LogAppend=true
+  weblogic.StartScriptEnabled=true
+  weblogic.StartScriptName=startWebLogic.sh
   weblogic.StopScriptEnabled=false
+  QuitEnabled=false
   StateCheckInterval=500
   CrashRecoveryEnabled=false
-  weblogic.StartScriptEnabled=true
-  LogFormatter=weblogic.nodemanager.server.LogFormatter
-  ListenBacklog=50
   LogFile=${nodemgr_log_file}
+  LogToStderr=true
+  LogFormatter=weblogic.nodemanager.server.LogFormatter
+  LogAppend=true
+  LogLimit=0
+  LogLevel=FINEST
+  LogCount=1
 
 EOF
 
-  [ ! $? -eq 0 ] && trace SEVERE "Failed to create '${nm_props_file}'." && exit 1
+[ ! $? -eq 0 ] && trace SEVERE "Failed to create '${nm_props_file}'." && exit 1
 
 ###############################################################################
 #
@@ -238,21 +241,11 @@ if [ ! "${SERVER_NAME}" = "introspector" ]; then
     [ ! $? -eq 0 ] && trace SEVERE "Could not remove stale file '$wl_state_file'." && exit 1
   fi
 
-
 cat <<EOF > ${wl_props_file}
 # Server startup properties
 AutoRestart=true
 RestartMax=2
-RotateLogOnStartup=false
-RotationType=bySize
-RotationTimeStart=00\\:00
-RotatedFileCount=100
-RestartDelaySeconds=0
-FileSizeKB=5000
-FileTimeSpanFactor=3600000
 RestartInterval=3600
-NumberOfFilesLimited=true
-FileTimeSpan=24
 NMHostName=${SERVICE_NAME}
 Arguments=${USER_MEM_ARGS} -Dweblogic.SituationalConfig.failBootOnError=${FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR} ${serverOutOption} ${JAVA_OPTIONS}
 
@@ -316,18 +309,16 @@ export JAVA_OPTIONS="${NODEMGR_MEM_ARGS} ${NODEMGR_JAVA_OPTIONS} -Dweblogic.Root
 ###############################################################################
 #
 #  Start the NM
-#  1) remove old NM log file, if it exists
+#  1) rotate old NM log file, and old NM out file, if they exist
 #  2) start NM in background
-#  3) wait up to 15 seconds for NM by monitoring log file
-#  4) 'exit 1' if wait more than 15 seconds
+#  3) wait up to ${NODE_MANAGER_MAX_WAIT:-60} seconds for NM by monitoring NM's .out file
+#  4) log SEVERE, log INFO with 'exit 1' if wait more than ${NODE_MANAGER_MAX_WAIT:-60} seconds
 # 
 
 trace "Start the nodemanager, node manager home is '${NODEMGR_HOME}', log file is '${nodemgr_log_file}', out file is '${nodemgr_out_file}'."
 
-rm -f ${nodemgr_log_file}
-[ ! $? -eq 0 ] && trace SEVERE "Could not remove old file '$nodemgr_log_file'." && exit 1
-rm -f ${nodemgr_out_file}
-[ ! $? -eq 0 ] && trace SEVERE "Could not remove old file '$nodemgr_out_file'." && exit 1
+logFileRotate ${nodemgr_log_file} ${NODEMGR_LOG_FILE_MAX:-11}
+logFileRotate ${nodemgr_out_file} ${NODEMGR_LOG_FILE_MAX:-11}
 
 ${stm_script} > ${nodemgr_out_file} 2>&1 &