changes for OWLS-80384 - Verify that operator deployment and WebLogic pods have good default cpu/memory resources

Author: ankedia

docs-source/content/faq/resource-settings.md

@@ -1,4 +1,9 @@
-# Considerations for Pod Resource (Memory and CPU) Requests and Limits
+---
+title: "Considerations for Pod Resource (Memory and CPU) Requests and Limits"
+date: 2020-06-30T08:55:00-05:00
+draft: true
+weight: 40
+---
 The operator creates a pod for each running WebLogic Server instance and each pod will have a container. It.s important that containers have enough resources in order for applications to run efficiently and expeditiously. 
 
 If a pod is scheduled on a node with limited resources, it.s possible for the node to run out of memory or CPU resources, and for applications to stop working properly or have degraded performance. It.s also possible for a rouge application to use all available memory and/or CPU, which makes other containers running on the same system unresponsive. The same problem can happen if an application has memory leak or bad configuration. 
@@ -8,32 +13,56 @@ A pod.s resource requests and limit parameters can be used to solve these proble
 ## Pod Quality Of Service (QoS) and Prioritization
 Pod.s Quality of Service (QoS) and priority is determined based on whether pod.s resource requests and limits are configured or not and how they.re configured.
 
-Best Effort QoS: If you don.t configure requests and limits, pod receives .best-effort. QoS and pod has the lowest priority. In cases where node runs out of non-shareable resources, kubelet.s out-of-resource eviction policy evicts/kills the pods with best-effort QoS first.
+**Best Effort QoS**: If you don.t configure requests and limits, pod receives .best-effort. QoS and pod has the **lowest priority**. In cases where node runs out of non-shareable resources, kubelet.s out-of-resource eviction policy evicts/kills the pods with best-effort QoS first.
 
-Burstable QoS: If you configure both resource requests and limits, and set the requests to be less than the limit, pod.s QoS will be .Burstable.. Similarly when you only configure the resource requests (without limits), the pod QoS is .Burstable.. When the node runs out of non-shareable resources, kubelet will kill .Burstable. Pods only when there are no more .best-effort. pods running. The Burstable pod receives medium priority.
+**Burstable QoS**: If you configure both resource requests and limits, and set the requests to be less than the limit, pod.s QoS will be .Burstable.. Similarly when you only configure the resource requests (without limits), the pod QoS is .Burstable.. When the node runs out of non-shareable resources, kubelet will kill .Burstable. Pods only when there are no more .best-effort. pods running. The Burstable pod receives **medium priority**.
 
-Guaranteed QoS:  If you set the requests and the limits to equal values, pod will have .Guranteed. QoS and pod will be considered as of the top most priority. These settings indicates that your pod will consume a fixed amount of memory and CPU. With this configuration, if a node runs out of shareable resources, Kubernetes will kill the best-effort and the burstable Pods first before terminating these Guaranteed QoS Pods. These are the highest priority Pods.
+**Guaranteed QoS**:  If you set the requests and the limits to equal values, pod will have .Guranteed. QoS and pod will be considered as of the top most priority. These settings indicates that your pod will consume a fixed amount of memory and CPU. With this configuration, if a node runs out of shareable resources, Kubernetes will kill the best-effort and the burstable Pods first before terminating these Guaranteed QoS Pods. These are the **highest priority** pods.
 
 ## Java heap size and pod memory request/limit considerations
 It.s extremely important to set correct heap size for JVM-based applications.  If available memory on node or memory allocated to container is not sufficient for specified JVM heap arguments (and additional off-heap memory), it is possible for WL process to run out of memory. In order to avoid this, you will need to make sure that configured heap sizes are not too big and that the pod is scheduled on the node with sufficient memory.
 With the latest Java version, it.s possible to rely on the default JVM heap settings which are safe but quite conservative. If you configure the memory limit for a container but don.t configure heap sizes (-Xms and -Xmx), JVM will configure max heap size to 25% (1/4th) of container memory limit by default. The minimum heap size is configured to 1.56% (1/64th) of limit value.
 
-### Default heap size and resource request values for sample WebLogic Server Pods:
-The samples configure default min and max heap size for WebLogic server java process to 256MB and 512MB respectively. This can be changed using USER_MEM_ARGS environment variable. The default min and max heap size for node-manager process is 64MB and 100MB. This can be changed by using NODEMGR_MEM_ARGS environment variable. 
+**Default heap sizes and resource request values for sample WebLogic Server Pods**:
+The WLS samples configure default min and max heap size for WebLogic server java process to 256MB and 512MB respectively. This can be changed using USER_MEM_ARGS environment variable. 
+```
+    resources:
+      env:
+      - name: "USER_MEM_ARGS"
+        value: "-Xms256m -Xmx512m -Djava.security.egd=file:/dev/./urandom"
+```        
 
-The default memory request in samples for WebLogic server pod is 768MB and default CPU request is 250m. This can be changed during domain creation in resources section.
+The default min and max heap size for node-manager process is 64MB and 100MB. This can be changed by using NODEMGR_MEM_ARGS environment variable. 
+
+The default pod memory request in WLS samples is 768MB and default CPU request is 250m. The requests values can be changed in resources section.
+```
+      requests:
+        cpu: "250m"
+        memory: "768Mi"
+```
 
 There.s no memory or CPU limit configured by default in samples and default QoS for WebLogic server pod is Burstable. If your use-case and workload requires higher QoS and priority, this can be achieved by setting memory and CPU limits. You.ll need to run tests and experiment with different memory/CPU limits to determine optimal limit values.
+```
+      limits:
+        cpu: 2
+        memory: "2048Mi"
+```
 
 ### Configure min/max heap size in percentages using "-XX:MinRAMPercentage" and "-XX:MaxRAMPercentage"
-If you specify pod memory limit, it's recommended to configure heap size as a percentage of the total RAM (memory) specified in the pod memory limit. These parameters allow you to fine-tune the heap size . the meaning of those settings is explained in this excellent answer on StackOverflow. Please note . they set the percentage, not the fixed values. Thanks to it changing container memory settings will not break anything. 
+If you specify pod memory limit, it's recommended to configure heap size as a percentage of the total RAM (memory) specified in the pod memory limit. These parameters allow you to fine-tune the heap size. Please note . they set the percentage, not the fixed values. Thanks to it changing container memory settings will not break anything. 
+```
+    resources:
+      env:
+      - name: JAVA_OPTIONS
+        value: "--XX:MinRAMPercentage=25.0 --XX:MaxRAMPercentage=50.0 -Dweblogic.StdoutDebugEnabled=false"
+```
 When configuring memory limits, it.s important to make sure that the limit is sufficiently big to accommodate the configured heap (and off-heap) requirements, but it's not too big to waste memory resource. Since pod memory will never go above the limit, if JVM's memory usage (sum of heap and native memory) goes above the limit, JVM process will be killed due to out-of-memory error and WebLogic container will be restarted due to liveness probe failure.   Additionally there's also a node-manager process that.s running in same container and it has it's own heap and off-heap requirements. You can also fine tune the node manager heap size in percentages by setting "-XX:MinRAMPercentage" and "-XX:MaxRAMPercentage" using .NODEMGR_JAVA_OPTIONS. environment variable. 
 
 ### Using "-Xms" and "-Xmx" parameters when not configuring limits 
 In some cases, it.s difficult to come up with a hard limit for the container and you might only want to configure memory requests but not configure memory limits. In such scenarios, you can use traditional approach to set min/max heap size using .-Xms. and .-Xmx..
 
 ### CPU requests and limits 
-It.s important that the containers running WebLogic applications have enough CPU resources, otherwise applications performance can suffer. You also don't want to set CPU requests and limit too high if your application don't need or use it. Since CPU is a shared resource, if the amount of CPU that you reserve is more than required by your application, the CPU cycles will go unused and be wasted. If no CPU request and limit is configured, it can end up using all CPU resources available on node. This can starve other containers from using shareable CPU cycles. 
+It.s important that the containers running WebLogic applications have enough CPU resources, otherwise applications performance can suffer. You also don't want to set CPU requests and limit too high if your application don't need or use allocated CPU resources. Since CPU is a shared resource, if the amount of CPU that you reserve is more than required by your application, the CPU cycles will go unused and be wasted. If no CPU request and limit is configured, it can end up using all CPU resources available on node. This can starve other containers from using shareable CPU cycles. 
 
 One other thing to keep in mind is that if pod CPU limit is not configured, it might lead to incorrect garbage collection (GC) strategy selection. WebLogic self-tuning work-manager uses pod CPU limit to configure the  number of threads in a default thread pool. If you don.t specify container CPU limit, the performance might be affected due to incorrect number of GC threads or wrong WebLogic server thread pool size. 
 
@@ -43,7 +72,7 @@ Just like CPU, if you put a memory request that.s larger than amount of memory o
 ## CPU Affinity and lock contention in k8s
 We observed much higher lock contention in k8s env when running some workloads in kubernetes as compared to traditional env. The lock contention seem to be caused by the lack of CPU cache affinity and/or scheduling latency when the workload moves to different CPU cores.  
 
-In traditional (non-k8s) environment, often tests are run with CPU affinity by binding WLS java process to particular CPU core(s) (using taskset command). This results in reduced lock contention and better performance. 
+In traditional (non-k8s) environment, often tests are run with CPU affinity achieved by binding WLS java process to particular CPU core(s) (using taskset command). This results in reduced lock contention and better performance. 
 
 In k8s environment. when CPU manager policy is configured to be "static" and QOS is "Guaranteed" for WLS pods, we see reduced lock contention and better performance. The default CPU manager policy is "none" (default). Please refer to controlling CPU management policies for more details.
 
@@ -52,3 +81,4 @@ In k8s environment. when CPU manager policy is configured to be "static" and QOS
 2) https://blog.softwaremill.com/docker-support-in-new-java-8-finally-fd595df0ca54
 3) https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
 4) https://www.magalix.com/blog/kubernetes-patterns-capacity-planning 
+

kubernetes/samples/scripts/common/domain-template.yaml

@@ -63,7 +63,7 @@ spec:
     - name: JAVA_OPTIONS
       value: "%JAVA_OPTIONS%"
     - name: USER_MEM_ARGS
-      value: "-Djava.security.egd=file:/dev/./urandom "
+      value: "-Djava.security.egd=file:/dev/./urandom -Xms256m -Xmx512m "
     %OPTIONAL_SERVERPOD_RESOURCES%
     %LOG_HOME_ON_PV_PREFIX%volumes:
     %LOG_HOME_ON_PV_PREFIX%- name: weblogic-domain-storage-volume

kubernetes/samples/scripts/common/jrf-domain-template.yaml

@@ -3,13 +3,12 @@
 #
 # This is an example of how to define a Domain resource.
 #
-apiVersion: "weblogic.oracle/v7"
+apiVersion: "weblogic.oracle/v8"
 kind: Domain
 metadata:
   name: %DOMAIN_UID%
   namespace: %NAMESPACE%
   labels:
-    weblogic.resourceVersion: domain-v2
     weblogic.domainUID: %DOMAIN_UID%
 spec:
   # The WebLogic Domain Home
@@ -50,11 +49,6 @@ spec:
   # data storage directories are determined from the WebLogic domain home configuration.
   dataHome: "%DATA_HOME%"
 
-  # Istio service mesh support is experimental.
-  %ISTIO_PREFIX%experimental:
-  %ISTIO_PREFIX%  istio:
-  %ISTIO_PREFIX%    enabled: %ISTIO_ENABLED%
-  %ISTIO_PREFIX%    readinessPort: %ISTIO_READINESS_PORT%
 
   # serverStartPolicy legal values are "NEVER", "IF_NEEDED", or "ADMIN_ONLY"
   # This determines which WebLogic Servers the Operator will start up when it discovers this Domain
@@ -121,3 +115,10 @@ spec:
     replicas: %INITIAL_MANAGED_SERVER_REPLICAS%
   # The number of managed servers to start for unlisted clusters
   # replicas: 1
+
+  # Istio
+  %ISTIO_PREFIX%configuration:
+  %ISTIO_PREFIX%  istio:
+  %ISTIO_PREFIX%    enabled: %ISTIO_ENABLED%
+  %ISTIO_PREFIX%    readinessPort: %ISTIO_READINESS_PORT%
+

kubernetes/samples/scripts/create-fmw-infrastructure-domain/domain-home-in-image/create-domain-inputs.yaml

@@ -142,16 +142,20 @@ domainHomeImageBase: container-registry.oracle.com/middleware/fmw-infrastructure
 # which uses WDT, instead of WLST, to generate the domain configuration.
 domainHomeImageBuildPath: ./docker-images/OracleFMWInfrastructure/samples/12213-domain-home-in-image
 
-# Uncomment and edit value(s) below to specify the maximum amount of
-# compute resources allowed, and minimum amount of compute resources
-# required for each server pod.
-# These are optional.
+# Resource request for each server pod (Memory and CPU). This is minimum amount of compute
+# resources required for each server pod. Edit value(s) below as per pod sizing requirements.
+# These are optional. 
 # Please refer to the kubernetes documentation on Managing Compute
 # Resources for Containers for details.
-#
-# serverPodMemoryRequest: "64Mi"
-# serverPodCpuRequest: "250m"
-# serverPodMemoryLimit: "1Gi"
+serverPodMemoryRequest: "1280Mi"
+serverPodCpuRequest: "500m"
+
+# Uncomment and edit value(s) below to specify the maximum amount of compute resources allowed 
+# for each server pod.
+# These are optional. 
+# Please refer to the kubernetes documentation on Managing Compute
+# Resources for Containers for details.
+# serverPodMemoryLimit: "2Gi"
 # serverPodCpuLimit: "1000m"
 
 #

kubernetes/samples/scripts/create-fmw-infrastructure-domain/domain-home-in-image/create-domain.sh

@@ -141,7 +141,7 @@ function initialize {
     validationError "The template file ${domainPropertiesInput} for creating a WebLogic domain was not found"
   fi
 
-  dcrInput="${scriptDir}/../../common/domain-template.yaml"
+  dcrInput="${scriptDir}/../../common/jrf-domain-template.yaml"
   if [ ! -f ${dcrInput} ]; then
     validationError "The template file ${dcrInput} for creating the domain resource was not found"
   fi

kubernetes/samples/scripts/create-fmw-infrastructure-domain/domain-home-on-pv/create-domain-inputs.yaml

@@ -142,16 +142,20 @@ createDomainScriptName: create-domain-job.sh
 # so that the Kubernetes pod can use the scripts and supporting files to create a domain home.
 createDomainFilesDir: wlst
 
-# Uncomment and edit value(s) below to specify the maximum amount of
-# compute resources allowed, and minimum amount of compute resources
-# required for each server pod.
-# These are optional.
+# Resource request for each server pod (Memory and CPU). This is minimum amount of compute
+# resources required for each server pod. Edit value(s) below as per pod sizing requirements.
+# These are optional. 
 # Please refer to the kubernetes documentation on Managing Compute
 # Resources for Containers for details.
-#
-# serverPodMemoryRequest: "64Mi"
-# serverPodCpuRequest: "250m"
-# serverPodMemoryLimit: "1Gi"
+serverPodMemoryRequest: "1280Mi"
+serverPodCpuRequest: "500m"
+
+# Uncomment and edit value(s) below to specify the maximum amount of compute resources allowed 
+# for each server pod.
+# These are optional. 
+# Please refer to the kubernetes documentation on Managing Compute
+# Resources for Containers for details.
+# serverPodMemoryLimit: "2Gi"
 # serverPodCpuLimit: "1000m"
 
 #

kubernetes/samples/scripts/create-fmw-infrastructure-domain/domain-home-on-pv/create-domain.sh

@@ -118,7 +118,7 @@ function initialize {
     validationError "The template file ${deleteJobInput} for deleting a WebLogic domain was not found"
   fi
 
-  dcrInput="${scriptDir}/../../common/domain-template.yaml"
+  dcrInput="${scriptDir}/../../common/jrf-domain-template.yaml"
   if [ ! -f ${dcrInput} ]; then
     validationError "The template file ${dcrInput} for creating the domain resource was not found"
   fi

kubernetes/samples/scripts/create-weblogic-domain/domain-home-in-image/create-domain-inputs.yaml

@@ -159,15 +159,19 @@ domainHomeImageBase: container-registry.oracle.com/middleware/weblogic:12.2.1.4
 # which uses WDT, instead of WLST, to generate the domain configuration.
 domainHomeImageBuildPath: ./docker-images/OracleWebLogic/samples/12213-domain-home-in-image
 
-# Uncomment and edit value(s) below to specify the maximum amount of 
-# compute resources allowed, and minimum amount of compute resources 
-# required for each server pod.
-# These are optional.                
+# Resource request for each server pod (Memory and CPU). This is minimum amount of compute
+# resources required for each server pod. Edit value(s) below as per pod sizing requirements.
+# These are optional. 
+# Please refer to the kubernetes documentation on Managing Compute
+# Resources for Containers for details.
+serverPodMemoryRequest: "768Mi"
+serverPodCpuRequest: "250m"
+
+# Uncomment and edit value(s) below to specify the maximum amount of compute resources allowed 
+# for each server pod.
+# These are optional. 
 # Please refer to the kubernetes documentation on Managing Compute
 # Resources for Containers for details.
-#
-# serverPodMemoryRequest: "64Mi"
-# serverPodCpuRequest: "250m"
 # serverPodMemoryLimit: "1Gi"
 # serverPodCpuLimit: "1000m"
 

kubernetes/samples/scripts/create-weblogic-domain/domain-home-on-pv/create-domain-inputs.yaml

@@ -145,16 +145,22 @@ createDomainScriptName: create-domain-job.sh
 # Kubernetes config map, which in turn is mounted to the `createDomainScriptsMountPath`,
 # so that the Kubernetes pod can use the scripts and supporting files to create a domain home.
 createDomainFilesDir: wlst
+ 
+# Resource request for each server pod (Memory and CPU). This is minimum amount of compute
+# resources required for each server pod. Edit value(s) below as per pod sizing requirements.
+# These are optional 
+# Please refer to the kubernetes documentation on Managing Compute
+# Resources for Containers for details.
+#
+serverPodMemoryRequest: "768Mi"
+serverPodCpuRequest: "250m"
 
-# Uncomment and edit value(s) below to specify the maximum amount of 
-# compute resources allowed, and minimum amount of compute resources
-# required for each server pod.
+# Uncomment and edit value(s) below to specify the maximum amount of compute resources allowed 
+# for each server pod.
 # These are optional. 
 # Please refer to the kubernetes documentation on Managing Compute
 # Resources for Containers for details.
 #
-# serverPodMemoryRequest: "64Mi"
-# serverPodCpuRequest: "250m"
 # serverPodMemoryLimit: "1Gi"
 # serverPodCpuLimit: "1000m"
 

kubernetes/samples/scripts/create-weblogic-domain/manually-create-domain/domain.yaml

@@ -65,7 +65,10 @@ spec:
     - name: JAVA_OPTIONS
       value: "-Dweblogic.StdoutDebugEnabled=false"
     - name: USER_MEM_ARGS
-      value: "-Xms64m -Xmx256m "
+      value: "-Xms256m -Xmx512m "
+    requests:
+      cpu: "250m"
+      memory: "768Mi"
 
     # If you are storing your domain on a persistent volume (as opposed to inside the Docker image),
     # then uncomment this section and provide the PVC details and mount path here (standard images
@@ -116,4 +119,4 @@ spec:
     replicas: 2
 
   # The number of managed servers to start for any unlisted clusters
-  # replicas: 1
+  # replicas: 1