Operator Does Not Apply Pod-Level Security Context to Introspector Pod

[srbhklkrn]

Summary
When deploying a WebLogic domain using the Oracle WebLogic Kubernetes Operator, the pod-level security context specified in the Domain resource is not being applied to the introspector pod. As a result, the introspector pod fails to meet the security requirements enforced by the namespace's Pod Security Standards. Notably, the container-level security context is applied correctly to the introspector pod.

Environment
Oracle WebLogic Kubernetes Image: 12.2.1.4-jdk8-ol8-240712
Oracle WebLogic Operator image: 4.2.7
Kubernetes Version: v1.30.3
Namespace Pod Security Level: restricted

Problem Description
The operator fails to apply the pod-level security context settings specified under serverPod.podSecurityContext in the Domain resource to the introspector pod. This omission causes the introspector pod creation to fail when the namespace enforces the restricted Pod Security Standards, as the required pod-level security settings are not present.

Error Message
Error creating: pods "accessdomain-introspector-8hnv2" is forbidden: violates PodSecurity "restricted:latest": seccompProfile (pod or container "accessdomain-introspector" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Domain YAML

apiVersion: "weblogic.oracle/v9"
kind: Domain
metadata:
  name: accessdomain
spec:
  domainUID: accessdomain
  ...
  serverPod:
    podSecurityContext:
      seccompProfile:
        type: RuntimeDefault
    containerSecurityContext:
      runAsUser: 1000
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

Expected Behavior

The operator should apply the specified serverPod.podSecurityContext settings to all pods it creates, including the introspector pod.
The introspector pod should be created successfully, complying with the restricted Pod Security Standards enforced in the namespace.

[rjeberhard]

Because domain.spec.serverPod.podSecurityContext didn't originally affect the introspector's pod, we couldn't change that behavior without affecting other customers. Therefore, there is a separate configuration that is specific to the introspector:

kubectl explain domain.spec.introspector.serverPod

GROUP:      weblogic.oracle
KIND:       Domain
VERSION:    v9

FIELD: serverPod <Object>


DESCRIPTION:
    Customization affecting the generation of the Introspector Job Pod.
    
FIELDS:
  env	<[]Object>
    A list of environment variables to set in the Introspector Job Pod
    container. More info:
    https://oracle.github.io/weblogic-kubernetes-operator/userguide/managing-domains/domain-resource/#jvm-memory-and-java-option-environment-variables.
    See `kubectl explain pods.spec.containers.env`.

  envFrom	<[]Object>
    List of sources to populate environment variables in the Introspector Job
    Pod container. The sources include either a config map or a secret. The
    operator will not expand the dependent variables in the 'envFrom' source.
    More details:
    https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#define-an-environment-variable-for-a-container.
    Also see:
    https://oracle.github.io/weblogic-kubernetes-operator/userguide/managing-domains/domain-resource/#jvm-memory-and-java-option-environment-variables.

  podSecurityContext	<Object>
    Pod-level security attributes. See `kubectl explain
    pods.spec.securityContext`. Beginning with operator version 4.0.5, if no
    value is specified for this field, the operator will use default content for
    the pod-level `securityContext`. More info:
    https://oracle.github.io/weblogic-kubernetes-operator/security/domain-security/pod-and-container/.

  resources	<Object>
    Memory and CPU minimum requirements and limits for the Introspector Job Pod.
    See `kubectl explain pods.spec.containers.resources`.```

[rjeberhard]

I'll look to updating the description of both fields so it's clearer what impacts the introspector.

[srbhklkrn]

@rjeberhard Thanks for getting back to me on weekends, I have updated my domain.yaml with your suggestion but I am still seeing the same error those pod level Security Context is not getting applied

# Copyright (c) 2017, 2021, Oracle and/or its affiliates.
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
#
# This is an example of how to define a Domain resource.
#
apiVersion: "weblogic.oracle/v9"
kind: Domain
metadata:
  name: accessdomain
  namespace: aim-sit2-wopns
  labels:
    weblogic.domainUID: accessdomain
spec:
  # The WebLogic Domain Home
  domainHome: /u01/oracle/user_projects/domains/accessdomain
 
  # The domain home source type
  # Set to PersistentVolume for domain-in-pv, Image for domain-in-image, or FromModel for model-in-image
  domainHomeSourceType: PersistentVolume
 
  # The WebLogic Server image that the Operator uses to start the domain
  image: "oam_cpu:12.2.1.4-jdk8-ol8-240712"
 
  # imagePullPolicy defaults to "Always" if image version is :latest
  imagePullPolicy: IfNotPresent
 
  # Identify which Secret contains the credentials for pulling an image
  imagePullSecrets:
  - name: orclcred
 
  # Identify which Secret contains the WebLogic Admin credentials (note that there is an example of
  # how to create that Secret at the end of this file)
  webLogicCredentialsSecret:
    name: accessdomain-credentials
 
  # Whether to include the server out file into the pod's stdout, default is true
  includeServerOutInPodLog: true
 
  # Whether to enable log home
  logHomeEnabled: true
 
  # Whether to write HTTP access log file to log home
  httpAccessLogInLogHome: true
 
  # The in-pod location for domain log, server logs, server out, introspector out, and Node Manager log files
  logHome: /u01/oracle/user_projects/domains/logs/accessdomain
  # An (optional) in-pod location for data storage of default and custom file stores.
  # If not specified or the value is either not set or empty (e.g. dataHome: "") then the
  # data storage directories are determined from the WebLogic domain home configuration.
  dataHome: ""
 
  # serverStartPolicy legal values are "Never, "IfNeeded", or "AdminOnly"
  # This determines which WebLogic Servers the Operator will start up when it discovers this Domain
  # - "Never" will not start any server in the domain
  # - "AdminOnly" will start up only the administration server (no managed servers will be started)
  # - "IfNeeded" will start all non-clustered servers, including the administration server and clustered servers up to the replica count
  serverStartPolicy: IfNeeded

  introspector:
    serverPod:
      podSecurityContext:
        # Pod-level security context for the introspector pod
        seccompProfile:
          type: RuntimeDefault

  serverPod:
    # an (optional) list of environment variable to be set on the servers
    env:
    - name: JAVA_OPTIONS
      value: "-Dweblogic.StdoutDebugEnabled=false"
    - name: USER_MEM_ARGS
      value: "-Djava.security.egd=file:/dev/./urandom -Xms256m -Xmx1024m "
    resources:
      requests:
        memory: "8Gi"
        cpu: "2000m"
      limits:
        memory: "16Gi"
        cpu: "4000m"
    volumes:
    - name: weblogic-domain-storage-volume
      persistentVolumeClaim:
        claimName: accessdomain-domain-pvc
    volumeMounts:
    - mountPath: /u01/oracle/user_projects/domains
      name: weblogic-domain-storage-volume

    podSecurityContext:
      fsGroup: 1000
      runAsUser: 1000
      seccompProfile:
        type: RuntimeDefault

    containerSecurityContext:
        runAsUser: 1000
        runAsNonRoot: true           
        privileged: false
        allowPrivilegeEscalation: false
        capabilities:
          drop:
          - ALL
 
  # adminServer is used to configure the desired behavior for starting the administration server.
  adminServer:
    # adminService:
    #   channels:
    # The Admin Server's NodePort
    #    - channelName: default
    #      nodePort: 30701
    # Uncomment to export the T3Channel as a service
    #    - channelName: T3Channel
    serverPod:
      # an (optional) list of environment variable to be set on the admin servers
      env:
      - name: USER_MEM_ARGS
        value: "-Djava.security.egd=file:/dev/./urandom -Xms512m -Xmx1024m "
      - name: CLASSPATH
        value: "/u01/oracle/wlserver/server/lib/weblogic.jar"

      podSecurityContext:
        seccompProfile:
          type: RuntimeDefault

      containerSecurityContext:
          runAsUser: 1000
          runAsNonRoot: true           
          privileged: false
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
 
  # References to Cluster resources that describe the lifecycle options for all
  # the Managed Server members of a WebLogic cluster, including Java
  # options, environment variables, additional Pod content, and the ability to
  # explicitly start, stop, or restart cluster members. The Cluster resource
  # must describe a cluster that already exists in the WebLogic domain
  # configuration.
  clusters:
  - name: accessdomain-oam-cluster
  - name: accessdomain-policy-cluster
 
  # The number of managed servers to start for unlisted clusters
  # replicas: 1
 
---
# This is an example of how to define a Cluster resource.
apiVersion: weblogic.oracle/v1
kind: Cluster
metadata:
  name: accessdomain-oam-cluster
  namespace: aim-sit2-wopns
spec:
  clusterName: oam_cluster
  serverService:
    precreateService: true
  replicas: 2
 
---
# This is an example of how to define a Cluster resource.
apiVersion: weblogic.oracle/v1
kind: Cluster
metadata:
  name: accessdomain-policy-cluster
  namespace: aim-sit2-wopns
spec:
  clusterName: policy_cluster
  serverService:
    precreateService: true
  replicas: 2

[rjeberhard]

I or another team member will have to investigate. I can't think of why that didn't work. You are using the latest version of the operator, correct? You listed the operator image as:

Oracle WebLogic Kubernetes Operator Image: 12.2.1.4-jdk8-ol8-240712

However, that has to be the WebLogic image.

[srbhklkrn]

Yes, I am using the latest Operator image version weblogic-kubernetes-operator:4.2.7 and WebLogic image 12.2.1.4-jdk8-ol8-240712

[nilgonsa]

We tried deploying the code with different images, but operator fails to apply the pod-level security context settings. Appreciate any help on this.

[rjeberhard]

This appears to be a real bug and we are working on a fix.

[srbhklkrn]

@rjeberhard is there any timeframe for the fix?

[rjeberhard]

Yes, I was going to give an update today. The original bug was pretty straightforward. However, we also have an expectation that running WebLogic instance pods won't restart simply because the operator is updated as this can cause an unexpected loss of service. Therefore, we have code that recognizes when the expected pods have changed and delays replacing already running pods until they are restarted for some other reason.

Since this incorrect behavior is somewhat old, it's taking time to get all of the unit-tests and behavior correct. I'd expect to have a release with a fix early next week.

[srbhklkrn]

@rjeberhard thank you for the quick response. We are looking forward to your next update.

[nilgonsa]

@rjeberhard Do you have any update on this?

[rjeberhard]

Thanks for your patience. We are not quite ready to release a bug-fix version, but I will update here as I have details.

[rjeberhard]

This is now resolved and we are in the process of releasing 4.2.8 containing this fix.

[melissalegasse]

Awesome, thank you so much for all your help.