OWLS 84741: Scaling failed on Jenkins when setting Dedicated to true & io.kubernetes.client.openapi.ApiException: Not Found (#1990)

* Use REST client's access token for authentication and authorization

* Enable testDedicatedModeSameNamespaceScale

* Add patch permissions to rolebinding

* Code cleanup

* Changes from initial code review

* Use TuningParameters to acccess property to control Operator's REST API authentiction and authorization implementation

* Code review changes

* documentation updates

* document patch verb

* documentation changes based on review

* use code font for appropriate parameters

Author: lennyphan

docs-source/content/userguide/managing-domains/domain-lifecycle/scaling.md

@@ -87,8 +87,27 @@ For example, when using `curl`:
 curl -v -k -H X-Requested-By:MyClient -H Content-Type:application/json -H Accept:application/json -H "Authorization:Bearer ..." -d '{ "managedServerCount": 3 }' https://.../scaling
 ```
 
-If you omit the header, you'll get a `400 (bad request)` response. If you omit the Bearer Authentication header, then you'll get a `401 (Unauthorized)` response.
+If you omit the header, you'll get a `400 (bad request)` response. If you omit the Bearer Authentication header, then you'll get a `401 (Unauthorized)` response.  If the service account or user associated with the `Bearer` token does not have permission to `patch` the WebLogic domain resource, then you'll get a `403 (Forbidden)` response.
 
+{{% notice note %}}
+To resolve a `403 (Forbidden)` response, when calling the operator's REST scaling API, you may need to add the `patch` request verb to the cluster role associated with the WebLogic `domains` resource.
+The example ClusterRole definition below grants `get`, `list`, `patch` and `update` access to the WebLogic `domains` resource 
+{{% /notice %}}
+
+```
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: weblogic-domain-cluster-role
+rules:
+- apiGroups: ["weblogic.oracle"]
+  resources: ["domains"]
+  verbs: ["get", "list", "patch", update"]
+- apiGroups: ["apiextensions.k8s.io"]
+  resources: ["customresourcedefinitions"]
+  verbs: ["get", "list"]
+---
+```
 ##### Operator REST endpoints
 
 The WebLogic Server Kubernetes Operator can expose both an internal and external REST HTTPS endpoint.
@@ -207,7 +226,7 @@ metadata:
 rules:
 - apiGroups: ["weblogic.oracle"]
   resources: ["domains"]
-  verbs: ["get", "list", "update"]
+  verbs: ["get", "list", "patch", update"]
 - apiGroups: ["apiextensions.k8s.io"]
   resources: ["customresourcedefinitions"]
   verbs: ["get", "list"]

docs-source/content/userguide/managing-operators/using-the-operator/the-rest-api.md

@@ -14,7 +14,12 @@ You can access most of the REST services using `GET`, for example:
 * To obtain a list of domains, send a `GET` request to the URL `/operator/latest/domains`
 * To obtain a list of clusters in a domain, send a `GET` request to the URL `/operator/latest/domains/<domainUID>/clusters`
 
-All of the REST services require authentication.  Callers must pass in a valid token header and a CA certificate file.  Callers should pass in the `Accept:/application/json` header.
+All of the REST services require authentication.  Callers must pass in a valid token header and a CA certificate file.  In previous operator versions, the operator performed authentication and authorization checks using the Kubernetes token review and subject access review APIs, and then updated the Domain resource using the operator's privileges.  Now, by default, the operator will use the caller's bearer token to perform the underlying update to the Domain resource using the caller's privileges and thus delegating authentication and authorization checks directly to the Kubernetes API Server (see [REST interface configuration]({{< relref "/userguide/managing-operators/using-the-operator/using-helm.md#rest-interface-configuration" >}})).  
+{{% notice note %}}
+When using the operator's REST services to scale up or down a WebLogic cluster, you may need to grant `patch` access to the user or service account associated with the caller's bearer token. This can be done with an RBAC ClusterRoleBinding between the user or service account and the ClusterRole that defines the permissions for the WebLogic `domains` resource.
+{{% /notice %}}
+
+Callers should pass in the `Accept:/application/json` header.
 
 To protect against Cross Site Request Forgery (CSRF) attacks, the operator REST API requires that you send in a `X-Requested-By` header when you invoke a REST endpoint that makes a change (for example, when you POST to the `/scale` endpoint).  The value is an arbitrary name such as `MyClient`. For example, when using `curl`:
 

docs-source/content/userguide/managing-operators/using-the-operator/using-helm.md

@@ -468,7 +468,22 @@ Example:
 ```
 externalOperatorKey: QmFnIEF0dHJpYnV0ZXMKICAgIGZyaWVuZGx5TmFtZTogd2VibG9naWMtb3B ...
 ```
-
+##### `tokenReviewAuthentication`
+If set to `true`, `tokenReviewAuthentication` specifies whether the the operator's REST API should use:
+   * Kubernetes token review API for authenticating users and
+   * Kubernetes subject access review API for authorizing a user's operation (`get`, `list`,
+      `patch`, and such) on a resource.
+   * Update the Domain resource using the operator's privileges.
+ 
+ If set to `false`, the operator's REST API will use the caller's bearer token for any update
+ to the Domain resource so that it is done using the caller's privileges.
+ 
+ Defaults to `false`.
+ 
+ Example:
+ ```
+ tokenReviewAuthentication: true
+ ```
 #### Debugging options
 
 ##### `remoteDebugNodePortEnabled`

integration-tests/src/test/java/oracle/weblogic/kubernetes/ItDedicatedMode.java

@@ -201,7 +201,6 @@ public void testDedicatedModeSameNamespace() {
    * scaling up cluster-1 in domain1Namespace succeeds.
    */
   @Test
-  @Disabled("Disable the test due to JIRA OWLS-84741")
   @Order(3)
   @DisplayName("Scale up cluster-1 in domain1Namespace and verify it succeeds")
   public void testDedicatedModeSameNamespaceScale() {

integration-tests/src/test/java/oracle/weblogic/kubernetes/actions/impl/Domain.java

@@ -566,6 +566,7 @@ private static boolean createRbacApiObjectsForWLDFScript(String domainNamespace,
               .addResourcesItem("domains")
               .addVerbsItem("get")
               .addVerbsItem("list")
+              .addVerbsItem("patch")
               .addVerbsItem("update"))
           .addRulesItem(new V1PolicyRule()
               .addApiGroupsItem("apiextensions.k8s.io")

kubernetes/charts/weblogic-operator/templates/_operator-cm.tpl

@@ -36,6 +36,9 @@ data:
   {{- if .clusterSizePaddingValidationEnabled }}
   clusterSizePaddingValidationEnabled: {{ .clusterSizePaddingValidationEnabled | quote }}
   {{- end }}
+  {{- if .tokenReviewAuthentication }}
+  tokenReviewAuthentication: {{ .tokenReviewAuthentication | quote }}
+  {{- end }}
 kind: "ConfigMap"
 metadata:
   labels:

kubernetes/charts/weblogic-operator/values.yaml

@@ -191,3 +191,13 @@ externalServiceNameSuffix: "-ext"
 # cluster size.
 # The default value is true.
 clusterSizePaddingValidationEnabled: true
+
+# tokenReviewAuthentication, if set to true, specifies whether the the operator's REST API should use
+#   1. Kubernetes token review API for authenticating users, and
+#   2. Kubernetes subject access review API for authorizing a user's operation (get, list,
+#      patch, etc) on a resource.
+#   3. Update the Domain resource using the operator's privileges.
+# This parameter, if set to false, will use the caller's bearer token for any update
+# to the Domain resource so that it is done using the caller's privileges.
+# The default value is false.
+#tokenReviewAuthentication: false

operator/src/main/java/oracle/kubernetes/operator/helpers/CallBuilder.java

@@ -3,10 +3,12 @@
 
 package oracle.kubernetes.operator.helpers;
 
+import java.io.IOException;
 import java.util.Optional;
 import java.util.function.Consumer;
 import javax.annotation.Nonnull;
 
+import com.google.common.base.Strings;
 import io.kubernetes.client.custom.V1Patch;
 import io.kubernetes.client.openapi.ApiCallback;
 import io.kubernetes.client.openapi.ApiClient;
@@ -42,6 +44,8 @@
 import io.kubernetes.client.openapi.models.V1TokenReview;
 import io.kubernetes.client.openapi.models.V1beta1CustomResourceDefinition;
 import io.kubernetes.client.openapi.models.VersionInfo;
+import io.kubernetes.client.util.ClientBuilder;
+import io.kubernetes.client.util.credentials.AccessTokenAuthentication;
 import okhttp3.Call;
 import oracle.kubernetes.operator.TuningParameters;
 import oracle.kubernetes.operator.TuningParameters.CallBuilderTuning;
@@ -87,7 +91,7 @@ public <T> T execute(
   private static SynchronousCallDispatcher DISPATCHER = DEFAULT_DISPATCHER;
   private static final AsyncRequestStepFactory DEFAULT_STEP_FACTORY = AsyncRequestStep::new;
   private static AsyncRequestStepFactory STEP_FACTORY = DEFAULT_STEP_FACTORY;
-  private final ClientPool helper;
+  private ClientPool helper;
   private final Boolean allowWatchBookmarks = false;
   private final String dryRun = null;
   private final String pretty = "false";
@@ -469,6 +473,10 @@ private CallBuilder(CallBuilderTuning tuning, ClientPool helper) {
     this.helper = helper;
   }
 
+  public CallBuilder(ClientPool pool) {
+    this(getCallBuilderTuning(), pool);
+  }
+
   private static CallBuilderTuning getCallBuilderTuning() {
     return Optional.ofNullable(TuningParameters.getInstance())
         .map(TuningParameters::getCallBuilderTuning)
@@ -1939,4 +1947,31 @@ private <T> Step createRequestAsync(
   private CancellableCall wrap(Call call) {
     return new CallWrapper(call);
   }
+
+  public ClientPool getClientPool() {
+    return this.helper;
+  }
+
+  /**
+   * Create AccessTokenAuthentication component for authenticating user represented by
+   * the given token.
+   * @param accessToken - User's Bearer token
+   * @return - this CallBuilder instance
+   */
+  public CallBuilder withAuthentication(String accessToken) {
+    if (!Strings.isNullOrEmpty(accessToken)) {
+      this.helper = new ClientPool().withApiClient(createApiClient(accessToken));
+    }
+    return this;
+  }
+
+  private ApiClient createApiClient(String accessToken) {
+    try {
+      ClientBuilder builder = ClientBuilder.standard();
+      return builder.setAuthentication(
+          new AccessTokenAuthentication(accessToken)).build();
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
+  }
 }

operator/src/main/java/oracle/kubernetes/operator/helpers/ClientPool.java

@@ -95,6 +95,11 @@ private ApiClient getApiClient() {
     return client;
   }
 
+  public ClientPool withApiClient(ApiClient apiClient) {
+    instance.getAndSet(apiClient);
+    return this;
+  }
+
   private static class DefaultClientFactory implements ClientFactory {
     private final AtomicBoolean first = new AtomicBoolean(true);
 

operator/src/main/java/oracle/kubernetes/operator/rest/RestBackendImpl.java

@@ -26,6 +26,7 @@
 import io.kubernetes.client.openapi.models.V1TokenReviewStatus;
 import io.kubernetes.client.openapi.models.V1UserInfo;
 import oracle.kubernetes.operator.Main;
+import oracle.kubernetes.operator.TuningParameters;
 import oracle.kubernetes.operator.helpers.AuthenticationProxy;
 import oracle.kubernetes.operator.helpers.AuthorizationProxy;
 import oracle.kubernetes.operator.helpers.AuthorizationProxy.Operation;
@@ -55,7 +56,7 @@ public class RestBackendImpl implements RestBackend {
   private static final LoggingFacade LOGGER = LoggingFactory.getLogger("Operator", "Operator");
   private static final String NEW_CLUSTER_REPLICAS =
       "{'clusterName':'%s','replicas':%d}".replaceAll("'", "\"");
-  public static final String INITIAL_VERSION = "1";
+  private static final String INITIAL_VERSION = "1";
 
   @SuppressWarnings("FieldMayBeFinal") // used by unit test
   private static TopologyRetriever INSTANCE =
@@ -68,10 +69,11 @@ public class RestBackendImpl implements RestBackend {
       };
 
   private final AuthenticationProxy atn = new AuthenticationProxy();
-  private final AuthorizationProxy atz = new AuthorizationProxy();
+  private AuthorizationProxy atz = new AuthorizationProxy();
   private final String principal;
   private final Supplier<Collection<String>> domainNamespaces;
   private V1UserInfo userInfo;
+  private CallBuilder callBuilder;
 
   /**
    * Construct a RestBackendImpl that is used to handle one WebLogic operator REST request.
@@ -85,10 +87,16 @@ public class RestBackendImpl implements RestBackend {
     this.domainNamespaces = domainNamespaces;
     this.principal = principal;
     userInfo = authenticate(accessToken);
+    callBuilder = userInfo != null ? new CallBuilder() :
+        new CallBuilder().withAuthentication(accessToken);
+    LOGGER.exiting();
   }
 
   private void authorize(String domainUid, Operation operation) {
     LOGGER.entering(domainUid, operation);
+    if (!authenticateWithTokenReview()) {
+      return;
+    }
     boolean authorized;
     if (domainUid == null) {
       authorized =
@@ -128,6 +136,9 @@ private String getNamespace(String domainUid) {
 
   private V1UserInfo authenticate(String accessToken) {
     LOGGER.entering();
+    if (!authenticateWithTokenReview()) {
+      return null;
+    }
     V1TokenReviewStatus status = atn.check(principal, accessToken,
         Main.isDedicated() ? getOperatorNamespace() : null);
     if (status == null) {
@@ -170,7 +181,7 @@ private Stream<Domain> getDomainStream() {
 
   private List<Domain> getDomains(String ns) {
     try {
-      return new CallBuilder().listDomain(ns).getItems();
+      return callBuilder.listDomain(ns).getItems();
     } catch (ApiException e) {
       throw handleApiException(e);
     }
@@ -315,7 +326,7 @@ private void patchClusterReplicas(Domain domain, String cluster, int replicas) {
 
   private void patchDomain(Domain domain, JsonPatchBuilder patchBuilder) {
     try {
-      new CallBuilder()
+      callBuilder
           .patchDomain(
               domain.getDomainUid(), domain.getMetadata().getNamespace(),
               new V1Patch(patchBuilder.build().toString()));
@@ -401,6 +412,25 @@ private WebApplicationException createWebApplicationException(int status, String
     return new WebApplicationException(rb.build());
   }
 
+  protected boolean authenticateWithTokenReview() {
+    return "true".equalsIgnoreCase(Optional.ofNullable(TuningParameters.getInstance().get("tokenReviewAuthentication"))
+        .orElse("false"));
+  }
+
+  V1UserInfo getUserInfo() {
+    return userInfo;
+  }
+
+  // Intended for unit tests
+  RestBackendImpl withAuthorizationProxy(AuthorizationProxy authorizationProxy) {
+    this.atz = authorizationProxy;
+    return this;
+  }
+
+  CallBuilder getCallBuilder() {
+    return this.callBuilder;
+  }
+
   interface TopologyRetriever {
     WlsDomainConfig getWlsDomainConfig(String ns, String domainUid);
   }