Skip to content

OWLS 84741: Scaling failed on Jenkins when setting Dedicated to true & io.kubernetes.client.openapi.ApiException: Not Found #1990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Oct 23, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,27 @@ For example, when using `curl`:
curl -v -k -H X-Requested-By:MyClient -H Content-Type:application/json -H Accept:application/json -H "Authorization:Bearer ..." -d '{ "managedServerCount": 3 }' https://.../scaling
```

If you omit the header, you'll get a `400 (bad request)` response. If you omit the Bearer Authentication header, then you'll get a `401 (Unauthorized)` response.
If you omit the header, you'll get a `400 (bad request)` response. If you omit the Bearer Authentication header, then you'll get a `401 (Unauthorized)` response. If the service account or user associated with the `Bearer` token does not have permission to `patch` the WebLogic domain resource, then you'll get a `403 (Forbidden)` response.

{{% notice note %}}
To resolve a `403 (Forbidden)` response, when calling the operator's REST scaling API, you may need to add the `patch` request verb to the cluster role associated with the WebLogic `domains` resource.
The example ClusterRole definition below grants `get`, `list`, `patch` and `update` access to the WebLogic `domains` resource
{{% /notice %}}

```
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: weblogic-domain-cluster-role
rules:
- apiGroups: ["weblogic.oracle"]
resources: ["domains"]
verbs: ["get", "list", "patch", update"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
---
```
##### Operator REST endpoints

The WebLogic Server Kubernetes Operator can expose both an internal and external REST HTTPS endpoint.
Expand Down Expand Up @@ -207,7 +226,7 @@ metadata:
rules:
- apiGroups: ["weblogic.oracle"]
resources: ["domains"]
verbs: ["get", "list", "update"]
verbs: ["get", "list", "patch", update"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,12 @@ You can access most of the REST services using `GET`, for example:
* To obtain a list of domains, send a `GET` request to the URL `/operator/latest/domains`
* To obtain a list of clusters in a domain, send a `GET` request to the URL `/operator/latest/domains/<domainUID>/clusters`

All of the REST services require authentication. Callers must pass in a valid token header and a CA certificate file. Callers should pass in the `Accept:/application/json` header.
All of the REST services require authentication. Callers must pass in a valid token header and a CA certificate file. In previous operator versions, the operator performed authentication and authorization checks using the Kubernetes token review and subject access review APIs, and then updated the Domain resource using the operator's privileges. Now, by default, the operator will use the caller's bearer token to perform the underlying update to the Domain resource using the caller's privileges and thus delegating authentication and authorization checks directly to the Kubernetes API Server (see [REST interface configuration]({{< relref "/userguide/managing-operators/using-the-operator/using-helm.md#rest-interface-configuration" >}})).
{{% notice note %}}
When using the operator's REST services to scale up or down a WebLogic cluster, you may need to grant `patch` access to the user or service account associated with the caller's bearer token. This can be done with an RBAC ClusterRoleBinding between the user or service account and the ClusterRole that defines the permissions for the WebLogic `domains` resource.
{{% /notice %}}

Callers should pass in the `Accept:/application/json` header.

To protect against Cross Site Request Forgery (CSRF) attacks, the operator REST API requires that you send in a `X-Requested-By` header when you invoke a REST endpoint that makes a change (for example, when you POST to the `/scale` endpoint). The value is an arbitrary name such as `MyClient`. For example, when using `curl`:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -468,7 +468,22 @@ Example:
```
externalOperatorKey: QmFnIEF0dHJpYnV0ZXMKICAgIGZyaWVuZGx5TmFtZTogd2VibG9naWMtb3B ...
```

##### `tokenReviewAuthentication`
If set to `true`, `tokenReviewAuthentication` specifies whether the the operator's REST API should use:
* Kubernetes token review API for authenticating users and
* Kubernetes subject access review API for authorizing a user's operation (`get`, `list`,
`patch`, and such) on a resource.
* Update the Domain resource using the operator's privileges.

If set to `false`, the operator's REST API will use the caller's bearer token for any update
to the Domain resource so that it is done using the caller's privileges.

Defaults to `false`.

Example:
```
tokenReviewAuthentication: true
```
#### Debugging options

##### `remoteDebugNodePortEnabled`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,6 @@ public void testDedicatedModeSameNamespace() {
* scaling up cluster-1 in domain1Namespace succeeds.
*/
@Test
@Disabled("Disable the test due to JIRA OWLS-84741")
@Order(3)
@DisplayName("Scale up cluster-1 in domain1Namespace and verify it succeeds")
public void testDedicatedModeSameNamespaceScale() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -566,6 +566,7 @@ private static boolean createRbacApiObjectsForWLDFScript(String domainNamespace,
.addResourcesItem("domains")
.addVerbsItem("get")
.addVerbsItem("list")
.addVerbsItem("patch")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are adding a required role verb (this seems correct), but reminds me that we need to document this requirement in the scaling and/or RBAC documentation. Similarly, we need to document how to create the correct roles and role bindings so that the service account used with the scaling API has the necessary permissions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated documentation in /userguide/managing-operators/using-the-operator/using-helm.md, /userguide/managing-operators/using-the-operator/the-rest-api.md, /userguide/managing-domains/domain-lifecycle/scaling.md, and /kubernetes/charts/weblogic-operator/values.yaml

.addVerbsItem("update"))
.addRulesItem(new V1PolicyRule()
.addApiGroupsItem("apiextensions.k8s.io")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ data:
{{- if .clusterSizePaddingValidationEnabled }}
clusterSizePaddingValidationEnabled: {{ .clusterSizePaddingValidationEnabled | quote }}
{{- end }}
{{- if .tokenReviewAuthentication }}
tokenReviewAuthentication: {{ .tokenReviewAuthentication | quote }}
{{- end }}
kind: "ConfigMap"
metadata:
labels:
Expand Down
10 changes: 10 additions & 0 deletions kubernetes/charts/weblogic-operator/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -191,3 +191,13 @@ externalServiceNameSuffix: "-ext"
# cluster size.
# The default value is true.
clusterSizePaddingValidationEnabled: true

# tokenReviewAuthentication, if set to true, specifies whether the the operator's REST API should use
# 1. Kubernetes token review API for authenticating users, and
# 2. Kubernetes subject access review API for authorizing a user's operation (get, list,
# patch, etc) on a resource.
# 3. Update the Domain resource using the operator's privileges.
# This parameter, if set to false, will use the caller's bearer token for any update
# to the Domain resource so that it is done using the caller's privileges.
# The default value is false.
#tokenReviewAuthentication: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The default value is false." would be a good thing to note.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This leaves ambiguous the meaning of "false." Would you please update this to describe that when the value is false that the caller's bearer token will be delegated so that the update to the Domain resource is done using the caller's privileges?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated with description.

Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,12 @@

package oracle.kubernetes.operator.helpers;

import java.io.IOException;
import java.util.Optional;
import java.util.function.Consumer;
import javax.annotation.Nonnull;

import com.google.common.base.Strings;
import io.kubernetes.client.custom.V1Patch;
import io.kubernetes.client.openapi.ApiCallback;
import io.kubernetes.client.openapi.ApiClient;
Expand Down Expand Up @@ -42,6 +44,8 @@
import io.kubernetes.client.openapi.models.V1TokenReview;
import io.kubernetes.client.openapi.models.V1beta1CustomResourceDefinition;
import io.kubernetes.client.openapi.models.VersionInfo;
import io.kubernetes.client.util.ClientBuilder;
import io.kubernetes.client.util.credentials.AccessTokenAuthentication;
import okhttp3.Call;
import oracle.kubernetes.operator.TuningParameters;
import oracle.kubernetes.operator.TuningParameters.CallBuilderTuning;
Expand Down Expand Up @@ -87,7 +91,7 @@ public <T> T execute(
private static SynchronousCallDispatcher DISPATCHER = DEFAULT_DISPATCHER;
private static final AsyncRequestStepFactory DEFAULT_STEP_FACTORY = AsyncRequestStep::new;
private static AsyncRequestStepFactory STEP_FACTORY = DEFAULT_STEP_FACTORY;
private final ClientPool helper;
private ClientPool helper;
private final Boolean allowWatchBookmarks = false;
private final String dryRun = null;
private final String pretty = "false";
Expand Down Expand Up @@ -469,6 +473,10 @@ private CallBuilder(CallBuilderTuning tuning, ClientPool helper) {
this.helper = helper;
}

public CallBuilder(ClientPool pool) {
this(getCallBuilderTuning(), pool);
}

private static CallBuilderTuning getCallBuilderTuning() {
return Optional.ofNullable(TuningParameters.getInstance())
.map(TuningParameters::getCallBuilderTuning)
Expand Down Expand Up @@ -1939,4 +1947,31 @@ private <T> Step createRequestAsync(
private CancellableCall wrap(Call call) {
return new CallWrapper(call);
}

public ClientPool getClientPool() {
return this.helper;
}

/**
* Create AccessTokenAuthentication component for authenticating user represented by
* the given token.
* @param accessToken - User's Bearer token
* @return - this CallBuilder instance
*/
public CallBuilder withAuthentication(String accessToken) {
if (!Strings.isNullOrEmpty(accessToken)) {
this.helper = new ClientPool().withApiClient(createApiClient(accessToken));
}
return this;
}

private ApiClient createApiClient(String accessToken) {
try {
ClientBuilder builder = ClientBuilder.standard();
return builder.setAuthentication(
new AccessTokenAuthentication(accessToken)).build();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,11 @@ private ApiClient getApiClient() {
return client;
}

public ClientPool withApiClient(ApiClient apiClient) {
instance.getAndSet(apiClient);
return this;
}

private static class DefaultClientFactory implements ClientFactory {
private final AtomicBoolean first = new AtomicBoolean(true);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import io.kubernetes.client.openapi.models.V1TokenReviewStatus;
import io.kubernetes.client.openapi.models.V1UserInfo;
import oracle.kubernetes.operator.Main;
import oracle.kubernetes.operator.TuningParameters;
import oracle.kubernetes.operator.helpers.AuthenticationProxy;
import oracle.kubernetes.operator.helpers.AuthorizationProxy;
import oracle.kubernetes.operator.helpers.AuthorizationProxy.Operation;
Expand Down Expand Up @@ -55,7 +56,7 @@ public class RestBackendImpl implements RestBackend {
private static final LoggingFacade LOGGER = LoggingFactory.getLogger("Operator", "Operator");
private static final String NEW_CLUSTER_REPLICAS =
"{'clusterName':'%s','replicas':%d}".replaceAll("'", "\"");
public static final String INITIAL_VERSION = "1";
private static final String INITIAL_VERSION = "1";

@SuppressWarnings("FieldMayBeFinal") // used by unit test
private static TopologyRetriever INSTANCE =
Expand All @@ -68,10 +69,11 @@ public class RestBackendImpl implements RestBackend {
};

private final AuthenticationProxy atn = new AuthenticationProxy();
private final AuthorizationProxy atz = new AuthorizationProxy();
private AuthorizationProxy atz = new AuthorizationProxy();
private final String principal;
private final Supplier<Collection<String>> domainNamespaces;
private V1UserInfo userInfo;
private CallBuilder callBuilder;

/**
* Construct a RestBackendImpl that is used to handle one WebLogic operator REST request.
Expand All @@ -85,10 +87,16 @@ public class RestBackendImpl implements RestBackend {
this.domainNamespaces = domainNamespaces;
this.principal = principal;
userInfo = authenticate(accessToken);
callBuilder = userInfo != null ? new CallBuilder() :
new CallBuilder().withAuthentication(accessToken);
LOGGER.exiting();
}

private void authorize(String domainUid, Operation operation) {
LOGGER.entering(domainUid, operation);
if (!authenticateWithTokenReview()) {
return;
}
boolean authorized;
if (domainUid == null) {
authorized =
Expand Down Expand Up @@ -128,6 +136,9 @@ private String getNamespace(String domainUid) {

private V1UserInfo authenticate(String accessToken) {
LOGGER.entering();
if (!authenticateWithTokenReview()) {
return null;
}
V1TokenReviewStatus status = atn.check(principal, accessToken,
Main.isDedicated() ? getOperatorNamespace() : null);
if (status == null) {
Expand Down Expand Up @@ -170,7 +181,7 @@ private Stream<Domain> getDomainStream() {

private List<Domain> getDomains(String ns) {
try {
return new CallBuilder().listDomain(ns).getItems();
return callBuilder.listDomain(ns).getItems();
} catch (ApiException e) {
throw handleApiException(e);
}
Expand Down Expand Up @@ -315,7 +326,7 @@ private void patchClusterReplicas(Domain domain, String cluster, int replicas) {

private void patchDomain(Domain domain, JsonPatchBuilder patchBuilder) {
try {
new CallBuilder()
callBuilder
.patchDomain(
domain.getDomainUid(), domain.getMetadata().getNamespace(),
new V1Patch(patchBuilder.build().toString()));
Expand Down Expand Up @@ -401,6 +412,25 @@ private WebApplicationException createWebApplicationException(int status, String
return new WebApplicationException(rb.build());
}

protected boolean authenticateWithTokenReview() {
return "true".equalsIgnoreCase(Optional.ofNullable(TuningParameters.getInstance().get("tokenReviewAuthentication"))
.orElse("false"));
}

V1UserInfo getUserInfo() {
return userInfo;
}

// Intended for unit tests
RestBackendImpl withAuthorizationProxy(AuthorizationProxy authorizationProxy) {
this.atz = authorizationProxy;
return this;
}

CallBuilder getCallBuilder() {
return this.callBuilder;
}

interface TopologyRetriever {
WlsDomainConfig getWlsDomainConfig(String ns, String domainUid);
}
Expand Down
Loading