Creating AWS resources manually

To install an {product-title} cluster that is configured to use the CCO in manual mode with STS, you must first manually create the required AWS resources.

Procedure

Generate a private key to sign the ServiceAccount object:
```
$ openssl genrsa -out sa-signer 4096
```

Generate a ServiceAccount object public key:

$ openssl rsa -in sa-signer -pubout -out sa-signer.pub

Create an S3 bucket to hold the OIDC configuration:
```
$ aws s3api create-bucket --bucket <oidc_bucket_name> --region <aws_region> --create-bucket-configuration LocationConstraint=<aws_region>
```
Note

If the value of <aws_region> is us-east-1, do not specify the LocationConstraint parameter.

Retain the S3 bucket URL:

OPENID_BUCKET_URL="https://<oidc_bucket_name>.s3.<aws_region>.amazonaws.com"

Build an OIDC configuration:

Create a file named keys.json that contains the following information:
```
{
    "keys": [
        {
            "use": "sig",
            "kty": "RSA",
            "kid": "<public_signing_key_id>",
            "alg": "RS256",
            "n": "<public_signing_key_modulus>",
            "e": "<public_signing_key_exponent>"
        }
    ]
}
```
where:
- <public_signing_key_id> is generated from the public key with:
  $ openssl rsa -in sa-signer.pub -pubin --outform DER | openssl dgst -binary -sha256 | openssl base64 | tr '/+' '_-' | tr -d '='
  This command converts the public key to DER format, performs a SHA-256 checksum on the binary representation, encodes the data with base64 encoding, and then changes the base64-encoded output to base64URL encoding.
- <public_signing_key_modulus> is generated from the public key with:
  $ openssl rsa -pubin -in sa-signer.pub -modulus -noout | sed -e 's/Modulus=//' | xxd -r -p | base64 -w0 | tr '/+' '_-' | tr -d '='
  This command prints the modulus of the public key, extracts the hex representation of the modulus, converts the ASCII hex to binary, encodes the data with base64 encoding, and then changes the base64-encoded output to base64URL encoding.
- <public_signing_key_exponent> is generated from the public key with:
  $ printf "%016x" $(openssl rsa -pubin -in sa-signer.pub -noout -text | grep Exponent | awk '{ print $2 }') | awk '{ sub(/(00)+/, "", $1); print $1 }' | xxd -r -p | base64 -w0 | tr '/+' '_-' | tr -d '='
  This command extracts the decimal representation of the public key exponent, prints it as hex with a padded 0 if needed, removes leading 00 pairs, converts the ASCII hex to binary, encodes the data with base64 encoding, and then changes the base64-encoded output to use only characters that can be used in a URL.

Create a file named openid-configuration that contains the following information:

{
	"issuer": "$OPENID_BUCKET_URL",
	"jwks_uri": "${OPENID_BUCKET_URL}/keys.json",
    "response_types_supported": [
        "id_token"
    ],
    "subject_types_supported": [
        "public"
    ],
    "id_token_signing_alg_values_supported": [
        "RS256"
    ],
    "claims_supported": [
        "aud",
        "exp",
        "sub",
        "iat",
        "iss",
        "sub"
    ]
}

Upload the OIDC configuration:

$ aws s3api put-object --bucket <oidc_bucket_name> --key keys.json --body ./keys.json

$ aws s3api put-object --bucket <oidc_bucket_name> --key '.well-known/openid-configuration' --body ./openid-configuration

Where <oidc_bucket_name> is the S3 bucket that was created to hold the OIDC configuration.

Allow the AWS IAM OpenID Connect (OIDC) identity provider to read these files:

$ aws s3api put-object-acl --bucket <oidc_bucket_name> --key keys.json --acl public-read

$ aws s3api put-object-acl --bucket <oidc_bucket_name> --key '.well-known/openid-configuration' --acl public-read

Create an AWS IAM OIDC identity provider:

Get the certificate chain from the server that hosts the OIDC configuration:

$ echo | openssl s_client -servername $<oidc_bucket_name>.s3.$<aws_region>.amazonaws.com -connect $<oidc_bucket_name>.s3.$<aws_region>.amazonaws.com:443 -showcerts 2>/dev/null | awk '/BEGIN/,/END/{ if(/BEGIN/){a++}; out="cert"a".pem"; print >out}'

Calculate the fingerprint for the certificate at the root of the chain:
```
$ export BUCKET_FINGERPRINT=$(openssl x509 -in cert<number>.pem -fingerprint -noout | sed -e 's/.*Fingerprint=//' -e 's/://g')
```
Where <number> is the highest number in the files that were saved. For example, if 2 is the highest number in the files that were saved, use cert2.pem.

Create the identity provider:

$ aws iam create-open-id-connect-provider --url $OPENID_BUCKET_URL --thumbprint-list $BUCKET_FINGERPRINT --client-id-list openshift sts.amazonaws.com

Retain the returned ARN of the newly created identity provider. This ARN is later referred to as <aws_iam_openid_arn>.

Generate IAM roles:

Locate all CredentialsRequest CRs in this release image that target the cloud you are deploying on:
```
$ oc adm release extract quay.io/openshift-release-dev/ocp-release:4.<y>.<z>-x86_64 --credentials-requests --cloud=aws
```
Where <y> and <z> are the numbers corresponding to the version of {product-title} you are installing.

For each CredentialsRequest CR, create an IAM role of type Web identity using the previously created IAM Identity Provider that grants the necessary permissions and establishes a trust relationship that trusts the identity provider previously created.

For example, for the openshift-machine-api-operator CredentialsRequest CR in 0000_30_machine-api-operator_00_credentials-request.yaml, create an IAM role that allows an identity from the created OIDC provider created for the cluster, similar to the following:

{
    "Role": {
        "Path": "/",
        "RoleName": "openshift-machine-api-aws-cloud-credentials",
        "RoleId": "ARSOMEROLEID",
        "Arn": "arn:aws:iam::123456789012:role/openshift-machine-api-aws-cloud-credentials",
        "CreateDate": "2021-01-06T15:54:13Z",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "<aws_iam_openid_arn>"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com/$BUCKET_NAME:aud": "openshift"
                        }
                    }
                }
            ]
        },
        "Description": "OpenShift role for openshift-machine-api/aws-cloud-credentials",
        "MaxSessionDuration": 3600,
        "RoleLastUsed": {
            "LastUsedDate": "2021-02-03T02:51:24Z",
            "Region": "<aws_region>"
        }
    }
}

Where <aws_iam_openid_arn> is the returned ARN of the newly created identity provider.

To further restrict the role such that only specific cluster ServiceAccount objects can assume the role, modify the trust relationship of each role by updating the .Role.AssumeRolePolicyDocument.Statement[].Condition field to the specific ServiceAccount objects for each component.

Modify the trust relationship of the cluster-image-registry-operator role to have the following condition:

"Condition": {
  "StringEquals": {
    "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
      "system:serviceaccount:openshift-image-registry:registry",
      "system:serviceaccount:openshift-image-registry:cluster-image-registry-operator"
    ]
  }
}

Modify the trust relationship of the openshift-ingress-operator to have the following condition:

"Condition": {
  "StringEquals": {
    "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
      "system:serviceaccount:openshift-ingress-operator:ingress-operator"
    ]
  }
}

Modify the trust relationship of the openshift-cluster-csi-drivers to have the following condition:

"Condition": {
  "StringEquals": {
    "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
      "system:serviceaccount:openshift-cluster-csi-drivers:aws-ebs-csi-driver-operator",
      "system:serviceaccount:openshift-cluster-csi-drivers:aws-ebs-csi-driver-controller-sa"
    ]
  }
}

Modify the trust relationship of the openshift-machine-api to have the following condition:

"Condition": {
  "StringEquals": {
    "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
      "system:serviceaccount:openshift-machine-api:machine-api-controllers"
    ]
  }
}

For each IAM role, attach an IAM policy to the role that reflects the required permissions from the corresponding CredentialsRequest objects.

For example, for openshift-machine-api, attach an IAM policy similar to the following:

{
    "RoleName": "openshift-machine-api-aws-cloud-credentials",
    "PolicyName": "openshift-machine-api-aws-cloud-credentials",
    "PolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateTags",
                    "ec2:DescribeAvailabilityZones",
                    "ec2:DescribeDhcpOptions",
                    "ec2:DescribeImages",
                    "ec2:DescribeInstances",
                    "ec2:DescribeSecurityGroups",
                    "ec2:DescribeSubnets",
                    "ec2:DescribeVpcs",
                    "ec2:RunInstances",
                    "ec2:TerminateInstances",
                    "elasticloadbalancing:DescribeLoadBalancers",
                    "elasticloadbalancing:DescribeTargetGroups",
                    "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                    "elasticloadbalancing:RegisterTargets",
                    "iam:PassRole",
                    "iam:CreateServiceLinkedRole"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "kms:Decrypt",
                    "kms:Encrypt",
                    "kms:GenerateDataKey",
                    "kms:GenerateDataKeyWithoutPlainText",
                    "kms:DescribeKey"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "kms:RevokeGrant",
                    "kms:CreateGrant",
                    "kms:ListGrants"
                ],
                "Resource": "*",
                "Condition": {
                    "Bool": {
                        "kms:GrantIsForAWSResource": true
                    }
                }
            }
        ]
    }
}

Prepare to run the {product-title} installer:

Create the install-config.yaml file:

$ ./openshift-install create install-config

Configure the cluster to install with the CCO in manual mode:
```
$ echo "credentialsMode: Manual" >> install-config.yaml
```
Create install manifests:
```
$ ./openshift-install create manifests
```
Create a tls directory, and copy the private key generated previously there:

Note

The target file name must be ./tls/bound-service-account-signing-key.key.
```
$ mkdir tls ; cp <path_to_service_account_signer> ./tls/bound-service-account-signing-key.key
```

Create a custom Authentication CR with the file name cluster-authentication-02-config.yaml:

$ cat << EOF > manifests/cluster-authentication-02-config.yaml
apiVersion: config.openshift.io/v1
kind: Authentication
metadata:
  name: cluster
spec:
  serviceAccountIssuer: $OPENID_BUCKET_URL
EOF

For each CredentialsRequest CR that is extracted from the release image, create a secret with the target namespace and target name that is indicated in each CredentialsRequest, substituting the AWS IAM role ARN created previously for each component:

Example secret manifest for openshift-machine-api:

$ cat manifests/openshift-machine-api-aws-cloud-credentials-credentials.yaml
apiVersion: v1
stringData:
  credentials: |-
    [default]
    role_arn = arn:aws:iam::123456789012:role/openshift-machine-api-aws-cloud-credentials
    web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
kind: Secret
metadata:
  name: aws-cloud-credentials
  namespace: openshift-machine-api
type: Opaque

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sts-mode-installing-manual-config.adoc

sts-mode-installing-manual-config.adoc

Creating AWS resources manually

Files

sts-mode-installing-manual-config.adoc

Latest commit

History

sts-mode-installing-manual-config.adoc

File metadata and controls

Creating AWS resources manually