Skip to content

Latest commit

 

History

History
308 lines (281 loc) · 7.59 KB

cloud-experts-rosa-cloudwatch-sts.adoc

File metadata and controls

308 lines (281 loc) · 7.59 KB

Configuring log forwarding for CloudWatch logs and STS

Use this tutorial to deploy the {clo} and configure it to use Security Token Services (STS) authentication to forward logs to CloudWatch.

Prerequisites

  • A {product-title} (ROSA) Classic cluster

  • The jq command-line interface (CLI)

  • The Amazon Web Services (AWS) CLI (aws)

Setting up your environment

  1. Configure the following environment variables, changing the cluster name to suit your cluster:

    Note

    You must be logged in as an administrator.

    		 
    $ export ROSA_CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
$ export REGION=$(rosa describe cluster -c ${ROSA_CLUSTER_NAME} --output json | jq -r .region.id)
$ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
$ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
$ export AWS_PAGER=""
$ export SCRATCH="/tmp/${ROSA_CLUSTER_NAME}/clf-cloudwatch-sts"
$ mkdir -p ${SCRATCH}

  2. Ensure all fields output correctly before moving to the next section:

    $ echo "Cluster: ${ROSA_CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

Preparing your AWS account

  1. Create an Identity Access Management (IAM) policy for {logging}:

    $ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaCloudWatch'].{ARN:Arn}" --output text)
$ if [[ -z "${POLICY_ARN}" ]]; then
cat << EOF > ${SCRATCH}/policy.json
{
  "Version": "2012-10-17",
  "Statement": [
     {
           "Effect": "Allow",
           "Action": [
              "logs:CreateLogGroup",
              "logs:CreateLogStream",
              "logs:DescribeLogGroups",
              "logs:DescribeLogStreams",
              "logs:PutLogEvents",
              "logs:PutRetentionPolicy"
           ],
           "Resource": "arn:aws:logs:*:*:*"
     }
]
}
EOF
POLICY_ARN=$(aws iam create-policy --policy-name "RosaCloudWatch" \
--policy-document file:///${SCRATCH}/policy.json --query Policy.Arn --output text)
fi
$ echo ${POLICY_ARN}

  2. Create an IAM role trust policy for the cluster:

    $ cat <<EOF > ${SCRATCH}/trust-policy.json
{
   "Version": "2012-10-17",
   "Statement": [{
     "Effect": "Allow",
     "Principal": {
       "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
     },
     "Action": "sts:AssumeRoleWithWebIdentity",
     "Condition": {
       "StringEquals": {
         "${OIDC_ENDPOINT}:sub": "system:serviceaccount:openshift-logging:logcollector"
       }
     }
   }]
}
EOF
$ ROLE_ARN=$(aws iam create-role --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
      --assume-role-policy-document file://${SCRATCH}/trust-policy.json \
      --query Role.Arn --output text)
$ echo ${ROLE_ARN}

  3. Attach the IAM policy to the IAM role:

    $ aws iam attach-role-policy --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
   --policy-arn ${POLICY_ARN}

Deploying Operators

  1. Deploy the {clo}:

    $  cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     labels:
      operators.coreos.com/cluster-logging.openshift-logging: ""
     name: cluster-logging
     namespace: openshift-logging
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: cluster-logging
     source: redhat-operators
     sourceNamespace: openshift-marketplace
EOF

  2. Create a secret:

    $  cat << EOF | oc apply -f -
  apiVersion: v1
  kind: Secret
  metadata:
    name: cloudwatch-credentials
    namespace: openshift-logging
  stringData:
    role_arn: $ROLE_ARN
EOF

Configuring cluster logging

  1. Create a ClusterLogForwarder custom resource (CR):

    $ cat << EOF | oc apply -f -
  apiVersion: "logging.openshift.io/v1"
  kind: ClusterLogForwarder
  metadata:
    name: instance
    namespace: openshift-logging
  spec:
    outputs:
      - name: cw
        type: cloudwatch
        cloudwatch:
          groupBy: namespaceName
          groupPrefix: rosa-${ROSA_CLUSTER_NAME}
          region: ${REGION}
        secret:
          name: cloudwatch-credentials
    pipelines:
      - name: to-cloudwatch
        inputRefs:
          - infrastructure
          - audit
          - application
        outputRefs:
          - cw
EOF

  2. Create a ClusterLogging CR:

    $ cat << EOF | oc apply -f -
  apiVersion: logging.openshift.io/v1
  kind: ClusterLogging
  metadata:
    name: instance
    namespace: openshift-logging
  spec:
    collection:
      logs:
         type: vector
    managementState: Managed
EOF

Checking CloudWatch for logs

  • Use either the AWS console or the AWS CLI to validate that there are log streams from the cluster.

    • To validate the logs in the AWS CLI, run the following command:

      $ aws logs describe-log-groups --log-group-name-prefix rosa-${ROSA_CLUSTER_NAME}
      Sample output
      {
  "logGroups": [
      {
        "logGroupName": "rosa-xxxx.audit",
        "creationTime": 1661286368369,
        "metricFilterCount": 0,
        "arn": "arn:aws:logs:us-east-2:xxxx:log-group:rosa-xxxx.audit:*",
        "storedBytes": 0
      },
      {
        "logGroupName": "rosa-xxxx.infrastructure",
        "creationTime": 1661286369821,
        "metricFilterCount": 0,
        "arn": "arn:aws:logs:us-east-2:xxxx:log-group:rosa-xxxx.infrastructure:*",
        "storedBytes": 0
      }
  ]
}
      Note

      If this is a new cluster, you might not see a log group for application logs as applications are not yet running.

Cleaning up your resources

  1. Delete the ClusterLogForwarder CR:

    $ oc delete -n openshift-logging clusterlogforwarder instance

  2. Delete the ClusterLogging CR:

    $ oc delete -n openshift-logging clusterlogging instance

  3. Detach the IAM policy to the IAM role:

    $ aws iam detach-role-policy --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
   --policy-arn "${POLICY_ARN}"

  4. Delete the IAM role:

    $ aws iam delete-role --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch"

  5. Delete the IAM policy:

    Important

    Only delete the IAM policy if there are no other resources using the policy.

    		 
    $ aws iam delete-policy --policy-arn "${POLICY_ARN}"

  6. Delete the CloudWatch log groups:

    $ aws logs delete-log-group --log-group-name "rosa-${ROSA_CLUSTER_NAME}.audit"
$ aws logs delete-log-group --log-group-name "rosa-${ROSA_CLUSTER_NAME}.infrastructure"