This project contains the rosa command line tool that simplifies the use of Red Hat OpenShift Service on AWS, also known as ROSA.
This guide walks through setting up your first Red Hat OpenShift Service on AWS cluster using rosa.
If you have already installed the required prerequisites, here are the commands you need to create a cluster.
$ rosa init ## Configures your AWS account and ensures everything is setup correctly
$ rosa create cluster --cluster-name <my-cluster-name> ## Starts the cluster creation process (~30-40minutes)
$ rosa logs install -c <my-cluster-name> --watch ## Watch your logs as your cluster creates
$ rosa create idp --cluster <my-cluster-name> --interactive ## Connect your IDP to your cluster
$ rosa create user --cluster <my-cluster-name> --dedicated-admins <admin-username> ## Promotes a user from your IDP to admin level
$ rosa describe cluster <my-cluster-name> ## Checks if your install is ready (look for State: Ready), and provides your Console URL to login to the web console.
If you get stuck or you are starting out and want more details, the rest of this guide includes the following steps:
- Installation prerequisites
- Preparing your AWS account for cluster installation
- Creating your cluster
- Accessing your cluster
- Creating admin users for your cluster
- Cleaning up
By the end of this guide you will have a Red Hat OpenShift Service on AWS cluster running in your AWS account.
Complete the following prerequisites before creating your ROSA cluster.
Unless you're just testing out ROSA, we recommend using a dedicated AWS account to run any production clusters. If you are utilizing AWS Organizations, you can use an AWS account within your organization or create a new one.
If you are using AWS organizations and you need to have a Service Control Policy (SCP) applied to the AWS account you plan to use, see the Red Hat Requirements for Customer Cloud Subscriptions for details on the minimum required SCP.
As part of the cluster creation process, rosa will create an osdCcsAdmin IAM user. This user will have Programmatic access enabled and have the AdministratorAccess policy attached to it. The AWS credentials provided in the next section will be used to create this user.
Install the aws-cli.
Configure the aws-cli to use the AWS account that you would like to deploy your cluster into. Modify your ~/.aws/credentials file to specify the correct aws_access_key_id and aws_secret_access_key.
$ cat ~/.aws/credentials
[default]
aws_access_key_id = <my-aws-access-key-id>
aws_secret_access_key = <my-aws-secret-access-key>
Modify your ~/.aws/config file to specify the AWS region you want to use.
$ cat ~/.aws/config
[default]
output = table
region = us-east-2
NOTE If you are not using the default profile for your AWS credentials, run the following command to export your profile settings to your shell, replacing
<my-profile>with the name of your AWS profile:export AWS_PROFILE=<my-profile>
To verify your configuration, run the following command to query the AWS api:
$ aws ec2 describe-regions
---------------------------------------------------------------------------------
| DescribeRegions |
+-------------------------------------------------------------------------------+
|| Regions ||
|+-----------------------------------+-----------------------+-----------------+|
|| Endpoint | OptInStatus | RegionName ||
|+-----------------------------------+-----------------------+-----------------+|
|| ec2.eu-north-1.amazonaws.com | opt-in-not-required | eu-north-1 ||
|| ec2.ap-south-1.amazonaws.com | opt-in-not-required | ap-south-1 ||
|| ec2.eu-west-3.amazonaws.com | opt-in-not-required | eu-west-3 ||
|| ec2.eu-west-2.amazonaws.com | opt-in-not-required | eu-west-2 ||
|| ec2.eu-west-1.amazonaws.com | opt-in-not-required | eu-west-1 ||
|| ec2.ap-northeast-2.amazonaws.com | opt-in-not-required | ap-northeast-2 ||
|| ec2.ap-northeast-1.amazonaws.com | opt-in-not-required | ap-northeast-1 ||
|| ec2.sa-east-1.amazonaws.com | opt-in-not-required | sa-east-1 ||
|| ec2.ca-central-1.amazonaws.com | opt-in-not-required | ca-central-1 ||
|| ec2.ap-southeast-1.amazonaws.com | opt-in-not-required | ap-southeast-1 ||
|| ec2.ap-southeast-2.amazonaws.com | opt-in-not-required | ap-southeast-2 ||
|| ec2.eu-central-1.amazonaws.com | opt-in-not-required | eu-central-1 ||
|| ec2.us-east-1.amazonaws.com | opt-in-not-required | us-east-1 ||
|| ec2.us-east-2.amazonaws.com | opt-in-not-required | us-east-2 ||
|| ec2.us-west-1.amazonaws.com | opt-in-not-required | us-west-1 ||
|| ec2.us-west-2.amazonaws.com | opt-in-not-required | us-west-2 ||
|+-----------------------------------+-----------------------+-----------------+|
Download the latest release of rosa and add it to your path.
Verify your installation by running the following command:
$ rosa
Command line tool for ROSA.
Usage:
rosa [command]
Available Commands:
completion Generates bash completion scripts
create Create a resource from stdin
delete Delete a specific resource
describe Show details of a specific resource
download Download necessary tools for using your cluster
edit Edit a specific resource
help Help about any command
init Applies templates to support Red Hat OpenShift Service on AWS
list List all resources of a specific type
login Log in to your Red Hat account
logout Log out
logs Show installation or uninstallation logs for a cluster
verify Verify resources are configured correctly for cluster install
version Prints the version of the tool
whoami Displays user account information
Flags:
--debug Enable debug mode.
-h, --help help for rosa
-v, --v Level log level for V logs
Use "rosa [command] --help" for more information about a command.
You can add rosa bash completion to your current terminal session by running the following commands:
$ rosa completion > ~/.rosa-completion.sh
$ source ~/.rosa-completion.sh
If you want to persist rosa bash completion on new terminal sessions, add the output from rosa completion to your .bashrc file, or to the appropriate location for your operating system.
Run the following command to verify that your AWS account has the necessary permissions:
$ rosa verify permissions
I: Validating SCP policies...
I: AWS SCP policies ok
Verify that your AWS account has the necessary quota to deploy an OpenShift cluster. Sometimes quota varies by region, which may prevent you from deploying:
$ rosa verify quota --region=us-west-2
I: Validating AWS quota...
E: Insufficient AWS quotas
E: Service ec2 quota code L-0263D0A3 Number of EIPs - VPC EIPs not valid, expected quota of at least 5, but got 2
If needed, try another region:
$ rosa verify quota --region=us-east-2
I: Validating AWS quota...
I: AWS quota ok
If you need to increase your quota, navigate to your AWS console, and request a quota increase for the service that failed.
Once both the permissions and quota checks pass, proceed to preparing your AWS account for cluster installation.
In this step you log in to your Red Hat account using rosa, and then initialize your AWS account.
If you do not already have a Red Hat account, create one here. Be sure to accept the required terms and conditions. Then, check your email for a verification link.
After creating your Red Hat account, follow this link to get an offline access token.
Run the following command to log in to your Red Hat account with rosa. Replace <my-offline-access-token> with your token:
$ rosa login --token="<my-offline-access-token>"
Run the following command to verify your Red Hat and AWS credentials are setup correctly. Check that your AWS Account ID, Default Region, and ARN match what you expect. You can safely ignore the rows beginning with OCM for now (OCM stands for OpenShift Cluster Manager).
$ rosa whoami
AWS Account ID: 000000000000
AWS Default Region: us-east-2
AWS ARN: arn:aws:iam::000000000000:user/hello
OCM API: https://api.openshift.com
OCM Account ID: 1DzGIdIhqEWyt8UUXQhSoWaaaaa
OCM Account Name: Your Name
OCM Account Username: you@domain.com
OCM Account Email: you@domain.com
OCM Organization ID: 1HopHfA2hcmhup5gCr2uH5aaaaa
OCM Organization Name: Red Hat
OCM Organization External ID: 0000000
This step runs a CloudFormation template that prepares your AWS account for OpenShift deployment and management. This step typically takes 1-2 minutes to complete.
$ rosa init
I: Logged in as 'rh-rosa-user' on 'https://api.openshift.com'
I: Validating AWS credentials...
I: AWS credentials are valid!
I: Validating SCP policies...
I: AWS SCP policies ok
I: Validating AWS quota...
I: AWS quota ok
I: Ensuring cluster administrator user 'osdCcsAdmin'...
I: Admin user 'osdCcsAdmin' created successfuly!
I: Verifying whether OpenShift command-line tool is available...
W: OpenShift command-line tool is not installed.
Run 'rosa download oc' to download the latest version, then add it to your PATH.
NOTE If you have not already installed the OpenShift Command Line Utility, also known as
oc, run the command in the output to download it now.
To view all of the available options when creating a cluster, run the following command:
$ rosa create cluster --help
Create cluster.
Usage:
rosa create cluster [flags]
Examples:
# Create a cluster named "mycluster"
rosa create cluster --cluster-name=mycluster
# Create a cluster in the us-east-2 region
rosa create cluster --cluster-name=mycluster --region=us-east-2
Flags:
-c, --cluster-name string Name of the cluster. This will be used when generating a sub-domain for your cluster on openshiftapps.com.
--multi-az Deploy to multiple data centers.
-r, --region string AWS region where your worker pool will be located. (overrides the AWS_REGION environment variable)
--version string Version of OpenShift that will be used to install the cluster, for example "4.3.10"
--channel-group string Channel group is the name of the group where this image belongs, for example "stable" or "fast". (default "stable")
--compute-machine-type string Instance type for the compute nodes. Determines the amount of memory and vCPU allocated to each compute node.
--compute-nodes int Number of worker nodes to provision per zone. Single zone clusters need at least 2 nodes, multizone clusters need at least 3 nodes. (default 2)
--machine-cidr ipNet Block of IP addresses used by OpenShift while installing the cluster, for example "10.0.0.0/16".
--service-cidr ipNet Block of IP addresses for services, for example "172.30.0.0/16".
--pod-cidr ipNet Block of IP addresses from which Pod IP addresses are allocated, for example "10.128.0.0/14".
--host-prefix int Subnet prefix length to assign to each individual node. For example, if host prefix is set to "23", then each node is assigned a /23 subnet out of the given CIDR.
--private Restrict master API endpoint and application routes to direct, private connectivity.
--disable-scp-checks Indicates if cloud permission checks are disabled when attempting installation of the cluster.
--watch Watch cluster installation logs.
--dry-run Simulate creating the cluster.
-h, --help help for cluster
Global Flags:
--debug Enable debug mode.
-i, --interactive Enable interactive mode.
-v, --v Level log level for V logs
You can step through each of these options interactively by using the --interactive flag.
$ rosa create cluster --interactive
Otherwise, run the following command to create your cluster with the default cluster settings. The default settings are as follows:
- The AWS region you have configured for the AWS CLI
- The most recent version of OpenShift available to rosa
- A single availability zone
- Public cluster (Public API)
- Master nodes: 3
- Infra nodes: 2
- Compute nodes: 2 (m5.xlarge instance types)
$ rosa create cluster --cluster-name=rh-rosa-test
I: Creating cluster with identifier '1de87g7c30g75qechgh7l5b2bha6r04e' and name 'rh-rosa-test'
I: To view a list of clusters and their status, run `rosa list clusters`
I: Cluster 'rh-rosa-test' has been created.
I: Once the cluster is 'Ready' you will need to add an Identity Provider and define the list of cluster administrators. See `rosa create idp --help` and `rosa create user --help` for more information.
I: To determine when your cluster is Ready, run `rosa describe cluster rh-rosa-test`.
Creating a cluster can take up to 40 minutes, during which the State will transition from pending to installing, and finally to ready.
After creating a cluster, run the following command to list all available clusters:
$ rosa list clusters
ID NAME STATE
1eids212dg6tkkr231t1sl25reskq0q7 rh-rosa-test pending
Run the following command to see more details and check the status of a specific cluster. Replace <my-cluster-name> with the name of your cluster.
$ rosa describe cluster <my-cluster-name>
Name: rh-rosa-test
ID: 1de87g7c30g75qechgh7l5b2bha6r04e
External ID: 34322be7-b2a7-45c2-af39-2c684ce624e1
API URL: https://api.rh-rosa-test.j9n4.s1.devshift.org:6443
Console URL: https://console-openshift-console.apps.rh-rosa-test.j9n4.s1.devshift.org
Nodes: Master: 3, Infra: 2, Compute: 2
Region: us-east-2
State: ready
Created: May 27, 2020
If installation fails or the State does not change to ready after 40 minutes, check the installation troubleshooting documentation for more details.
You can follow the installer logs to track the progress of your cluster:
rosa logs install -c rh-rosa-test --watch
To log in to your cluster, you must configure an Identity Provider (IDP).
For this guide we will use GitHub as an example IDP.
For other supported IDPs, run rosa create idp --help, and consult the OpenShift documentation on configuring an IDP for more information.
The following command to creates an IDP backed by GitHub. Follow the interactive prompts from the output to access your Github developer settings and configure a new OAuth application.
Here are the options we will configure and the values to select:
- Type of identity provider: github
- Restrict to members of: organizations (if you do not have a GitHub Organization, you can create one now.)
- GitHub organizations: rh-test-org (enter the name of your org)
Follow the URL from the output. This will create a new OAuth application in the GitHub organization you specified. Click Register applicaton to access your Client ID and Client Secret.
- Client ID: <my-github-client-id>
- Client Secret: [? for help] <my-github-client-secret>
- Hostname: (optional, you can leave it blank for now)
- Mapping method: claim
$ rosa create idp --cluster=rh-rosa-test --interactive
I: Interactive mode enabled.
Any optional fields can be left empty and a default will be selected.
? Type of identity provider: github
? Restrict to members of: organizations
? GitHub organizations: rh-test-org
? To use GitHub as an identity provider, you must first register the application:
- Open the following URL:
https://github.com/organizations/rh-rosa-test/settings/applications/new?oauth_application%5Bcallback_url%5D=https%3A%2F%2Foauth-openshift.apps.rh-rosa-test.z7v0.s1.devshift.org%2Foauth2callback%2Fgithub-1&oauth_application%5Bname%5D=rh-rosa-test&oauth_application%5Burl%5D=https%3A%2F%2Fconsole-openshift-console.apps.rh-rosa-test.z7v0.s1.devshift.org
- Click on 'Register application'
? Client ID: <my-github-client-id&
? Client Secret: [? for help] <my-github-client-secret&
? Hostname:
? Mapping method: claim
I: Configuring IDP for cluster 'rh-rosa-test'
I: Identity Provider 'github-1' has been created. You need to ensure that there is a list of cluster administrators defined. See 'rosa create user --help' for more information. To login into the console, open https://console-openshift-console.apps.rh-test-org.z7v0.s1.devshift.org and click on github-1
The IDP can take 1-2 minutes to be configured within your cluster.
Run the following command to verify that your IDP has been configured correctly:
$ rosa list idps --cluster rh-rosa-test
NAME TYPE AUTH URL
github-1 GitHub https://oauth-openshift.apps.rh-rosa-test.j9n4.s1.devshift.org/oauth2callback/github-1
At this point you should be able to log in to your cluster. The follow examples continue to use GitHub as an example IDP.
First, run the following command to get the Console URL of your cluster:
$ rosa describe cluster rh-rosa-test
Name: rh-rosa-test
ID: 1de87g7c30g75qechgh7l5b2bha6r04e
External ID: 34322be7-b2a7-45c2-af39-2c684ce624e1
API URL: https://api.rh-rosa-test.j9n4.s1.devshift.org:6443
Console URL: https://console-openshift-console.apps.rh-rosa-test.j9n4.s1.devshift.org
Nodes: Master: 3, Infra: 2, Compute: 2
Region: us-east-2
State: ready
Created: May 27, 2020
Navigate to the Console URL and log in using your GitHub credentials (or the credentials for the IDP you added to your cluster).
Once you are logged into your cluster, follow these steps to get your oc login command. In the top right of the OpenShift console, click your name and click Copy Login Command. Click github-1 and finally click Display Token. Copy and paste the oc login command into your terminal.
$ oc login --token=z3sgOGVDk0k4vbqo_wFqBQQTnT-nA-nQLb8XEmWnw4X --server=https://api.rh-rosa-test.j9n4.s1.devshift.org:6443
Logged into "https://api.rh-rosa-test.j9n4.s1.devshift.org:6443" as "rh-rosa-test-user" using the token provided.
You have access to 67 projects, the list has been suppressed. You can list all projects with 'oc projects'
Using project "default".
Run a simple oc command to verify everything is setup properly and you are logged in.
$ oc version
Client Version: 4.4.0-202005231254-4a4cd75
Server Version: 4.3.18
Kubernetes Version: v1.16.2
Run the following command to promote your Github user to dedicated-admin:
$ rosa create user --cluster rh-rosa-test --dedicated-admins=rh-rosa-test-user
Run the following command to verify your user now has dedicated-admin access. As a dedicated-admin you should receive some errors when running the following command:
$ oc get all -n openshift-apiserver
NAME READY STATUS RESTARTS AGE
pod/apiserver-6ndg2 1/1 Running 0 17h
pod/apiserver-lrmxs 1/1 Running 0 17h
pod/apiserver-tsqhz 1/1 Running 0 17h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/api ClusterIP 172.30.23.241 <none> 443/TCP 17h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/apiserver 3 3 3 3 3 node-role.kubernetes.io/master= 17h
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "rh-rosa-test-user" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace
"openshift-apiserver"
Error from server (Forbidden): jobs.batch is forbidden: User "rh-rosa-test-user" cannot list resource "jobs" in API group "batch" in the namespace "openshift-apiserver"
Error from server (Forbidden): cronjobs.batch is forbidden: User "rh-rosa-test-user" cannot list resource "cronjobs" in API group "batch" in the namespace "openshift-apiserver"
Error from server (Forbidden): imagestreams.image.openshift.io is forbidden: User "rh-rosa-test-user" cannot list resource "imagestreams" in API group "image.openshift.io" in the namespace "openshift
-apiserver"
To add a cluster-admin user, first enable cluster-admin capability on the cluster:
$ rosa edit cluster rh-rosa-test --enable-cluster-admins
Next give your user cluster-admin privileges:
$ rosa create user --cluster rh-rosa-test --cluster-admins rh-rosa-test-user
$ rosa list users --cluster rh-rosa-test
GROUP NAME
cluster-admins rh-rosa-test-user
dedicated-admins rh-rosa-test-user
Run the following command to verify your user now has cluster-admin access. As a cluster-admin you should be able to run the following command without errors.
$ oc get all -n openshift-apiserver
NAME READY STATUS RESTARTS AGE
pod/apiserver-6ndg2 1/1 Running 0 17h
pod/apiserver-lrmxs 1/1 Running 0 17h
pod/apiserver-tsqhz 1/1 Running 0 17h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/api ClusterIP 172.30.23.241 <none> 443/TCP 18h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/apiserver 3 3 3 3 3 node-role.kubernetes.io/master= 18h
After installing your cluster you can move on to installing an example app, or clean up if you are just giving ROSA a test drive.
Run the following command to delete your cluster, replacing <my-cluster> with the name of your cluster:
rosa delete cluster -c <my-cluster>
Once the cluster is uninstalled, you can clean up your CloudFormation stack (this was created when you ran rosa init) by running the following command:
rosa init --delete-stack
If you'd like to build this project from source use the following steps:
- Checkout the repostiory into your
$GOPATH
go get -u github.com/openshift/rosa
cdto the checkout out source directory
cd $GOPATH/src/github.com/openshift/rosa
- Install the binary (This will install to
$GOPATH/bin)
make install
NOTE: If you don't have $GOPATH/bin in your $PATH you need to add it or move rosa to a standard system directory eg. for Linux/OSX:
sudo mv $GOPATH/bin/rosa /usr/local/bin
We want to hear it. Open and issue against the repo and someone from the team will be in touch.