Kubernetes Cluster in Google Kubernetes Engine (GKE)

• Kubernetes Cluster in Google Kubernetes Engine (GKE)
  • Features
  • TL;TR
  • Before you begin
    • GCE configuration
  • GCS remote state storage for Terraform
  • Terraform initialization
  • Apply Terraform plan
  • Troubleshooting
  • Cluster authentication
  • Post install
    • Helm installation

Terraform configuration for deploying a Kubernetes cluster in the Google Kubernetes Engine (GKE) in the Google Cloud Platform (GCP).

Features

• Public cluster on GKE.
• Ability to use preemptible VM instances for cluster nodes. Note that you need to have at least 3 nodes (throughout all zones) to minimize cluster downtime.

TL;TR

gcloud auth login
gcloud config set account $ACCOUNT

gcloud projects create $PROJECT_ID [--name=$NAME] [--organization=$ORGANIZATION_ID]
gcloud alpha billing accounts list
gcloud alpha billing projects link $PROJECT_ID --billing-account $BILLING_ACCOUNT_ID
gcloud config set project $PROJECT_ID
gcloud services enable compute.googleapis.com
gcloud services enable container.googleapis.com
gcloud services enable storage-component.googleapis.com

gcloud iam service-accounts create terraform-sa --display-name "Terraform Service Account"
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:terraform-sa@$PROJECT_ID.iam.gserviceaccount.com --role roles/editor
gcloud iam service-accounts keys create ~/key.json --iam-account terraform-sa@$PROJECT_ID.iam.gserviceaccount.com

gsutil mb -l us-central1 gs://terraform-state-storage/

export GOOGLE_APPLICATION_CREDENTIALS="~/key.json"

terraform init -backend-config "bucket=terraform-state-storage" -backend-config "prefix=cluster/example"
terraform apply

gcloud container clusters get-credentials $CLUSTER_NAME

helm init
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p "{\"spec\":{\"template\":{\"spec\":{\"serviceAccount\":\"tiller\"}}}}"

Before you begin

The following prerequisites need to be installed and configured:

• Terraform
• Google Cloud SDK (run gcloud components update to update SDK to the latest version if you have it already installed)

GCE configuration

Make sure you are logged in to a correct Google account. To list all available accounts, run:

gcloud auth list

To login to a new account, run:

gcloud auth login

To set the active account, run:

gcloud config set account $ACCOUNT

• $ACCOUNT should be replaced with the account's e-mail.

Optionally create a new GCS project for your deployment:

gcloud projects create $PROJECT_ID [--name=$NAME] [--organization=$ORGANIZATION_ID]

• $PROJECT_ID should be replaced with the ID of the project to create.
• $NAME is an optional, and should be replaced with the name of the project to create.
• $ORGANIZATION_ID is an optional, and should be replaced with the ID of your organization.

Run the following command to check whether the project succesfully created:

gcloud projects list

In order to be able to use Compute Engine and/or Kubernetes Engine, you need to enable billing for a new project either via Google Cloud Console, or using the following command:

gcloud alpha billing projects link $PROJECT_ID --billing-account $BILLING_ACCOUNT_ID

• $PROJECT_ID should be replaced with the ID of the project you created.
• $BILLING_ACCOUNT_ID should be replaced with the ID of the billing account to link to the project.

Run following command to list all your billing accounts:

gcloud alpha billing accounts list

NOTE To be able to run gcloud alpha command you need to have gcloud Alpha Commands component installed. Use gcloud components list to list all available components. gcloud components install alpha to install Alpha Commands component.

Set the project you created as an active:

gcloud config set project $PROJECT_ID

• $PROJECT_ID should be replaced with the ID of the project you've created.

Enable Compute, Kubernetes, and Cloud Storage engines for the project:

gcloud services enable compute.googleapis.com
gcloud services enable container.googleapis.com
gcloud services enable storage-component.googleapis.com

Create a Service Account for Terraform, and grant it the Editor role.

gcloud iam service-accounts create terraform-sa --display-name "Terraform Service Account"
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:terraform-sa@$PROJECT_ID.iam.gserviceaccount.com --role roles/editor

Create service account keys:

gcloud iam service-accounts keys create ~/key.json --iam-account terraform-sa@[$PROJECT_ID].iam.gserviceaccount.com

• $PROJECT_ID should be replaced with the ID of the project you've created.

Set the value of GOOGLE_APPLICATION_CREDENTIALS environment variable to a path of the generated key file.

For example (Windows PowerShell):

$env:GOOGLE_APPLICATION_CREDENTIALS = "~/key.json"

GCS remote state storage for Terraform

Create GCS bucket for storing the Terraform state in a central remote location:

gsutil mb -l $REGION gs://$BUCKET_NAME/

• $REGION should be replaced with a region name, for example us-central1. Refer to documentation for more information.
• $BUCKET_NAME should be replaced with a globally unique bucket name.

Terraform initialization

Copy terraform.tfvars.example file to terraform.tfvars and set input variables values as per your needs. Then initialize Terraform with init command:

terraform init -backend-config "bucket=$BUCKET_NAME" -backend-config "prefix=cluster/$CLUSTER_NAME" -backend-config "region=$REGION"

• $REGION should be replaced with a region name.
• $CLUSTER_NAME should be replaced with the name of a cluster.
• $BUCKET_NAME should be replaced with a GCS Terraform state storage bucket name.

Apply Terraform plan

To apply Terraform plan, run:

terraform apply

Troubleshooting

At the time of writing, Kubernetes Terraform provider has a problem of creating new StorageClass resource. There is no way to specify which cluster this resource should go in, and it gets created in the default cluster. Therefore you may get the following error message:

Error: storageclasses.storage.k8s.io "fast" already exists

  on main.tf line 171, in resource "kubernetes_storage_class" "fast":
 171: resource "kubernetes_storage_class" "fast" {

In this case you need to switch to the newly created cluster first:

kubectl config use-context $CLUSTER_NAME

and run terraform apply again.

Cluster authentication

To authenticate to the newly created cluster, run:

gcloud container clusters get-credentials $CLUSTER_NAME

• $CLUSTER_NAME should be replaced with a name of the cluster.

To view general cluster information, run:

kubectl cluster-info

Post install

Helm installation

Install Helm to the Kubernetes cluster:

helm init

Create service account and grant admin role to Tiller (Helm server component):

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p "{\"spec\":{\"template\":{\"spec\":{\"serviceAccount\":\"tiller\"}}}}"

---

Happy Kuberneting!