Skip to content

Commit

Permalink
Replace Classic DNS with CIS
Browse files Browse the repository at this point in the history 
Fixes #496

Signed-off-by: Yussuf Shaikh <yussuf.shaikh1@ibm.com>
  • Loading branch information
@yussufsh
yussufsh committed Nov 20, 2023
1 parent 27056b2 commit 70d6e4c
Show file tree
Hide file tree
Showing 7 changed files with 109 additions and 120 deletions.
2 changes: 1 addition & 1 deletion docs/quickstart.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ Once the deployment is completed successfully, you can safely delete the bootstr

#### Create API and Ingress DNS Records

Please skip this section if your `cluster_domain` is one of the online wildcard DNS domains: nip.io, xip.io and sslip.io.
Please skip this section if your `cluster_domain` is one of the online wildcard DNS domains: nip.io, xip.io and sslip.io. Also, not required when using `use_ibm_cloud_services = true`.

For all other domains, you can use one of the following options.

Expand Down
17 changes: 4 additions & 13 deletions docs/var.tfvars-doc.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -234,9 +234,9 @@ Note: Once fips_compliant set to true it will enable FIPS on the OCP cluster and

### Using IBM Cloud Services

You can use IBM Cloud classic DNS and VPC Load Balancer services for running the OCP cluster. When this feature is enabled the services called named (DNS) and haproxy (Load Balancer) will not be running on the bastion/helpernode.
You can use IBM Cloud Internet Services (CIS) and Load Balancer services on VPC for running the OCP cluster. When this feature is enabled the services called `named` (DNS) and `haproxy` (Load Balancer) will not be running on the bastion/helpernode.

Ensure you have setup [DirectLink](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-ordering-direct-link-connect) with IBM Cloud VPC over the private network in cloud instance. Also, ensure you have registered a [DNS](https://cloud.ibm.com/docs/dns?topic=dns-register-a-new-domain) domain and use it as given in `cluster_domain` variable.
Ensure you have setup [Cloud Connection](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-cloud-connections) or [DirectLink](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-ordering-direct-link-connect) with IBM Cloud VPC over the private network in cloud instance. Also, ensure you have registered a [DNS domain](https://cloud.ibm.com/docs/cis?topic=cis-about-ibm-cloud-internet-services-cis) and use it as given in `cluster_domain` variable.

**IMPORTANT**: This is an **experimental** feature at present. Please manually set variables `setup_snat = true` and `setup_squid_proxy = false` for using IBM Cloud services. This will allow the cluster nodes have public internet access without a proxy server.

Expand All @@ -246,19 +246,10 @@ Below variables needs to be set in order to use the IBM Cloud services.
use_ibm_cloud_services = true
ibm_cloud_vpc_name = "ocp-vpc"
ibm_cloud_vpc_subnet_name = "ocp-subnet"
iaas_vpc_region = "us-south" # the VPC region for accessing IBM Cloud services. If empty, will default to ibmcloud_region.
ibm_cloud_cis_crn = "crn:v1:bluemix:public:internet-svcs:global:a/<account_id>:<cis_instance_id>::" # CRN of the CIS instance where domain is registered.
```

These set of variables specify the username and API key for accessing IBM Cloud services. The default combination should suffice for most of the common use-cases.

```
iaas_classic_username = "apikey"
iaas_classic_api_key = "" # if empty, will default to ibmcloud_api_key.
iaas_vpc_region = "" # if empty, will default to ibmcloud_region.
```

Note: `iaas_classic_username`, `iaas_classic_api_key` and `iaas_vpc_region` variables are optional, These variables need to be set only when using a different classic username, key and vpc region. By default `apikey` will be used as the `iaas_class_username`, `ibmcloud_api_key` will be used as the `iaas_classic_api_key` and `ibmcloud_region` will be used as the `iaas_vpc_region`. Note that non-default values for these variables can also be passed via environment variables `IAAS_CLASSIC_USERNAME` and `IAAS_CLASSIC_API_KEY` respectively.


### Misc Customizations

These variables provides miscellaneous customizations. For common usage scenarios these are not required and should be left unchanged.
Expand Down
129 changes: 67 additions & 62 deletions modules/7_ibmcloud/dns.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,75 +22,80 @@
##### DNS
#####################################

data "ibm_dns_domain" "domain" {
name = var.cluster_domain
data "ibm_cis_domain" "domain" {
cis_id = var.ibm_cloud_cis_crn
domain = var.cluster_domain
}

resource "ibm_dns_record" "bastion" {
count = var.bastion_count
data = var.bastion_ip[count.index]
domain_id = data.ibm_dns_domain.domain.id
host = "${var.name_prefix}bastion-${count.index}.${var.cluster_id}"
responsible_person = "root.${var.cluster_domain}."
ttl = 900
type = "a"
resource "ibm_cis_dns_record" "bastion" {
count = var.bastion_count
cis_id = var.ibm_cloud_cis_crn
content = var.bastion_ip[count.index]
domain_id = data.ibm_cis_domain.domain.id
name = "${var.name_prefix}bastion-${count.index}.${var.cluster_id}.${var.cluster_domain}"
ttl = 900
type = "A"
}
resource "ibm_dns_record" "registry" {
data = var.bastion_vip != "" ? var.bastion_vip : var.bastion_ip[0]
domain_id = data.ibm_dns_domain.domain.id
host = "registry.${var.cluster_id}"
responsible_person = "root.${var.cluster_domain}."
ttl = 900
type = "a"
resource "ibm_cis_dns_record" "registry" {
cis_id = var.ibm_cloud_cis_crn
content = var.bastion_vip != "" ? var.bastion_vip : var.bastion_ip[0]
domain_id = data.ibm_cis_domain.domain.id
name = "registry.${var.cluster_id}.${var.cluster_domain}"
ttl = 900
type = "A"
}
resource "ibm_dns_record" "bootstrap" {
count = var.bootstrap_count
data = var.bootstrap_ip
domain_id = data.ibm_dns_domain.domain.id
host = "${var.node_prefix}bootstrap.${var.cluster_id}"
responsible_person = "root.${var.cluster_domain}."
ttl = 900
type = "a"
resource "ibm_cis_dns_record" "bootstrap" {
count = var.bootstrap_count
cis_id = var.ibm_cloud_cis_crn
content = var.bootstrap_ip
domain_id = data.ibm_cis_domain.domain.id
name = "${var.node_prefix}bootstrap.${var.cluster_id}.${var.cluster_domain}"
ttl = 900
type = "A"
}
resource "ibm_dns_record" "master" {
count = var.master_count
data = var.master_ips[count.index]
domain_id = data.ibm_dns_domain.domain.id
host = "${var.node_prefix}master-${count.index}.${var.cluster_id}"
responsible_person = "root.${var.cluster_domain}."
ttl = 900
type = "a"
resource "ibm_cis_dns_record" "master" {
count = var.master_count
cis_id = var.ibm_cloud_cis_crn
content = var.master_ips[count.index]
domain_id = data.ibm_cis_domain.domain.id
name = "${var.node_prefix}master-${count.index}.${var.cluster_id}.${var.cluster_domain}"
ttl = 900
type = "A"
}
resource "ibm_dns_record" "worker" {
count = var.worker_count
data = var.worker_ips[count.index]
domain_id = data.ibm_dns_domain.domain.id
host = "${var.node_prefix}worker-${count.index}.${var.cluster_id}"
responsible_person = "root.${var.cluster_domain}."
ttl = 900
type = "a"
resource "ibm_cis_dns_record" "worker" {
count = var.worker_count
cis_id = var.ibm_cloud_cis_crn
content = var.worker_ips[count.index]
domain_id = data.ibm_cis_domain.domain.id
name = "${var.node_prefix}worker-${count.index}.${var.cluster_id}.${var.cluster_domain}"
ttl = 900
type = "A"
}
resource "ibm_dns_record" "api" {
data = "${ibm_is_lb.load_balancer_external.hostname}."
domain_id = data.ibm_dns_domain.domain.id
host = "api.${var.cluster_id}"
responsible_person = "root.${var.cluster_domain}."
ttl = 900
type = "cname"

#####################################
##### Kubernetes
#####################################
resource "ibm_cis_dns_record" "api" {
cis_id = var.ibm_cloud_cis_crn
content = ibm_is_lb.load_balancer_external.hostname
domain_id = data.ibm_cis_domain.domain.id
name = "api.${var.cluster_id}.${var.cluster_domain}"
ttl = 900
type = "CNAME"
}
resource "ibm_dns_record" "api-int" {
data = "${ibm_is_lb.load_balancer_internal.hostname}."
domain_id = data.ibm_dns_domain.domain.id
host = "api-int.${var.cluster_id}"
responsible_person = "root.${var.cluster_domain}."
ttl = 900
type = "cname"
resource "ibm_cis_dns_record" "api-int" {
cis_id = var.ibm_cloud_cis_crn
content = ibm_is_lb.load_balancer_internal.hostname
domain_id = data.ibm_cis_domain.domain.id
name = "api-int.${var.cluster_id}.${var.cluster_domain}"
ttl = 900
type = "CNAME"
}
resource "ibm_dns_record" "apps" {
data = "${ibm_is_lb.load_balancer_external.hostname}."
domain_id = data.ibm_dns_domain.domain.id
host = "*.apps.${var.cluster_id}"
responsible_person = "root.${var.cluster_domain}."
ttl = 900
type = "cname"
resource "ibm_cis_dns_record" "apps" {
cis_id = var.ibm_cloud_cis_crn
content = ibm_is_lb.load_balancer_external.hostname
domain_id = data.ibm_cis_domain.domain.id
name = "*.apps.${var.cluster_id}.${var.cluster_domain}"
ttl = 900
type = "CNAME"
}
1 change: 1 addition & 0 deletions modules/7_ibmcloud/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ variable "node_prefix" {}

variable "vpc_name" {}
variable "vpc_subnet_id" {}
variable "ibm_cloud_cis_crn" {}

variable "bastion_count" {}
variable "bootstrap_count" {}
Expand Down
43 changes: 21 additions & 22 deletions ocp.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,9 @@ provider "ibm" {
}

provider "ibm" {
alias = "classic"
region = local.iaas_vpc_region
ibmcloud_api_key = var.ibmcloud_api_key
iaas_classic_username = var.iaas_classic_username
iaas_classic_api_key = local.iaas_classic_api_key
alias = "vpc"
region = local.iaas_vpc_region
ibmcloud_api_key = var.ibmcloud_api_key
}

resource "random_id" "label" {
Expand Down Expand Up @@ -38,7 +36,7 @@ locals {
}

data "ibm_is_subnet" "vpc_subnet" {
provider = ibm.classic
provider = ibm.vpc
count = var.use_ibm_cloud_services ? 1 : 0
name = var.ibm_cloud_vpc_subnet_name
}
Expand Down Expand Up @@ -209,22 +207,23 @@ module "ibmcloud" {
count = var.use_ibm_cloud_services ? 1 : 0
source = "./modules/7_ibmcloud"
providers = {
ibm = ibm.classic
ibm = ibm.vpc
}

cluster_domain = module.nodes.cluster_domain
cluster_id = local.cluster_id
name_prefix = local.name_prefix
node_prefix = local.node_prefix
bastion_count = lookup(var.bastion, "count", 1)
bootstrap_count = var.bootstrap["count"]
master_count = var.master["count"]
worker_count = var.worker["count"]
bastion_vip = module.prepare.bastion_vip
bastion_ip = module.prepare.bastion_ip
bootstrap_ip = module.nodes.bootstrap_ip
master_ips = module.nodes.master_ips
worker_ips = module.nodes.worker_ips
vpc_name = var.ibm_cloud_vpc_name
vpc_subnet_id = var.use_ibm_cloud_services ? data.ibm_is_subnet.vpc_subnet[0].id : ""
cluster_domain = module.nodes.cluster_domain
cluster_id = local.cluster_id
name_prefix = local.name_prefix
node_prefix = local.node_prefix
bastion_count = lookup(var.bastion, "count", 1)
bootstrap_count = var.bootstrap["count"]
master_count = var.master["count"]
worker_count = var.worker["count"]
bastion_vip = module.prepare.bastion_vip
bastion_ip = module.prepare.bastion_ip
bootstrap_ip = module.nodes.bootstrap_ip
master_ips = module.nodes.master_ips
worker_ips = module.nodes.worker_ips
vpc_name = var.ibm_cloud_vpc_name
vpc_subnet_id = var.use_ibm_cloud_services ? data.ibm_is_subnet.vpc_subnet[0].id : ""
ibm_cloud_cis_crn = var.ibm_cloud_cis_crn
}
5 changes: 2 additions & 3 deletions var.tfvars
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,8 @@ use_zone_info_for_names = true # If set it to false, the zone info would not be
#use_ibm_cloud_services = true
#ibm_cloud_vpc_name = "ocp-vpc"
#ibm_cloud_vpc_subnet_name = "ocp-subnet"
#iaas_classic_username = "apikey" # Can be passed via environment variable IAAS_CLASSIC_USERNAME
#iaas_classic_api_key = "" # if empty, will default to ibmcloud_api_key. Can be passed via environment variable IAAS_CLASSIC_API_KEY
#iaas_vpc_region = "" # if empty, will default to ibmcloud_region.
# iaas_vpc_region = "" # if empty, will default to ibmcloud_region.
# ibm_cloud_cis_crn = "crn:v1:bluemix:public:internet-svcs:global:a/<account_id>:<cis_instance_id>::"


### Misc Customizations
Expand Down
32 changes: 13 additions & 19 deletions variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ variable "rhel_smt" {
################################################################
variable "use_ibm_cloud_services" {
type = bool
description = "Experimental: Flag to use IBM Cloud DNS and VPC Loadbalancer instead of bastion services. Please set variables setup_snat=true and setup_squid_proxy=false"
description = "Experimental: Flag to use Internet Services (CIS) and Loadbalancer services on VPC instead of bastion services. Please set variables setup_snat=true and setup_squid_proxy=false"
default = false
}
variable "ibm_cloud_vpc_name" {
Expand All @@ -232,23 +232,18 @@ variable "ibm_cloud_vpc_subnet_name" {
description = "Name of the VPC subnet having DirectLink access to the private network. Required if use_ibm_cloud_services = true."
default = "ocp-subnet"
}
variable "iaas_classic_username" {
type = string
description = "IBM Cloud Classic Infrastructure user name (Hint: <account_id>_<email>). User should have access to update the DNS forward zones. Uses IAAS_CLASSIC_USERNAME envrionment variable if not provided. Required if use_ibm_cloud_services = true."
default = "apikey"
}
variable "iaas_classic_api_key" {
type = string
description = "IBM Cloud Classic Infrastructure API key. Uses IAAS_CLASSIC_API_KEY envrionment variable if not provided. Required if use_ibm_cloud_services = true."
default = ""
# if empty, will default to ibmcloud_api_key
}
variable "iaas_vpc_region" {
type = string
description = "IBM Cloud VPC Infrastructure region."
default = ""
# if empty, will default to ibmcloud_region
}
variable "ibm_cloud_cis_crn" {
# cli: `ibmcloud resource service-instance <cis name>`
type = string
description = "IBM Cloud Intenet Service instance CRN. Required if use_ibm_cloud_services = true."
default = ""
}

################################################################
### Instrumentation
Expand Down Expand Up @@ -409,12 +404,11 @@ variable "proxy" {
}

locals {
private_key_file = var.private_key_file == "" ? "${path.cwd}/data/id_rsa" : var.private_key_file
public_key_file = var.public_key_file == "" ? "${path.cwd}/data/id_rsa.pub" : var.public_key_file
private_key = var.private_key == "" ? file(coalesce(local.private_key_file, "/dev/null")) : var.private_key
public_key = var.public_key == "" ? file(coalesce(local.public_key_file, "/dev/null")) : var.public_key
iaas_classic_api_key = var.iaas_classic_api_key == "" ? var.ibmcloud_api_key : var.iaas_classic_api_key
iaas_vpc_region = var.iaas_vpc_region == "" ? var.ibmcloud_region : var.iaas_vpc_region
private_key_file = var.private_key_file == "" ? "${path.cwd}/data/id_rsa" : var.private_key_file
public_key_file = var.public_key_file == "" ? "${path.cwd}/data/id_rsa.pub" : var.public_key_file
private_key = var.private_key == "" ? file(coalesce(local.private_key_file, "/dev/null")) : var.private_key
public_key = var.public_key == "" ? file(coalesce(local.public_key_file, "/dev/null")) : var.public_key
iaas_vpc_region = var.iaas_vpc_region == "" ? var.ibmcloud_region : var.iaas_vpc_region
}

################################################################
Expand All @@ -439,7 +433,7 @@ variable "release_image_override" {
variable "cluster_domain" {
type = string
default = "ibm.com"
description = "Domain name to use to setup the cluster. A DNS Forward Zone should be a registered in IBM Cloud if use_ibm_cloud_services = true"
description = "Domain name to use to setup the cluster. A CIS Domain should be a registered in IBM Cloud if use_ibm_cloud_services = true"

validation {
condition = can(regex("^[a-z0-9]+[a-zA-Z0-9_\\-.]*[a-z0-9]+$", var.cluster_domain))
Expand Down

0 comments on commit 70d6e4c

Please sign in to comment.