Merge branch 'ebs_disks_naming_fix' into 'master'

Multiple terraform module enhancements

See merge request postgres-ai/terraform-postgres-ai-database-lab!43

Author: DmitryFomin1

README.md

@@ -1,6 +1,6 @@
 # Database Lab Terraform Module
 
-This [Terraform Module](https://www.terraform.io/docs/language/modules/index.html) is responsible for deploying the [Database Lab Engine](https://gitlab.com/postgres-ai/database-lab) to cloud hosting providers.
+This [Terraform Module](https://www.terraform.io/docs/language/modules/index.html) can be used as a template for deploying the [Database Lab Engine](https://gitlab.com/postgres-ai/database-lab) to cloud hosting providers. Please feel free to tailor it to meet your requirements. 
 
 Your source PostgreSQL database can be located anywhere, but DLE with other components will be created on an EC2 instance under your AWS account. Currently, only "logical" mode of data retrieval (dump/restore) is supported – the only available method for managed PostgreSQL cloud services such as RDS Postgres, RDS Aurora Postgres, Azure Postgres, or Heroku. "Physical" mode is not yet supported, but it will be in the future. More about various data retrieval options for DLE: https://postgres.ai/docs/how-to-guides/administration/data.
 
@@ -12,7 +12,7 @@ Your source PostgreSQL database can be located anywhere, but DLE with other comp
 - [Terraform Installed](https://learn.hashicorp.com/tutorials/terraform/install-cli) (minimal version: 1.0.0)
 - AWS [Route 53](https://aws.amazon.com/route53/) Hosted Zone (For setting up TLS) for a domain or sub-domain you control
 - You must have AWS Access Keys and a default region in your Terraform environment (See section on required IAM Permissions)
-- The DLE runs on an EC2 instance which can be accessed using a selected set of SSH keys uploaded to EC2. Use the Terraform parameter `aws_keypair` to specify which EC2 Keypair to use
+- The DLE runs on an EC2 instance which can be accessed using a selected set of SSH keys uploaded to EC2.
 - Required IAM Permissions: to successfully run this Terraform module, the IAM User/Role must have the following permissions:
     * Read/Write permissions on EC2
     * Read/Write permissions on Route53
@@ -49,16 +49,15 @@ The following steps were tested on Ubuntu 20.04 but supposed to be valid for oth
     ```
 1. Edit `terraform.tfvars` file. In our example, we will use Heroku demo database as a source:
     ```config
-    dle_version_full = "2.4.1"
+    dle_version_full = "2.5.0"
 
     aws_ami_name = "DBLABserver*"
-    aws_keypair = "YOUR_AWS_KEYPAIR"
 
     aws_deploy_region = "us-east-1"
     aws_deploy_ebs_availability_zone = "us-east-1a"
-    aws_deploy_ec2_instance_type = "t2.large"
+    aws_deploy_ec2_instance_type = "c5.large"
     aws_deploy_ec2_instance_tag_name = "DBLABserver-ec2instance"
-    aws_deploy_ebs_size = "40"
+    aws_deploy_ebs_size = "10"
     aws_deploy_ebs_type = "gp2"
     aws_deploy_allow_ssh_from_cidrs = ["0.0.0.0/0"]
     aws_deploy_dns_api_subdomain = "tf-test" # subdomain in aws.postgres.ai, fqdn will be ${dns_api_subdomain}-engine.aws.postgres
@@ -67,13 +66,14 @@ The following steps were tested on Ubuntu 20.04 but supposed to be valid for oth
     source_postgres_host = "ec2-3-215-57-87.compute-1.amazonaws.com"
     source_postgres_port = "5432"
     source_postgres_dbname = "d3dljqkrnopdvg" # this is an existing DB (Heroku example DB)
-    source_postgres_username = "postgres"
-
+    source_postgres_username = "bfxuriuhcfpftt" # in secret.tfvars, use:   source_postgres_password = "dfe01cbd809a71efbaecafec5311a36b439460ace161627e5973e278dfe960b7"
     dle_debug_mode = "true"
     dle_retrieval_refresh_timetable = "0 0 * * 0"
     postgres_config_shared_preload_libraries = "pg_stat_statements,logerrors" # DB Migration Checker requires logerrors extension
 
     platform_project_name = "aws_test_tf"
+
+    ssh_public_keys_files_list = ["~/.ssh/id_rsa.pub"]
     ```
 1. Create `secret.tfvars` containing `source_postgres_password`, `platform_access_token`, and `vcs_github_secret_token`. An example:
     ```config
@@ -106,7 +106,7 @@ The following steps were tested on Ubuntu 20.04 but supposed to be valid for oth
     public_dns_name = "demo-api-engine.aws.postgres.ai"  # todo: this should be URL, not hostname – further we'll need URL, with protocol – `https://`
     ```
 
-1. To verify result and check the progress, you might want to connect to the just-created EC2 machine using IP address or hostname from the Terraform output. In our example, it can be done using this one-liner (you can find more about DLE logs and configuration on this page: https://postgres.ai/docs/how-to-guides/administration/engine-manage):
+1. To verify result and check the progress, you might want to connect to the just-created EC2 machine using IP address or hostname from the Terraform output and ssh key from ssh_public_keys_files_list and/or ssh_public_keys_list variables. In our example, it can be done using this one-liner (you can find more about DLE logs and configuration on this page: https://postgres.ai/docs/how-to-guides/administration/engine-manage):
     ```shell
     echo "sudo docker logs dblab_server -f" | ssh ubuntu@18.118.126.25 -i postgres_ext_test.pem
     ```

dle-logical-init.sh.tpl

@@ -86,17 +86,31 @@ EOF
 sudo systemctl enable envoy
 sudo systemctl start envoy
 
-#create zfs pools
-disks=(${dle_disks})
-for i in $${!disks[@]}; do
+# create zfs pools
+# Get the full list of disks available and then make attempts
+# to create zpool on each. Here we assume that the system disk
+# will be skipped because it already has a filesystem.
+# This is a "brute force" approach that we probably want to
+# rework, but for now we leave it as is because it seems that 
+# `/dev/../by-id` doesn't really work for all EC2 types.
+
+disks=$(lsblk -ndp -e7 --output NAME)   # TODO: this is not needed, used now for debug only
+
+i=1
+
+sleep 10 # Not elegant at all, we need a better way to wait till the moment when all disks are available
+
+# Show all disks in alphabetic order; "-e7" to exclude loop devices
+for disk in $disks; do
   sudo zpool create -f \
-  -O compression=on \
-  -O atime=off \
-  -O recordsize=128k \
-  -O logbias=throughput \
-  -m /var/lib/dblab/dblab_pool_0$i\
-  dblab_pool_0$i \
-  $${disks[$i]}
+    -O compression=on \
+    -O atime=off \
+    -O recordsize=128k \
+    -O logbias=throughput \
+    -m /var/lib/dblab/dblab_pool_$(printf "%02d" $i)\
+    dblab_pool_$(printf "%02d" $i) \
+    $disk \
+    && ((i=i+1)) # increment if succeeded
 done
 
 # Adjust DLE config

terraform.tfvars

@@ -1,15 +1,15 @@
-dle_version = "2.5.0"
+dle_version = "2.5.0"  # it is also possible to use branch name here (e.g., "master")
 joe_version = "0.10.0"
 
 aws_ami_name = "DBLABserver*"
-aws_keypair = "YOUR_AWS_KEYPAIR"
 
 aws_deploy_region = "us-east-1"
 aws_deploy_ebs_availability_zone = "us-east-1a"
-aws_deploy_ec2_instance_type = "t2.large"
+aws_deploy_ec2_instance_type = "c5.large"
 aws_deploy_ec2_instance_tag_name = "DBLABserver-ec2instance"
-aws_deploy_ebs_size = "40"
+aws_deploy_ebs_size = "10"
 aws_deploy_ebs_type = "gp2"
+aws_deploy_ec2_volumes_names = ["/dev/xvdf", "/dev/xvdg",]
 aws_deploy_allow_ssh_from_cidrs = ["0.0.0.0/0"]
 aws_deploy_dns_api_subdomain = "tf-test" # subdomain in aws.postgres.ai, fqdn will be ${dns_api_subdomain}.aws.postgres.ai
 
@@ -24,3 +24,8 @@ dle_retrieval_refresh_timetable = "0 0 * * 0"
 postgres_config_shared_preload_libraries = "pg_stat_statements,logerrors" # DB Migration Checker requires logerrors extension
 
 platform_project_name = "aws_test_tf"
+
+# Edit this list to have all public keys that will be placed to 
+# have them placed to authorized_keys. Instead of ssh_public_keys_files_list,
+# it is possible to use ssh_public_keys_list containing public keys as text values.
+ssh_public_keys_files_list = ["~/.ssh/id_rsa.pub"]

variables.tf

@@ -27,11 +27,6 @@ variable "aws_deploy_ec2_instance_type" {
     default = "t2.micro"
 }
 
-variable "aws_keypair" {
-    description = "Key pair to access the EC2 instance"
-    default = "default"
-}
-
 variable "aws_deploy_allow_ssh_from_cidrs" {
     description = "List of CIDRs allowed to connect to SSH"
     default = ["0.0.0.0/0"]
@@ -67,6 +62,11 @@ variable "aws_deploy_ebs_availability_zone" {
    default = "us-east-1a"
 }
 
+variable "aws_deploy_ebs_encrypted" {
+   description = "If EBS volumes used by DLE are encrypted"
+   default = "true"
+}
+
 variable "aws_deploy_ebs_size" {
    description = "The size (GiB) for data volumes used by DLE"
    default = "1"
@@ -77,12 +77,17 @@ variable "aws_deploy_ebs_type" {
    default = "gp2"
 }
 
+# If we need to have more data disks, this array has to be extended.
+# TODO: change logic – user sets the number of disks only, not thinking about names
 variable "aws_deploy_ec2_volumes_names" {
   description = "List of paths for EBS volumes mounts"
+  # This list is of "non-nitro" instances. For "nitro" ones,
+  # the real disk names will be different and in fact these names 
+  # will be ignored. However, we still need to pass something here
+  # to proceed with the disk attachment.
   default = [
     "/dev/xvdf",
     "/dev/xvdg",
-    "/dev/xvdh",
   ]
 }
 

volumes.tf

@@ -9,6 +9,7 @@ resource "aws_volume_attachment" "ebs_att" {
 resource "aws_ebs_volume" "DLEVolume" {
   count = "${length(tolist(var.aws_deploy_ec2_volumes_names))}"
   availability_zone = "${var.aws_deploy_ebs_availability_zone}"
+  encrypted = "${var.aws_deploy_ebs_encrypted}"
   size  = "${var.aws_deploy_ebs_size}"
   type = "${var.aws_deploy_ebs_type}"
   tags = {