Skip to content

Commit 40ff91e

Browse files
DmitryFomin1DmitryFomin1
committed
Merge branch 'tf-destroy' into 'master'
Reminder to run terraform destroy See merge request postgres-ai/terraform-postgres-ai-database-lab!27
2 parents d28d530 + 29551da commit 40ff91e

File tree

1 file changed

+166
-1
lines changed

1 file changed

+166
-1
lines changed

‎README.md

Lines changed: 166 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,172 @@ Your source PostgreSQL database can be located anywhere, but DLE with other comp
77
## Supported Cloud Platforms
88
- AWS
99

10-
## Installation
10+
## Prerequisites
11+
- [AWS Account](https://aws.amazon.com)
12+
- [Terraform Installed](https://learn.hashicorp.com/tutorials/terraform/install-cli) (minimal version: 1.0.0)
13+
- AWS [Route 53](https://aws.amazon.com/route53/) Hosted Zone (For setting up TLS) for a domain or sub-domain you control
14+
- You must have AWS Access Keys and a default region in your Terraform environment (See section on required IAM Permissions)
15+
- The DLE runs on an EC2 instance which can be accessed using a selected set of SSH keys uploaded to EC2. Use the Terraform parameter `aws_keypair` to specify which EC2 Keypair to use
16+
- Required IAM Permissions: to successfully run this Terraform module, the IAM User/Role must have the following permissions:
17+
* Read/Write permissions on EC2
18+
* Read/Write permissions on Route53
19+
* Read/Write permissions on Cloudwatch
20+
21+
## How to use
22+
- :construction: Currently, it is supposed that you run `terraform` commands on a Linux machine. MacOS and Windows support is not yet implemented (but planned).
23+
- It is recommended to clone this Git repository and adjust for your needs. Below we provide the detailed step-by-step instructions for quick start (see "Quick start") for a PoC setup
24+
- To configure parameters used by Terraform (and the Database Lab Engine itself), you will need to modify `terraform.tfvars` and create a file with secrets (`secret.tfvars`)
25+
- This Terraform module can be run independently or combined with any other standard Terraform module. You can learn more about using Terraform and the Terraform CLI [here](https://www.terraform.io/docs/cli/commands/index.html)
26+
- The variables can be set in multiple ways with the following precedence order (lowest to highest):
27+
- default values in `variables.tf`
28+
- values defined in `terraform.tfvars`
29+
- values passed on the command line
30+
- All variables starting with `postgres_` represent the source database connection information for the data (from that database) to be fetched by the DLE. That database must be accessible from the instance hosting the DLE (that one created by Terraform)
31+
32+
## Quick start
33+
The following steps were tested on Ubuntu 20.04 but supposed to be valid for other Linux distributions without significant modification.
34+
35+
1. SSH to any machine with internet access, it will be used as deployment machine
36+
1. Install Terraform https://learn.hashicorp.com/tutorials/terraform/install-cli. Example for Ubuntu:
37+
```shell
38+
sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl
39+
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
40+
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" # Adjust if you have ARM platform.
41+
sudo apt-get update && sudo apt-get install terraform
42+
# Verify installation.
43+
terraform -help
44+
```
45+
1. Get TF code for Database Lab:
46+
```shell
47+
git clone https://gitlab.com/postgres-ai/database-lab-infrastructure.git
48+
cd database-lab-infrastructure/
49+
```
50+
1. Edit `terraform.tfvars` file. In our example, we will use Heroku demo database as a source:
51+
```config
52+
dle_version_full = "2.4.1"
53+
54+
aws_ami_name = "DBLABserver*"
55+
aws_keypair = "YOUR_AWS_KEYPAIR"
56+
57+
aws_deploy_region = "us-east-1"
58+
aws_deploy_ebs_availability_zone = "us-east-1a"
59+
aws_deploy_ec2_instance_type = "t2.large"
60+
aws_deploy_ec2_instance_tag_name = "DBLABserver-ec2instance"
61+
aws_deploy_ebs_size = "40"
62+
aws_deploy_ebs_type = "gp2"
63+
aws_deploy_allow_ssh_from_cidrs = ["0.0.0.0/0"]
64+
aws_deploy_dns_api_subdomain = "tf-test" # subdomain in aws.postgres.ai, fqdn will be ${dns_api_subdomain}-engine.aws.postgres
65+
66+
source_postgres_version = "13"
67+
source_postgres_host = "ec2-3-215-57-87.compute-1.amazonaws.com"
68+
source_postgres_port = "5432"
69+
source_postgres_dbname = "d3dljqkrnopdvg" # this is an existing DB (Heroku example DB)
70+
source_postgres_username = "postgres"
71+
72+
dle_debug_mode = "true"
73+
dle_retrieval_refresh_timetable = "0 0 * * 0"
74+
postgres_config_shared_preload_libraries = "pg_stat_statements,logerrors" # DB Migration Checker requires logerrors extension
75+
76+
platform_project_name = "aws_test_tf"
77+
```
78+
1. Create `secret.tfvars` containing `source_postgres_password`, `platform_access_token`, and `vcs_github_secret_token`. An example:
79+
```config
80+
source_postgres_password = "YOUR_DB_PASSWORD" # todo: put pwd for heroku example DB here
81+
platform_access_token = "YOUR_ACCESS_TOKEN" # to generate, open https://console.postgres.ai/, choose your organization,
82+
# then "Access tokens" in the left menu
83+
vcs_github_secret_token = "vcs_secret_token" # to generate, open https://github.com/settings/tokens/new
84+
```
85+
1. Initialize
86+
```shell
87+
terraform init
88+
```
89+
1. Set environment variables with AWS credentials:
90+
```shell
91+
export AWS_ACCESS_KEY_ID = "keyid" # todo: how to get it?
92+
export AWS_SECRET_ACCESS_KEY = "accesskey"
93+
```
94+
1. Deploy:
95+
```
96+
terraform apply -var-file="secret.tfvars" -auto-approve
97+
```
98+
1. If everything goes well, you should get an output like this:
99+
```config
100+
vcs_db_migration_checker_verification_token = "gsio7KmgaxECfJ80kUx2tUeIf4kEXZex"
101+
dle_verification_token = "zXPodd13LyQaKgVXGmSCeB8TUtnGNnIa"
102+
ec2_public_dns = "ec2-18-118-126-25.us-east-2.compute.amazonaws.com"
103+
ec2instance = "i-0b07738148950af25"
104+
ip = "18.118.126.25"
105+
platform_joe_signing_secret = "lG23qZbUh2kq0ULIBfW6TRwKzqGZu1aP"
106+
public_dns_name = "demo-api-engine.aws.postgres.ai" # todo: this should be URL, not hostname – further we'll need URL, with protocol – `https://`
107+
```
108+
109+
1. To verify result and check the progress, you might want to connect to the just-created EC2 machine using IP address or hostname from the Terraform output. In our example, it can be done using this one-liner (you can find more about DLE logs and configuration on this page: https://postgres.ai/docs/how-to-guides/administration/engine-manage):
110+
```shell
111+
echo "sudo docker logs dblab_server -f" | ssh ubuntu@18.118.126.25 -i postgres_ext_test.pem
112+
```
113+
114+
Once you see the message like:
115+
```
116+
2021/07/02 10:28:51 [INFO] Server started listening on :2345.
117+
```
118+
– it means that the DLE server started successfully and is waiting for you commands
119+
120+
1. Sign in to the [Postgres.ai Platform](https://console.postgres.ai/) and register your new DLE server:
121+
1. Go to `Database Lab > Instances` in the left menu
122+
1. Press the "Add instance" button
123+
1. `Project` – specify any name (this is how your DLE server will be named in the platform)
124+
1. `Verification token` – use the token generated above (`verification_token` value); do NOT press the "Generate" button here
125+
1. `URL` – use the value generated above // todo: not convenient, we need URL but reported was only hostname
126+
1. Press the "Verify URL" button to check the connectivity. Then press "Add". If everything is right, you should see the DLE page with green "OK" status:
127+
<img src="/uploads/8371e7f79de199aa017ff2df82b8f704/image.png" width="400" />
128+
1. Add Joe chatbot for efficient SQL optimization workflow:
129+
1. Go to the "SQL Optimization > Ask Joe" page using the left menu, click the "Add instance" button, specify the same project as you defined in the previous step
130+
1. `Signing secret` – use `platform_joe_signing_secret` from the Terraform output
131+
1. `URL` – use `public_dns_name` values from the Terraform output with port `444`; in our example, it's `https://demo-api-engine.aws.postgres.ai:444`
132+
1. Press "Verify URL" to check connectivity and then press "Add". You should see:
133+
<img src="/uploads/252e5f74cd324fc4df301bbf7c2bdd25/image.png" width="400" />
134+
135+
Now you can start using Joe chatbot for SQL execution plans troubleshooting and verification of optimization ideas. As a quick test, go to `SQL Optimization > Ask Joe` in the left menu, and enter `\dt+` command (a psql command to show the list of tables with sizes). You should see how Joe created a thin clone behind the scenes and immediately ran this psql command, presenting the result to you:
136+
<img src="/uploads/d9e9e1fdafb0ded3504691cec9018868/image.png" width="400" />
137+
138+
1. Set up [DB migration checker](https://postgres.ai/docs/db-migration-checker). Prepare a repository with your DB migrations(Flyway, Sqitch, Liquibase, etc.):
139+
1. Add secrets:
140+
- `DLMC_CI_ENDPOINT` - an endpoint of your Database Lab Migration Checker service – use `vcs_db_migration_checker_registration_url` from the Terraform output
141+
- `DLMC_VERIFICATION_TOKEN` - verification token for the Database Lab Migration Checker API – use `vcs_db_migration_checker_verification_token` from the Terraform output
142+
1. Configure a new workflow in the created repository (see an example of configuration: https://github.com/postgres-ai/green-zone/blob/master/.github/workflows/main.yml)
143+
- add a custom action: https://github.com/marketplace/actions/database-lab-realistic-db-testing-in-ci
144+
- provide input params for the action (the full list of available input params)
145+
- provide environment variables:
146+
- `DLMC_CI_ENDPOINT` - use a CI Checker endpoint from the repository secrets
147+
- `DLMC_VERIFICATION_TOKEN` - use a verification token from the repository secrets
148+
149+
1. Install and try the client CLI (`dblab`)
150+
1. Follow the [guide](https://postgres.ai/docs/how-to-guides/cli/cli-install-init) to install Database Lab CLI
151+
1. Initialize CLI:
152+
```shell
153+
dblab init --environment-id=<ANY NAME FOR ENVIRONMENT> --url=https://<public_dns_name> --token=<your_personal_token_from_postgres_ai_platform>
154+
```
155+
1. Try it:
156+
```shell
157+
dblab instance status
158+
```
159+
It should return the OK status:
160+
```json
161+
{
162+
"status": {
163+
"code": "OK",
164+
"message": "Instance is ready"
165+
},
166+
...
167+
}
168+
```
169+
170+
This is it!
171+
172+
If you need to remove everything created by this Terraform module, you can simply run `terraform destroy`. Do not forget to do it if you're just experimenting. Otherwise, if you leave infrastructure blocks running, they might significantly affect your AWS bill (depending on the EC2 instance family you've chosen, the disk type, and size).
173+
174+
## Important Note
175+
When the DLE creates new database clones, it makes them available on incremental ports in the 6000 range (e.g. 6000, 6001, ...). The DLE CLI will also report that the clone is available on a port in the 6000 range. However, please note that these are the ports when accessing the DLE from `localhost`. This Terraform module deploys [Envoy](https://www.envoyproxy.io/) to handle SSL termination and port forwarding to connect to DLE generated clones.
11176
12177
Follow the [how-to guide](https://postgres.ai/docs/how-to-guides/administration/install-database-lab-with-terraform) to install Database Lab with Terraform on AWS
13178

0 commit comments

Comments
 (0)