Refactor cloud build deploy instructions

0-bootstrap/.gitignore

@@ -28,6 +28,9 @@ Session.vim
 *.tfstate
 *.tfstate.*
 
+# tf lock file
+**/.terraform.lock.hcl
+
 # Crash log files
 crash.log
 

0-bootstrap/README-Jenkins.md

@@ -345,7 +345,7 @@ Here you will configure a VPN Network tunnel to enable connectivity between the
 1. Check if your organization already has an Access Context Manager Policy.
 
    ```bash
-   export ORGANIZATION_ID=$(terraform -chdir="../0-bootstrap/" output -json common_config | jq '.org_id' | tr -d '"')
+   export ORGANIZATION_ID=$(terraform -chdir="../0-bootstrap/" output -json common_config | jq '.org_id' --raw-output)
    export ACCESS_CONTEXT_MANAGER_ID=$(gcloud access-context-manager policies list --organization ${ORGANIZATION_ID} --format="value(name)")
    echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}"
    ```
@@ -559,7 +559,7 @@ Here you will configure a VPN Network tunnel to enable connectivity between the
 1. Use `terraform output` to get the backend bucket and networks step Terraform Service Account values from 0-bootstrap output.
 
    ```bash
-   export ORGANIZATION_ID=$(terraform -chdir="../0-bootstrap/" output -json common_config | jq '.org_id' | tr -d '"')
+   export ORGANIZATION_ID=$(terraform -chdir="../0-bootstrap/" output -json common_config | jq '.org_id' --raw-output)
    export ACCESS_CONTEXT_MANAGER_ID=$(gcloud access-context-manager policies list --organization ${ORGANIZATION_ID} --format="value(name)")
    echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}"
    sed -i "s/ACCESS_CONTEXT_MANAGER_ID/${ACCESS_CONTEXT_MANAGER_ID}/" ./access_context.auto.tfvars
@@ -716,7 +716,7 @@ Here you will configure a VPN Network tunnel to enable connectivity between the
 1. Use `terraform output` to get the backend bucket and networks step Terraform Service Account values from 0-bootstrap output.
 
    ```bash
-   export ORGANIZATION_ID=$(terraform -chdir="../0-bootstrap/" output -json common_config | jq '.org_id' | tr -d '"')
+   export ORGANIZATION_ID=$(terraform -chdir="../0-bootstrap/" output -json common_config | jq '.org_id' --raw-output)
    export ACCESS_CONTEXT_MANAGER_ID=$(gcloud access-context-manager policies list --organization ${ORGANIZATION_ID} --format="value(name)")
    echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}"
    sed -i "s/ACCESS_CONTEXT_MANAGER_ID/${ACCESS_CONTEXT_MANAGER_ID}/" ./access_context.auto.tfvars

0-bootstrap/README.md

@@ -80,27 +80,18 @@ Also make sure that you've done the following:
 1. Added the user who will use Terraform to the `group_org_admins` group.
    They must be in this group, or they won't have
    `roles/resourcemanager.projectCreator` access.
-1. For the user who will run the procedures in this document, granted the
-   following roles:
-   -  The `roles/resourcemanager.organizationAdmin` role on the Google Cloud organization.
-   -  The `roles/orgpolicy.policyAdmin` role on the Google Cloud organization.
-   -  The `roles/billing.admin` role on the billing account.
-   -  The `roles/resourcemanager.folderCreator` role.
+1. For the user who will run the procedures in this document, granted the following roles:
+   - The `roles/resourcemanager.organizationAdmin` role on the Google Cloud organization.
+   - The `roles/orgpolicy.policyAdmin` role on the Google Cloud organization.
+   - The `roles/billing.admin` role on the billing account.
+   - The `roles/resourcemanager.folderCreator` role.
 
 If other users need to be able to run these procedures, add them to the group
 represented by the `org_project_creators` variable.
 For more information about the permissions that are required, and the resources
 that are created, see the organization bootstrap module
 [documentation.](https://github.com/terraform-google-modules/terraform-google-bootstrap)
 
-Use the helper script [validate-requirements.sh](../scripts/validate-requirements.sh) to validate your environment:
-
-```shell
-./scripts/validate-requirements.sh -o <ORGANIZATION_ID> -b <BILLING_ACCOUNT_ID> -u <END_USER_EMAIL>
-```
-
-**Note:** The script is not able to validate if the user is in a Cloud Identity or Google Workspace group with the required roles.
-
 ### Optional - Automatic creation of Google Cloud Identity groups
 
 Google Cloud Identity groups are used for [authentication and access management](https://cloud.google.com/architecture/security-foundations/authentication-authorization) in the foundation.
@@ -132,52 +123,114 @@ your current Jenkins manager (controller) environment.
 
 1. Go to the `0-bootstrap` folder.
 1. Rename `terraform.example.tfvars` to `terraform.tfvars` and update the file with values from your environment:
-    ```
-    mv terraform.example.tfvars terraform.tfvars
-    ```
-1. Run `terraform init`.
-1. Run `terraform plan` and review the output.
+
+   ```bash
+   mv terraform.example.tfvars terraform.tfvars
+   ```
+
+1. Use the helper script [validate-requirements.sh](../scripts/validate-requirements.sh) to validate your environment:
+
+   ```bash
+   ../scripts/validate-requirements.sh -o <ORGANIZATION_ID> -b <BILLING_ACCOUNT_ID> -u <END_USER_EMAIL>
+   ```
+
+   **Note:** The script is not able to validate if the user is in a Cloud Identity or Google Workspace group with the required roles.
+
+1. Run `terraform init` and `terraform plan` and review the output.
+
+   ```bash
+   terraform init
+   terraform plan -input=false -out bootstrap.tfplan
+   ```
+
 1. To run `gcloud beta terraform vet` steps please follow the [instructions](https://cloud.google.com/docs/terraform/policy-validation/validate-policies#install) to install the terraform-tools component.
-    1. Run `terraform plan -input=false -out bootstrap.tfplan`
-    1. Run `terraform show -json bootstrap.tfplan > bootstrap.json`
-    1. Run `gcloud beta terraform vet bootstrap.json --policy-library="../policy-library" --project <A-VALID-PROJECT-ID>` and check for violations (`<A-VALID-PROJECT-ID>` must be an existing project you have access to, this is necessary because Terraform-validator needs to link resources to a valid Google Cloud Platform project).
+1. Run the following commands and check for violations:
+
+   ```bash
+   export VET_PROJECT_ID=A-VALID-PROJECT-ID
+   terraform show -json bootstrap.tfplan > bootstrap.json
+   gcloud beta terraform vet bootstrap.json --policy-library="../policy-library" --project ${VET_PROJECT_ID}
+   ```
+
+   - *`A-VALID-PROJECT-ID`* must be an existing project you have access to, this is necessary because Terraform-validator needs to link resources to a valid Google Cloud Platform project.
 1. Run `terraform apply`.
-1. Run `terraform output organization_step_terraform_service_account_email` to get the email address of the admin of step `1-org`. You need this address in a later procedure.
-1. Run `terraform output environment_step_terraform_service_account_email` to get the email address of the admin of step `2-environments`. You need this address in a later procedure.
-1. Run `terraform output networks_step_terraform_service_account_email` to get the email address of the admin of steps `3-networks-dual-svpc` and `3-networks-hub-and-spoke`. You need this address in a later procedure.
-1. Run `terraform output projects_step_terraform_service_account_email` to get the email address of the admin of step `4-projects`. You need this address in a later procedure.
-1. Run `terraform output cloudbuild_project_id` to get the ID of your Cloud Build project.
-1. Run `terraform output gcs_bucket_tfstate` to get your Google Cloud bucket name from Terraform's state.
-1. Run `terraform output projects_gcs_bucket_tfstate` to get the Google Cloud bucket name from stage 4-projects Terraform's state.
-1. Copy the backend:
+
+   ```bash
+   terraform apply bootstrap.tfplan
+   ```
+
+1. Run `terraform output` to get the email address of the terraform service accounts that will be used to run manual steps for `shared` environments in steps `3-networks-dual-svpc`, `3-networks-hub-and-spoke`, and `4-projects` and the state bucket that will be used by step 4-projects.
+
+   ```bash
+   export network_step_sa=$(terraform output -raw networks_step_terraform_service_account_email)
+   export projects_step_sa=$(terraform output -raw projects_step_terraform_service_account_email)
+   export projects_gcs_bucket_tfstate=$(terraform output -raw projects_gcs_bucket_tfstate)
+
+   echo "network step service account = ${network_step_sa}"
+   echo "projects step service account = ${projects_step_sa}"
+   echo "projects gcs bucket tfstate = ${projects_gcs_bucket_tfstate}"
    ```
+
+1. Run `terraform output` to get the ID of your Cloud Build project:
+
+   ```bash
+   export cloudbuild_project_id=$(terraform output -raw cloudbuild_project_id)
+   echo "cloud build project ID = ${cloudbuild_project_id}"
+   ```
+
+1. Copy the backend and update `backend.tf` with the name of your Google Cloud bucket for Terraform's state. Also update the `backend.tf` of all steps.
+
+   ```bash
+   export backend_bucket=$(terraform output -raw gcs_bucket_tfstate)
+   echo "backend_bucket = ${backend_bucket}"
+
+   export backend_bucket_projects=$(terraform output -raw projects_gcs_bucket_tfstate)
+   echo "backend_bucket_projects = ${backend_bucket_projects}"
+
    cp backend.tf.example backend.tf
+   cd ..
+
+   for i in `find -name 'backend.tf'`; do sed -i "s/UPDATE_ME/${backend_bucket}/" $i; done
+   for i in `find -name 'backend.tf'`; do sed -i "s/UPDATE_PROJECTS_BACKEND/${backend_bucket_projects}/" $i; done
+
+   cd 0-bootstrap
    ```
-1. Update `backend.tf` with the name of your Cloud Storage bucket.
-1. Re-run `terraform init`. When you're prompted, agree to copy state to
-   Cloud Storage.
-1. (Optional) Run `terraform apply` to verify that state is configured
-   correctly. You should see no changes from the previous state.
+
+1. Re-run `terraform init`. When you're prompted, agree to copy Terraform state to Cloud Storage.
+
+   ```bash
+   terraform init
+   ```
+
+1. (Optional) Run `terraform plan` to verify that state is configured correctly. You should see no changes from the previous state.
+1. Save `0-bootstrap` Terraform configuration to `gcp-bootstrap` source repository:
+
+   ```bash
+   cd ../..
+
+   gcloud source repos clone gcp-bootstrap --project=${cloudbuild_project_id}
+   cd gcp-bootstrap
+   git checkout -b production
+   cp -RT ../terraform-example-foundation/0-bootstrap/ .
+
+   git add .
+   git commit -m 'Initialize bootstrap'
+   git push --set-upstream origin production
+   ```
+
+1. You can now move to the instructions in the [1-org](../1-org/README.md) step.
 
 **Note 1:** The stages after `0-bootstrap` use `terraform_remote_state` data source to read common configuration like the organization ID from the output of the `0-bootstrap` stage. They will [fail](../docs/TROUBLESHOOTING.md#error-unsupported-attribute) if the state is not copied to the Cloud Storage bucket.
 
-**Note 2:** After the deploy, even if you did not receive the project quota error described in the [Troubleshooting guide](../docs/TROUBLESHOOTING.md#project-quota-exceeded), we recommend that you request 50 additional projects for the four service accounts created in this step.
+**Note 2:** After the deploy, even if you did not receive the project quota error described in the [Troubleshooting guide](../docs/TROUBLESHOOTING.md#project-quota-exceeded), we recommend that you request 50 additional projects for the **projects step service account** created in this step.
 
 ## Running Terraform locally
 
 If you deploy using Cloud Build, the bucket information is replaced in the state
-backends as a part of the build process when the build is executed by Cloud
-Build. If you want to execute Terraform locally, you need to add your Cloud
-Storage bucket to the `backend.tf` files. You can update all of these files with
-the following steps:
-
-1. Go to the `terraform-example-foundation` directory.
-1. Run the following command:
-   ```
-   for i in `find -name 'backend.tf'`; do sed -i 's/UPDATE_ME/GCS_BUCKET_NAME/' $i; done
-   ```
-   where `GCS_BUCKET_NAME` is the name of your bucket from the steps you ran
-   earlier.
+backends as a part of the build process when the build is executed by Cloud Build.
+If you want to execute Terraform locally, you need to add your Cloud
+Storage bucket to the `backend.tf` files.
+Each step has instructions for this change.
 
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 ## Inputs

0-bootstrap/outputs.tf

@@ -104,7 +104,13 @@ output "cloud_builder_artifact_repo" {
 
 output "csr_repos" {
   description = "List of Cloud Source Repos created by the module, linked to Cloud Build triggers."
-  value       = module.tf_source.csr_repos
+  value = { for k, v in module.tf_source.csr_repos : k => {
+    "id"      = v.id,
+    "name"    = v.name,
+    "project" = v.project,
+    "url"     = v.url,
+    }
+  }
 }
 
 /* ----------------------------------------