Feat: extract python library, write a notifier that uses it (#3)

* Created a new notifier: `release-auto-promoter` advances releases by creating rollouts for the next target in a release pipeline when the previous succeeded.
* Extracted a python library to simplify the process of building a notifier.
* bugfixes and various additions across the terraform modules.

.devcontainer/Dockerfile

@@ -40,7 +40,7 @@ RUN echo "installing local dev dependencies" && \
     poetry config virtualenvs.create false && \
     # install terraform
     curl https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip > terraform_linux_amd64.zip && \
-    # TODO(brandonjbjelland): verify download against checksum  b8cf184dee15dfa89713fe56085313ab23db22e17284a9a27c0999c67ce3021e
+    # TODO(bjb): verify download against checksum  b8cf184dee15dfa89713fe56085313ab23db22e17284a9a27c0999c67ce3021e
     unzip -o terraform_linux_amd64.zip -d /usr/local/bin/ && \
     chmod +x /usr/local/bin/terraform && \
     rm terraform_linux_amd64.zip && \

.gitignore

@@ -14,7 +14,6 @@ dist/
 downloads/
 eggs/
 .eggs/
-lib/
 lib64/
 parts/
 sdist/
@@ -133,3 +132,4 @@ config.hcl
 .terraform
 .terraform.lock.hcl
 .vscode/settings.json
+.env

CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - Jan 31, 2022
+
+### Added
+
+- Everything. This is the initial repo contents.
+- Created 2 new notifiers: `echo-fastapi` demonstrates the basics of how to build a notifier using fastapi. `release-auto-promoter` advances releases by creating rollouts for the next target in a release pipeline when the previous succeeded.
+- Extracted a python library to simplify the process of building a notifier.
+- Created a terraform module (`terraform/cloud-deploy-notification-infra`) for the notification infrastructure needing to be instantiated once on a project that hosts notifiers.
+- Created a terraform module (`terraform/cloud-deploy-notifier`) that can manage all infra components of a typical notifier. Both `echo-fastapi` and `release-auto-promoter` though having slightly different infra requirements, are able to use this module to create all necessary resources.
+- The `/demo` directory packs both an example application (`nginx-app`) with its release pipeline IaC and a set of configurations in a `terragrunt` directory. The `terragrunt.hcl` configurations wire together and call the various terraform modules thorughout this repo, showing how all the components relate and can be deployed as separate units.
+- `data-samples/` provides 3 example PubSub message payloads so devs can quickly understand the payloads we're working with.
+- `.devcontainer` provides configurations for developing this project within a reproducable containerized environment.

README.md

@@ -5,17 +5,18 @@ state transitions.
 
 Similar to the collection of cloud builders at
 [cloud-builders-community](https://github.com/GoogleCloudPlatform/cloud-builders-community)
-and [cloud-build-notifiers](https://github.com/GoogleCloudPlatform/cloud-build-notifiers),
+and notifiers at
+[cloud-build-notifiers](https://github.com/GoogleCloudPlatform/cloud-build-notifiers),
 this repo aims to be a commons of utilities to enhance
 [Google Cloud Deploy](https://cloud.google.com/deploy/docs/overview). Google
 Cloud Deploy provides essential primitives, a useful UI, and granular permissions
-for managing deployments that can span many environments in Google Cloud. Cloud
+for managing releases that can span many environments in Google Cloud. Cloud
 Deploy also lacks some features and functionality of established CI/CD tools
-(notifications, automatic release promotion, auto-rollback on verification failure,
+(notifications, conditional release promotion, auto-rollback on verification failure,
 post-deploy image tagging, etc.) but instead gives extension points via
 [PubSub notifications](https://cloud.google.com/deploy/docs/subscribe-deploy-notifications)
-to flexibly fill these gaps. The solutions here stand up infrastructure and
-services necessary to build on top of that notification platform.
+to flexibly fill these gaps. The notifier solutions here stand up infrastructure
+and services necessary to build on top of that notification platform.
 
 ## Architecture
 
@@ -24,32 +25,38 @@ in cloud-builders-community and cloud-build-notifiers. Namely, source code,
 dockerfiles, and cloudbuild configurations are packaged for users to build, store,
 and deploy container images to their GCP projects.
 
-Expanding on this pattern, notifier authors are encouraged to ship a small terraform
-module to stand up all the necessary infra components of a notifier (typically
-manages the notifier workload, a PubSub subscription, a service account, and the
-required role memberships). This gives an more ideal deployment scenario with
-users only needing to build an image and invoke that module to make a deploy
-notifier available.
-
-The way notifiers are configured for usage by workload deploy pipelines differs
-to the cloud-build repos. Following a kubernetes-style configuration approach,
-once a deploy notifier is deployed, it can operate against notifications
-originating from any Cloud Deploy pipeline but until a deploy pipeline opts-in,
-a notifier should do nothing.
-
-A workload's deploy pipeline opts-in to using a given notifier via an annotation
-on the pipeline (and potentially annotations on targets). The annotation value
-should point to a secret in secret manager that the notifier can use for
-configuration values during an invocation. This pattern allows any number of
-notifiers to be deployed and available to workload pipelines while giving
-pipeline owners a simple mechanism to enable and configure a custom set of
-notifiers on their pipelines.
+Expanding on this pattern, to manage notifier infra, authors should either:
+
+1. verify the `terraform/cloud-deploy-notifier` root module is sufficient to
+manage all dependent infra for the notifier or
+2. build a small terraform module, that likely calls `terraform/cloud-deploy-notifier`
+to manage all dependent infra.
+
+In either case, including an example of the variable inputs in an `example.tfvars`
+file is a simple way to guide notifier consumers on how to run this terraform.
+
+## Per Pipeline notifier configuration
+
+Once deployed, notifiers are configured via workload deploy pipelines that opt-in
+to using them. Following a kubernetes-style configuration approach, each notifier
+has a distinct configuration annotation key that Cloud Deploy Pipelines must
+include if they want to leverage a notifier.
+
+The annotation value should point to a user-configured secret in Secret Manager.
+This secret should contain configuration values that the notifier unpacks during
+an execution. This is a powerful pattern for a few reasons:
+
+1. It's secure - some notifiers will require secret data, others won't necessarily, but it's not a bad practice to treat all configuration as sensitive and RBAC controlled.
+2. Notifiers can be liberally deployed without affecting existing deploy pipelines - enabling a notifier requires an annotation on opting-in deployment pipelines and the configuration in secret manager.
 
 ## Deploy notifier index
 
-* [echo-fastapi](notifiers/echo-fastapi/) - an example deploy notifier in Python
-that echos the payload.
-* echo-go - an example deploy notifier in go.
+* [echo-fastapi](notifiers/echo-fastapi/) is an example deploy notifier in Python
+that echos the payload, the configuration secret contents, and kwargs.
+* [release-auto-promoter](notifiers/release-auto-promoter/) is a notifier that
+promotes releases as rollouts in deploy pipeline succeed. This reduces a manual
+task from release managers or engineers who just want successful deployments to
+be promoted to higher envs.
 
 ## Development
 
@@ -71,6 +78,9 @@ an environment variable which many/most/maybe all Google Cloud SDKs support:
 export GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token)
 ```
 
-## What's forthcoming?
+## What's upcoming?
 
-1. TODO(brandonjbjelland): Build out a demo that operates against a pipeline targeting a GKE workload
+1. Build out a demo that operates against a pipeline targeting a GKE workload
+2. create an example notifier in go
+3. build a second go notifier, extract an interface
+4. blog posts discussing the nuts and bolts of the project

demo/nginx-app/terraform/iam.tf

@@ -50,3 +50,10 @@ resource "google_secret_manager_secret_iam_member" "cloud_deploy_notifiers_secre
   role      = "roles/secretmanager.viewer"
   member    = "serviceAccount:${each.value.labels.notifier}@${var.project_id}.iam.gserviceaccount.com"
 }
+
+resource "google_service_account_iam_member" "deployer_sa_users" {
+  for_each           = toset(var.deployer_service_account_users)
+  service_account_id = google_service_account.app_deployer.name
+  role               = "roles/iam.serviceAccountUser"
+  member             = each.key
+}

demo/nginx-app/terraform/variables.tf

@@ -4,18 +4,24 @@ variable "annotations" {
   type        = map(string)
 }
 
-variable "enabled_cloud_deploy_notifiers" {
-  default     = []
-  description = "Fully qualified Cloud Deploy notifiers to enable on the app's deployment pipeline. This varible is the interface for creating annotations, secrets, and secret permissions necessary for Cloud Deploy notifiers. List entries should be in the form 'subdomain.domain.tld/notifier' matching that of the Cloud Deployer. e.g. [\"deploy-notifiers.computeclub.io/echo-fastapi\",]"
-  type        = list(string)
-}
-
 variable "app_name" {
   default     = "nginx-app"
   description = "The name of the app being deployed"
   type        = string
 }
 
+variable "deployer_service_account_users" {
+  default     = []
+  description = "A list of fully qualified IAM members given serviceAccountUser access to the deployer SA. e.g. [ \"serviceAccount:my-notifier-sa@project-id.gserviceaccount.iam.com\"]"
+  type        = list(string)
+}
+
+variable "enabled_cloud_deploy_notifiers" {
+  default     = []
+  description = "Fully qualified Cloud Deploy notifiers to enable on the app's deployment pipeline. This varible is the interface for creating annotations, secrets, and secret permissions necessary for Cloud Deploy notifiers. List entries should be in the form 'subdomain.domain.tld/notifier' matching that of the Cloud Deployer. e.g. [\"deploy-notifiers.computeclub.io/echo-fastapi\",]"
+  type        = list(string)
+}
+
 variable "labels" {
   default     = {}
   description = "the labels to attach to the pipeline. These are purely metadata for humans"

demo/terragrunt/cloud-deploy-foundation/terragrunt.hcl

@@ -3,7 +3,7 @@ include {
 }
 
 terraform {
-  # TODO(brandonjbjelland): switch to a tag ref
+  # TODO(bjb): switch to a tag ref
   # source = "github.com/computeclub/gcp-cloud-deploy-notifiers//terraform/cloud-deploy-notification-infra?ref=main"
   source = "${find_in_parent_folders("gcp-cloud-deploy-notifiers")}//terraform/cloud-deploy-notification-infra"
 }

demo/terragrunt/echo-fastapi/terragrunt.hcl

@@ -3,7 +3,7 @@ include {
 }
 
 terraform {
-  # TODO(brandonjbjelland): switch to a tag ref
+  # TODO(bjb): switch to a tag ref
   # source = "github.com/computeclub/gcp-cloud-deploy-notifiers//terraform/cloud-deploy-notifier?ref=main"
   source = "${find_in_parent_folders("gcp-cloud-deploy-notifiers")}//terraform/cloud-deploy-notifier"
   before_hook "before_hook" {
@@ -31,6 +31,8 @@ dependency "cloud_deploy_foundation" {
     artifact_registry_repo = {
       location = "foo"
       name     = "bar"
+      format   = "docker"
+      project  = "my-project-id"
     }
     cloud_deploy_pubsub_topics = {
       clouddeploy-resources  = {}
@@ -48,8 +50,18 @@ inputs = {
       value = "INFO"
     },
     {
-      name  = "SRC_SHA1"
-      value = sha1(join("", [for f in fileset("${local.notifier_path}/src", "*") : filesha1("${local.notifier_path}/src/${f}")]))
+      name = "SRC_SHA1"
+      value = sha1(
+        join(
+          "",
+          [
+            for f in fileset(
+              "${local.notifier_path}/",
+              "**"
+            ) : filesha1("${local.notifier_path}/${f}")
+          ]
+        )
+      )
     },
   ]
   notifier_name = "echo-fastapi"

demo/terragrunt/nginx-app-infra/terragrunt.hcl

@@ -3,7 +3,7 @@ include {
 }
 
 terraform {
-  # TODO(brandonjbjelland): switch to a tag ref
+  # TODO(bjb): switch to a tag ref
   # source = "github.com/computeclub/gcp-cloud-deploy-notifiers//demo/nginx-app/terraform?ref=main"
   source = "${find_in_parent_folders("nginx-app")}//terraform"
   after_hook "after_hook" {
@@ -34,10 +34,18 @@ dependency "echo_fastapi" {
   mock_outputs = {}
 }
 
+dependency "release_auto_promoter_notifier" {
+  config_path  = "${find_in_parent_folders("terragrunt")}/release-auto-promoter-notifier"
+  mock_outputs = {}
+}
 
 inputs = {
+  deployer_service_account_users = [
+    "serviceAccount:${dependency.release_auto_promoter_notifier.outputs.workload_service_account.email}"
+  ]
   enabled_cloud_deploy_notifiers = [
     dependency.echo_fastapi.outputs.config_annotation,
+    dependency.release_auto_promoter_notifier.outputs.config_annotation,
   ]
   project_id = local.config.locals.project_id
 }

demo/terragrunt/release-auto-promoter-notifier/terragrunt.hcl

@@ -0,0 +1,79 @@
+include {
+  path = find_in_parent_folders("terragrunt.hcl")
+}
+
+terraform {
+  # TODO(bjb): switch to a tag ref
+  # source = "github.com/computeclub/gcp-cloud-deploy-notifiers//terraform/cloud-deploy-notifier?ref=main"
+  source = "${find_in_parent_folders("gcp-cloud-deploy-notifiers")}//terraform/cloud-deploy-notifier"
+  before_hook "before_hook" {
+    commands = ["apply"]
+    execute = [
+      "gcloud",
+      "builds",
+      "submit",
+      "--config=${local.notifier_path}/cloudbuild.yaml",
+      "--project=${local.config.locals.project_id}",
+      "--substitutions=_REGISTRY_REPO_URL=${dependency.cloud_deploy_foundation.outputs.artifact_registry_repo_endpoint}",
+      "${local.notifier_path}/"
+    ]
+  }
+}
+
+locals {
+  config        = read_terragrunt_config(find_in_parent_folders("config.hcl"))
+  notifier_path = "${find_in_parent_folders("gcp-cloud-deploy-notifiers")}/notifiers/release-auto-promoter"
+}
+
+dependency "cloud_deploy_foundation" {
+  config_path = "${find_in_parent_folders("terragrunt")}/cloud-deploy-foundation"
+  mock_outputs = {
+    artifact_registry_repo = {
+      location = "foo"
+      name     = "bar"
+      format   = "docker"
+      project  = "my-project-id"
+    }
+    cloud_deploy_pubsub_topics = {
+      clouddeploy-resources  = {}
+      clouddeploy-operations = {}
+      clouddeploy-approvals  = {}
+    }
+  }
+}
+
+inputs = {
+  artifact_registry_repo = dependency.cloud_deploy_foundation.outputs.artifact_registry_repo
+  cloud_deploy_notification_subscriptions = [
+    {
+      topic  = "clouddeploy-operations"
+      filter = "attributes.Action = \"Succeed\" AND attributes.ResourceType = \"Rollout\""
+    },
+  ]
+  env_vars = [
+    {
+      name  = "LOG_LEVEL"
+      value = "INFO"
+    },
+    {
+      name = "SRC_SHA1"
+      value = sha1(
+        join(
+          "",
+          [
+            for f in fileset(
+              "${local.notifier_path}/",
+              "**"
+            ) : filesha1("${local.notifier_path}/${f}")
+          ]
+        )
+      )
+    },
+  ]
+  notifier_name = "release-auto-promoter"
+  project_id    = local.config.locals.project_id
+  region        = "us-central1"
+  workload_sa_project_roles = [
+    "roles/clouddeploy.releaser",
+  ]
+}

lib/python-notifier/README.md

@@ -0,0 +1,5 @@
+# python-notifier
+
+A Python library to simplify building a Cloud Deploy notifier. This optional
+library provides a base class `BaseNotifier` with an `action()` method needing
+to be implemented by inheriting classes.

lib/python-notifier/clouddeploy_notifier/exceptions.py

@@ -0,0 +1,7 @@
+# -*- coding: utf-8 -*-
+class UnknownMessageType(Exception):
+    pass
+
+
+class UnkownPipeline(Exception):
+    pass

notifiers/echo-fastapi/src/config.py → lib/python-notifier/clouddeploy_notifier/log_config.py

@@ -1,29 +1,10 @@
 # -*- coding: utf-8 -*-
-"""config contains logging and app configurations."""
+"""."""
+# -*- coding: utf-8 -*-
 import logging
-import os
 
 from google.cloud import logging as cloud_logging
-from pydantic import BaseSettings
-
-
-class Settings(BaseSettings):
-    """
-    Application settings.
-    """
-
-    app_name: str = "echo-fastapi"
-    log_level: str = os.environ.get("LOG_LEVEL") or logging.getLevelName(
-        logging.getLogger("uvicorn.error").level
-    )
-
-    class Config:
-        """Enables dotenv files to be read at startup."""
-
-        env_file = ".env"
-
-
-settings = Settings()  # type: ignore
+from clouddeploy_notifier.settings import settings
 
 LOGGING_CONFIG_DICT = {
     "version": 1,