Add tool to convert from GCE VM instance disks from Google-managed en…

…cryption to CMEK encryption (#238)

* Add tool to convert from Google-managed encryption to CMEK

* Address all review comments less docstrings.

README.md

@@ -62,6 +62,7 @@ The tools folder contains ready-made utilities which can simpilfy Google Cloud P
   there are a few scripts for generating an AutoML Vision dataset CSV file from
   either raw images or image annotation files in PASCAL VOC format.
 * [DNS Sync](tools/dns-sync) - Sync a Cloud DNS zone with GCE resources. Instances and load balancers are added to the cloud DNS zone as they start from compute_engine_activity log events sent from a pub/sub push subscription. Can sync multiple projects to a single Cloud DNS zone.
+* [GCE Disk Encryption Converter](tools/gce-google-keys-to-cmek) - A tool that converts disks attached to a GCE VM instnace from Google-managed keys to a customer-managed key stored in Cloud KMS.
 * [GCE Quota Sync](tools/gce-quota-sync) - A tool that fetches resource quota usage from the GCE API and synchronizes it to Stackdriver as a custom metric, where it can be used to define automated alerts.
 * [GCP Architecture Visualizer](/tools/gcp-arch-viz) - A tool that takes CSV output from a Forseti Inventory scan and draws out a dynamic hierarchical tree diagram of org -> folders -> projects -> gcp_resources using the D3.js javascript library.
 * [GCS Bucket Mover](tools/gcs-bucket-mover) - A tool to move user's bucket, including objects, metadata, and ACL, from one project to another.

tools/gce-google-keys-to-cmek/README.md

@@ -0,0 +1,49 @@
+# Google-managed key to CMEK
+
+Convert disks attached to a GCE instance from Google-managed encryption keys
+to customer-managed encryption keys stored in Cloud KMS. This operation is
+performed in-place so the GCE VM instance maintains its current IP-address.
+
+## Script Process
+
+1. Stop the VM instance
+1. Get a list of the disks attached to the VM instance
+1. For each disk attached to the disk
+    1. If it is encrypted with a CMEK or Customer-provided key we skip the disk
+    1. Detach the disk from the VM instance
+    1. Snapshot the disk
+    1. We then create a new disk using the CMEK key stored in Cloud KMS
+1. Attach the disk to the VM instance
+1. Start the VM instance
+1. Delete the old disks and snapshots created during the process
+
+## Usage
+
+A command-line runnable Python 3 script has been provided and has the following
+usage:
+
+```
+usage: main.py [-h] --project PROJECT --zone ZONE --instance INSTANCE
+               --key-ring KEYRING --key-name KEYNAME --key-version KEYVERSION
+               [--destructive]
+
+arguments:
+  -h, --help            show this help message and exit
+  --project PROJECT     Project containing the GCE instance.
+  --zone ZONE           Zone containing the GCE instance.
+  --instance INSTANCE   Instance name.
+  --key-ring KEYRING    Name of the key ring containing the key to encrypt the
+                        disks. Must be in the same zone as the instance.
+  --key-name KEYNAME    Name of the key to encrypt the disks. Must be in the
+                        same zone as the instance.
+  --key-version KEYVERSION
+                        Version of the key to encrypt the disks.
+  --destructive         Upon completion, delete source disks and snapshots
+                        created during migration process.
+```
+
+The script uses Google [Application Default Credentials](https://cloud.google.com/docs/authentication/production).
+
+If further automation is required, e.g. running the process for multiple GCP
+instances, the `migrate_instance_to_cmek` function is a good starting point and
+it takes the same parameters as the command-line interface.

tools/gce-google-keys-to-cmek/main.py

@@ -0,0 +1,309 @@
+#!/usr/bin/env python3
+
+# Copyright 2019 Google, LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import logging
+import re
+import sys
+import time
+
+import googleapiclient
+import googleapiclient.discovery
+
+
+def main():
+  parser = argparse.ArgumentParser(
+      description='Convert disks attached to a GCE instance from Google-managed encryption keys to customer-managed encryption keys.'
+  )
+  parser.add_argument(
+      '--project',
+      required=True,
+      dest='project',
+      action='store',
+      type=str,
+      help='Project containing the GCE instance.')
+  parser.add_argument(
+      '--zone',
+      required=True,
+      dest='zone',
+      action='store',
+      type=str,
+      help='Zone containing the GCE instance.')
+  parser.add_argument(
+      '--instance',
+      required=True,
+      dest='instance',
+      action='store',
+      type=str,
+      help='Instance name.')
+  parser.add_argument(
+      '--key-ring',
+      required=True,
+      dest='key_ring',
+      action='store',
+      type=str,
+      help='Name of the key ring containing the key to encrypt the disks. Must be in the same zone as the instance.'
+  )
+  parser.add_argument(
+      '--key-name',
+      required=True,
+      dest='key_name',
+      action='store',
+      type=str,
+      help='Name of the key to encrypt the disks. Must be in the same zone as the instance.'
+  )
+  parser.add_argument(
+      '--key-version',
+      required=True,
+      dest='key_version',
+      action='store',
+      type=int,
+      help='Version of the key to encrypt the disks.')
+  parser.add_argument(
+      '--destructive',
+      dest='destructive',
+      action='store_const',
+      const=True,
+      default=False,
+      help='Upon completion, delete source disks and snapshots created during migration process.'
+  )
+  args = parser.parse_args()
+
+  migrate_instance_to_cmek(args.project, args.zone, args.instance,
+                           args.key_ring, args.key_name, args.key_version,
+                           args.destructive)
+
+
+def migrate_instance_to_cmek(project, zone, instance, key_ring, key_name,
+                             key_version, destructive):
+  start = time.time()
+
+  zone_regexp = r'^(\w\w-\w*\d)-(\w)$'
+  region = re.search(zone_regexp, zone).group(1)
+
+  compute = googleapiclient.discovery.build('compute', 'v1')
+
+  stop_instance(compute, project, zone, instance)
+  disks = get_instance_disks(compute, project, zone, instance)
+  for source_disk in disks:
+    disk_regexp = r'^https:\/\/www\.googleapis\.com\/compute\/v1\/projects\/(.*?)\/zones\/(.*?)\/disks\/(.*?)$'
+    disk_url = source_disk['source']
+    existing_disk_name = re.search(disk_regexp, disk_url).group(3)
+
+    if 'diskEncryptionKey' in source_disk:
+      logging.info('Skipping %s, already encrypyed with %s', existing_disk_name,
+                   source_disk['diskEncryptionKey'])
+      continue
+
+    snapshot_name = '{}-goog-to-cmek'.format(existing_disk_name)
+    new_disk_name = '{}-cmek'.format(existing_disk_name)
+    disk_type = get_disk_type(compute, project, zone, existing_disk_name)
+
+    create_snapshot(compute, project, zone, existing_disk_name, snapshot_name)
+    key_name = 'projects/{0}/locations/{1}/keyRings/{2}/cryptoKeys/{3}/cryptoKeyVersions/{4}'.format(
+        project, region, key_ring, key_name, key_version)
+    create_disk(compute, project, region, zone, snapshot_name, new_disk_name,
+                disk_type, key_name)
+    detach_disk(compute, project, zone, instance, existing_disk_name)
+
+    boot = source_disk['boot']
+    auto_delete = source_disk['autoDelete']
+    attach_disk(compute, project, zone, instance, new_disk_name, boot,
+                auto_delete)
+    if destructive:
+      delete_disk(compute, project, zone, existing_disk_name)
+      delete_snapshot(compute, project, snapshot_name)
+
+  start_instance(compute, project, zone, instance)
+
+  end = time.time()
+  logging.info('Migration took %s seconds.', end - start)
+
+
+def get_disk_type(compute, project, zone, disk_name):
+  logging.debug('Getting project=%s, zone=%s, disk_name=%s metadata', project,
+                zone, disk_name)
+  result = compute.disks().get(
+      project=project, zone=zone, disk=disk_name).execute()
+  logging.debug('Getting project=%s, zone=%s, disk_name=%s metadata complete.',
+                project, zone, disk_name)
+  return result['type']
+
+
+def get_instance_disks(compute, project, zone, instance):
+  logging.debug('Getting project=%s, zone=%s, instance=%s disks', project, zone,
+                instance)
+  result = compute.instances().get(
+      project=project, zone=zone, instance=instance).execute()
+  logging.debug('Getting project=%s, zone=%s, instance=%s disks complete.',
+                project, zone, instance)
+  return result['disks']
+
+
+def create_snapshot(compute, project, zone, disk, snapshot_name):
+  body = {
+      'name': snapshot_name,
+  }
+  logging.debug('Creating snapshot of disk project=%s, zone=%s, disk=%s',
+                project, zone, disk)
+  operation = compute.disks().createSnapshot(
+      project=project, zone=zone, disk=disk, body=body).execute()
+  result = wait_for_zonal_operation(compute, project, zone, operation)
+  logging.debug('Snapshotting of disk project=%s, zone=%s, disk=%s complete.',
+                project, zone, disk)
+  return result
+
+
+def delete_snapshot(compute, project, snapshot_name):
+  logging.debug('Deleting snapshot project=%s, snapshot_name=%s', project,
+                snapshot_name)
+  operation = compute.snapshots().delete(
+      project=project, snapshot=snapshot_name).execute()
+  result = wait_for_global_operation(compute, project, operation)
+  logging.debug('Deleting snapshot project=%s,  snapshot_name=%s complete.',
+                project, snapshot_name)
+  return result
+
+
+def attach_disk(compute, project, zone, instance, disk, boot, auto_delete):
+  """ Attaches disk to instance.
+
+  Requries iam.serviceAccountUser
+  """
+  disk_url = 'projects/{0}/zones/{1}/disks/{2}'.format(project, zone, disk)
+  body = {
+      'autoDelete': auto_delete,
+      'boot': boot,
+      'source': disk_url,
+  }
+  logging.debug('Attaching disk project=%s, zone=%s, instance=%s, disk=%s',
+                project, zone, instance, disk_url)
+  operation = compute.instances().attachDisk(
+      project=project, zone=zone, instance=instance, body=body).execute()
+  result = wait_for_zonal_operation(compute, project, zone, operation)
+  logging.debug(
+      'Attaching disk project=%s, zone=%s, instance=%s, disk=%s complete.',
+      project, zone, instance, disk_url)
+  return result
+
+
+def detach_disk(compute, project, zone, instance, disk):
+  logging.debug('Detaching disk project=%s, zone=%s, instance=%s, disk=%s',
+                project, zone, instance, disk)
+  operation = compute.instances().detachDisk(
+      project=project, zone=zone, instance=instance, deviceName=disk).execute()
+  result = wait_for_zonal_operation(compute, project, zone, operation)
+  logging.debug(
+      'Detaching disk project=%s, zone=%s, instance=%s, disk=%s complete.',
+      project, zone, instance, disk)
+  return result
+
+
+def delete_disk(compute, project, zone, disk):
+  logging.debug('Deleting disk project=%s, zone=%s, disk=%s', project, zone,
+                disk)
+  operation = compute.disks().delete(
+      project=project, zone=zone, disk=disk).execute()
+  result = wait_for_zonal_operation(compute, project, zone, operation)
+  logging.debug('Deleting disk project=%s, zone=%s, disk=%s complete.', project,
+                zone, disk)
+  return result
+
+
+def create_disk(compute, project, region, zone, snapshot_name, disk_name,
+                disk_type, key_name):
+  """Creates a new CMEK encrypted persistent disk from a snapshot"""
+  source_snapshot = 'projects/{0}/global/snapshots/{1}'.format(
+      project, snapshot_name)
+  body = {
+      'name': disk_name,
+      'sourceSnapshot': source_snapshot,
+      'type': disk_type,
+      'diskEncryptionKey': {
+          'kmsKeyName': key_name,
+      },
+  }
+  logging.debug(
+      'Creating new disk project=%s, zone=%s, name=%s source_snapshot=%s, kmsKeyName=%s',
+      project, zone, disk_name, source_snapshot, key_name)
+  operation = compute.disks().insert(
+      project=project, zone=zone, body=body).execute()
+  result = wait_for_zonal_operation(compute, project, zone, operation)
+  logging.debug(
+      'Creating new disk project=%s, zone=%s, name=%s source_snapshot=%s, kmsKeyName=%s complete.',
+      project, zone, disk_name, source_snapshot, key_name)
+  return result
+
+
+def start_instance(compute, project, zone, instance):
+  logging.debug('Starting project=%s, zone=%s, instance=%s', project, zone,
+                instance)
+  operation = compute.instances().start(
+      project=project, zone=zone, instance=instance).execute()
+  result = wait_for_zonal_operation(compute, project, zone, operation)
+  logging.debug('Starting project=%s, zone=%s, instance=%s complete.', project,
+                zone, instance)
+  return result
+
+
+def stop_instance(compute, project, zone, instance):
+  logging.debug('Stopping project=%s, zone=%s, instance=%s', project, zone,
+                instance)
+  operation = compute.instances().stop(
+      project=project, zone=zone, instance=instance).execute()
+  result = wait_for_zonal_operation(compute, project, zone, operation)
+  logging.debug('Stopping project=%s, zone=%s, instance=%s complete.', project,
+                zone, instance)
+  return result
+
+
+def wait_for_global_operation(compute, project, operation):
+  operation = operation['name']
+
+  def build():
+    return compute.globalOperations().get(project=project, operation=operation)
+
+  return _wait_for_operation(operation, build)
+
+
+def wait_for_zonal_operation(compute, project, zone, operation):
+  operation = operation['name']
+
+  def build():
+    return compute.zoneOperations().get(
+        project=project, zone=zone, operation=operation)
+
+  return _wait_for_operation(operation, build)
+
+
+def _wait_for_operation(operation, build_request):
+  """Helper for waiting for operation to complete."""
+  logging.debug('Waiting for %s', operation, end='')
+  while True:
+    sys.stdout.flush()
+    result = build_request().execute()
+    if result['status'] == 'DONE':
+      logging.debug('done!')
+      if 'error' in result:
+        logging.error('finished with an error')
+        logging.error('Error %s', result['error'])
+        raise Exception(result['error'])
+      return result
+    time.sleep(5)
+
+
+if __name__ == '__main__':
+  main()

tools/gce-google-keys-to-cmek/requirements.txt

@@ -0,0 +1 @@
+google-api-python-client