Check in expected kpt output for Kptfile refactoring

* Per GoogleCloudPlatform/kubeflow-distribution#89 we need to get rid of the legacy
  partial setters and move to using a KptFile and substitutions.

* In preparation for that we want to check in a set of test data
  that is the result of running our kpt cfg set with a given set of
  values

* This test data will be used to verify that the refactoring to use
  a KptFile doesn't change the output.

* After adding the KptFile we can simply regenerate the testdata
  and then look at the diff to ensure there are no unexpected changes.

hack/generate_gcp_kpt_data.py

@@ -0,0 +1,61 @@
+"""Regenerate tests."""
+
+import fire
+import logging
+import os
+import shutil
+import subprocess
+
+class Generator:
+  @staticmethod
+  def write_gcp_kpt(kpt="kpt"):
+    """Create test data based on running the kpt commands.
+
+    This will allow us to see any diffs if we refactor the commands
+    """
+    repo_root = subprocess.check_output(["git", "rev-parse", "--show-toplevel"])
+    repo_root = repo_root.decode()
+    repo_root = repo_root.strip()
+
+    data_dir = os.path.join(repo_root, "tests", "testdata_gcp_kpt")
+    if os.path.exists(data_dir):
+      shutil.rmtree(data_dir)
+
+    logging.info("Removing directory %s", data_dir)
+
+    subdirs = ["v2", "Kptfile"]
+
+    for s in subdirs:
+      src = os.path.join(repo_root, "gcp", s)
+      if not os.path.exists(src):
+        continue
+      shutil.copytree(src, os.path.join(data_dir, "gcp", s))
+
+    # Run a bunch of kpt commands. We want to change all the setters
+    # unique values so we can see how the substitutions play out
+    setters = {
+      "gcloud.core.project": "customerProject",
+      "gcloud.project.projectNumber": "999911112222",
+      "gcloud.compute.zone": "testZone",
+      "gcloud.compute.region": "testRegion",
+      "location": "testLocation",
+      "name": "testKptName",
+      "log-firewalls": "kptLogFirewalls",
+    }
+
+    for k, v in setters.items():
+      command = [kpt, "cfg", "set", ".", k, v]
+      logging.info("Executing:\n%s", " ".join(command))
+      subprocess.check_call(command, cwd=data_dir)
+
+
+if __name__ == "__main__":
+
+  logging.basicConfig(
+      level=logging.INFO,
+      format=('%(levelname)s|%(asctime)s'
+              '|%(pathname)s|%(lineno)d| %(message)s'),
+      datefmt='%Y-%m-%dT%H:%M:%S',
+  )
+  logging.getLogger().setLevel(logging.INFO)
+  fire.Fire(Generator)

tests/testdata_gcp_kpt/gcp/v2/README.md

@@ -0,0 +1,88 @@
+# Alpha: Kubeflow on KCC Installation Guide
+
+This instruction explains how to set up Kubeflow on top of Config Connector (KCC) and Anthos Service Mesh (ASM).
+Compared with the currently documented GCP deployment, this architecture uses KCC instead of Deployment Manager, and service mesh in the form of ASM instead of open source Istio.
+
+Assume using IAP to protect the kubeflow UI endpoint.
+
+### Benefits of using  KCC
+
+[KCC](https://cloud.google.com/config-connector) is a Google Kubernetes Engine (GKE) addon that allows you to manage your Google Cloud resources through Kubernetes configuration.
+With KCC users can manage their Google Cloud infrastructure the same way as manage Kubernetes applications (Infrastructure as code).
+
+
+### Benefits of using ASM
+
+[ASM](https://cloud.google.com/service-mesh/docs/overview) is a GCP distribution of Istio with more Observability features & Security features.
+
+## Installation Steps
+
+
+#### Step 0: Setup KCC
+If you don't have a running KCC controller yet, follow [KCC instructions](https://cloud.google.com/config-connector/docs/how-to/install-upgrade-uninstall) to create a KCC controller for your organization.
+We recommend “Namespaced mode” for KCC controller setup.
+
+From now on assume your KCC controller was hosted in project `kcc-host-project-id`.
+Each Project managed by KCC will have a namespace in the KCC cluster named after project id. For example Project “kubeflow-project-id” will linked to a namespace named “kubeflow-project-id” in KCC cluster.
+Kfctl | anthoscli | ACP
+
+#### Step 1: Create GCP resources through KCC
+* Install kpt
+
+  ```
+  gcloud components install kpt alpha
+  gcloud components update
+  ```
+
+* Set project-id / zone / cluster name
+
+  Checkout latest kubeflow/manifests repo; cd manifests/gcp
+
+  Choose a cluster name `export CLUSTER_NAME=choose-name`
+
+  ```
+  kpt cfg set v2 gcloud.core.project $(gcloud config get-value project)
+  kpt cfg set v2 cluster-name $(CLUSTER_NAME)
+  kpt cfg set v2 gcloud.compute.zone $(gcloud config get-value compute/zone)
+  ```
+
+* Connect kubectl to KCC cluster
+
+  `gcloud container clusters get-credentials <cluster-name> --zone <> --project <kcc-host-project-id>`
+
+* Apply CNRM resources
+
+  `kustomize build v2/cnrm | kubectl apply -n <kubeflow-project-id> -f -`
+
+
+#### Step 2: Install ASM
+Install ASM on the newly created kubeflow cluster `CLUSTER_NAME`
+
+* Connect kubectl to the new kubeflow cluster `CLUSTER_NAME`
+
+  `gcloud container clusters get-credentials $(CLUSTER_NAME) --zone <> --project <kubeflow-project-id>`
+
+* [Set credentials and permissions](https://cloud.google.com/service-mesh/docs/gke-install-existing-cluster#set_credentials_and_permissions)
+
+* [Download istioctl released by GCP](https://cloud.google.com/service-mesh/docs/gke-install-existing-cluster#download_the_installation_file)
+
+* Run Istioctl (download in previous step)
+
+  `istioctl manifest apply -f v2/asm/istio-operator.yaml`
+
+
+#### Step 3: Deploy Kubeflow components
+
+* [Setup Environment Variables for IAP](https://www.kubeflow.org/docs/gke/deploy/oauth-setup/)
+
+	```
+	export CLIENT_ID=
+	export CLIENT_SECRET=
+  ```
+
+* Install Kubeflow on the newly created cluster
+
+  ```
+  mkdir $(CLUSTER_NAME) && cd $(CLUSTER_NAME)
+  kfctl apply -V -f https://raw.githubusercontent.com/kubeflow/manifests/master/kfdef/kfctl_gcp_asm_exp.yaml
+  ```

tests/testdata_gcp_kpt/gcp/v2/asm/istio-operator.yaml

@@ -0,0 +1,38 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: install.istio.io/v1alpha2
+kind: IstioControlPlane
+metadata:
+  clusterName: "customerProject/testLocation/testKptName" # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"},{"name":"name","value":"testKptName"},{"name":"location","value":"testLocation"}]}}
+spec:
+  profile: asm
+  hub: gcr.io/gke-release/asm
+  tag: 1.4.7-asm.0
+  values:
+    gateways:
+      istio-ingressgateway:
+        type: NodePort
+    global:
+      meshID: "customerProject_testLocation_testKptName" # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"},{"name":"name","value":"testKptName"},{"name":"location","value":"testLocation"}]}}
+      trustDomain: "customerProject.svc.id.goog" # {"type":"string","x-kustomize":{"partialSetters":[{"name":"gcloud.core.project","value":"customerProject"}]}}
+      sds:
+        token:
+          aud: "customerProject.svc.id.goog" # {"type":"string","x-kustomize":{"partialSetters":[{"name":"gcloud.core.project","value":"customerProject"}]}}
+      proxy:
+        env:
+          GCP_METADATA: "customerProject|999911112222|testKptName|testZone" # {"type":"string","x-kustomize":{"partialSetters":[{"name":"gcloud.core.project","value":"customerProject"},{"name":"gcloud.project.projectNumber","value":"999911112222"},{"name":"name","value":"testKptName"},{"name":"gcloud.compute.zone","value":"testZone"}]}}
+    nodeagent:
+      env:
+        GKE_CLUSTER_URL: "https://container.googleapis.com/v1/projects/customerProject/locations/testLocation/clusters/testKptName" # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"},{"name":"name","value":"testKptName"},{"name":"location","value":"testLocation"}]}}

tests/testdata_gcp_kpt/gcp/v2/cnrm/cluster/cluster.yaml

@@ -0,0 +1,62 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# TODO(kunming): kustomize this config to include all the options we currently offer through DM
+
+apiVersion: container.cnrm.cloud.google.com/v1beta1
+kind: ContainerCluster
+metadata:
+  clusterName: "customerProject/testZone/testKptName" # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"},{"name":"name","value":"testKptName"},{"name":"gcloud.compute.zone","value":"testZone"}]}}
+  labels:
+    mesh_id: "customerProject_testZone_testKptName" # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"},{"name":"name","value":"testKptName"},{"name":"gcloud.compute.zone","value":"testZone"}]}}
+  name: testKptName # {"type":"string","x-kustomize":{"setter":{"name":"name","value":"testKptName"}}}
+spec:
+  initialNodeCount: 2
+  clusterAutoscaling:
+    enabled: true
+    autoProvisioningDefaults:
+      oauthScopes:
+      - https://www.googleapis.com/auth/logging.write
+      - https://www.googleapis.com/auth/monitoring
+      - https://www.googleapis.com/auth/devstorage.read_only
+      serviceAccountRef:
+        name: testKptName-vm # {"type":"string","x-kustomize":{"partialSetters":[{"name":"name","value":"testKptName"}]}}
+    resourceLimits:
+    - resourceType: cpu
+      maximum: 128
+    - resourceType: memory
+      maximum: 2000
+    - resourceType: nvidia-tesla-k80
+      maximum: 16
+  releaseChannel:
+    # Per https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/194
+    # use upper case for the channels
+    channel: REGULAR
+  location: testZone # {"type":"string","x-kustomize":{"setBy":"kpt","setter":{"name":"gcloud.compute.zone","value":"testZone"}}}
+  workloadIdentityConfig:
+    identityNamespace: customerProject.svc.id.goog # {"type":"string","x-kustomize":{"partialSetters":[{"name":"gcloud.core.project","value":"customerProject"}]}}
+  loggingService: logging.googleapis.com/kubernetes
+  monitoringService: monitoring.googleapis.com/kubernetes
+  nodeConfig:
+    machineType: n1-standard-8
+    metadata:
+      disable-legacy-endpoints: "true"
+    oauthScopes:
+    - https://www.googleapis.com/auth/logging.write
+    - https://www.googleapis.com/auth/monitoring
+    - https://www.googleapis.com/auth/devstorage.read_only
+    serviceAccountRef:
+      name: testKptName-vm # {"type":"string","x-kustomize":{"partialSetters":[{"name":"name","value":"testKptName"}]}}
+    workloadMetadataConfig:
+      nodeMetadata: GKE_METADATA_SERVER

tests/testdata_gcp_kpt/gcp/v2/cnrm/cluster/kf-vm-policy.yaml

@@ -0,0 +1,71 @@
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: testKptName-vm-logging # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"}]}}
+spec:
+  member: serviceAccount:testKptName-vm@customerProject.iam.gserviceaccount.com # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"},{"name":"gcloud.core.project","value":"customerProject"}]}}
+  role: roles/logging.logWriter
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Project
+    external: projects/customerProject # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"}]}}
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: testKptName-vm-policy-monitoring # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"}]}}
+spec:
+  member: serviceAccount:testKptName-vm@customerProject.iam.gserviceaccount.com # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"},{"name":"gcloud.core.project","value":"customerProject"}]}}
+  role: roles/monitoring.metricWriter
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Project
+    external: projects/customerProject # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"}]}}
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: testKptName-vm-policy-meshtelemetry # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"}]}}
+spec:
+  member: serviceAccount:testKptName-vm@customerProject.iam.gserviceaccount.com # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"},{"name":"gcloud.core.project","value":"customerProject"}]}}
+  role: roles/meshtelemetry.reporter
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Project
+    external: projects/customerProject # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"}]}}
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: testKptName-vm-policy-cloudtrace # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"}]}}
+spec:
+  member: serviceAccount:testKptName-vm@customerProject.iam.gserviceaccount.com # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"},{"name":"gcloud.core.project","value":"customerProject"}]}}
+  role: roles/cloudtrace.agent
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Project
+    external: projects/customerProject # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"}]}}
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: testKptName-vm-policy-monitoring-viewer # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"}]}}
+spec:
+  member: serviceAccount:testKptName-vm@customerProject.iam.gserviceaccount.com # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"},{"name":"gcloud.core.project","value":"customerProject"}]}}
+  role: roles/monitoring.viewer
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Project
+    external: projects/customerProject # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"}]}}
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: testKptName-vm-policy-storage # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"}]}}
+spec:
+  member: serviceAccount:testKptName-vm@customerProject.iam.gserviceaccount.com # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"},{"name":"gcloud.core.project","value":"customerProject"}]}}
+  role: roles/storage.objectViewer
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Project
+    external: projects/customerProject # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"}]}}

tests/testdata_gcp_kpt/gcp/v2/cnrm/cluster/kf-vm-sa.yaml

@@ -0,0 +1,21 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMServiceAccount
+metadata:
+  name: testKptName-vm # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"}]}}
+  namespace: "customerProject" # {"type":"string","x-kustomize":{"setBy":"kpt","setter":{"name":"gcloud.core.project","value":"customerProject"}}}
+spec:
+  displayName: kubeflow vm service account

tests/testdata_gcp_kpt/gcp/v2/cnrm/cluster/kustomization.yaml

@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- cluster.yaml
+- kf-vm-policy.yaml
+- kf-vm-sa.yaml

tests/testdata_gcp_kpt/gcp/v2/cnrm/cluster/nodepool.yaml

@@ -0,0 +1,36 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: container.cnrm.cloud.google.com/v1beta1
+kind: ContainerNodePool
+metadata:
+  clusterName: "customerProject/testZone/testKptName" # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"gcloud.core.project","value":"customerProject"},{"name":"name","value":"testKptName"},{"name":"gcloud.compute.zone","value":"testZone"}]}}
+  name: testKptName-cpu-pool-v1 # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"}]}}
+  namespace: "customerProject" # {"type":"string","x-kustomize":{"setter":{"name":"gcloud.core.project","value":"customerProject"}}}
+spec:
+  initialNodeCount: 2
+  autoscaling:
+    minNodeCount: 2
+    maxNodeCount: 8 # {"type":"integer","x-kustomize":{"setter":{"name":"max-nodes","value":"8"}}}
+  nodeConfig:
+    machineType: n1-standard-8
+    minCpuPlatform: 'Intel Broadwell'
+    metadata:
+      disable-legacy-endpoints: "true"
+    serviceAccountRef:
+      name: testKptName-vm@customerProject.iam.gserviceaccount.com # {"type":"string","x-kustomize":{"setBy":"kpt","partialSetters":[{"name":"name","value":"testKptName"},{"name":"gcloud.core.project","value":"customerProject"}]}}
+    workloadMetadataConfig:
+      nodeMetadata: GKE_METADATA_SERVER
+  clusterRef:
+    name: testKptName # {"type":"string","x-kustomize":{"setter":{"name":"name","value":"testKptName"}}}