Open
Description
On GKE, upgrading with grafana persistence enabled leads to a Failed to attach volume error:
$ k version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:53:09Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:34:32Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
$ cat values-only-grafana-persistent.yaml
monitor:
grafana:
persistence:
enabled: true
influxdb:
persistence:
enabled: false
$ helm install workflow-dev/workflow --namespace deis --name deis-workflow -f values-only-grafana-persistent.yaml
...
$ # (wait until all up and running)
$ helm upgrade deis-workflow -f values-only-grafana-persistent.yaml workflow/workflow
...
$ kd get po | grep grafana
deis-monitor-grafana-1674584155-qz1pq 0/1 ContainerCreating 0 40s
deis-monitor-grafana-2030706665-cg04s 1/1 Running 0 4m
$ kd describe po deis-monitor-grafana-1674584155-qz1pq
Name: deis-monitor-grafana-1674584155-qz1pq
Namespace: deis
Node: gke-vrd-default-pool-7890514e-p65q/10.240.0.22
Start Time: Tue, 28 Mar 2017 12:10:58 -0600
Labels: app=deis-monitor-grafana
pod-template-hash=1674584155
Status: Pending
IP:
Controllers: ReplicaSet/deis-monitor-grafana-1674584155
Containers:
deis-monitor-grafana:
Container ID:
Image: quay.io/deis/grafana:v2.8.0
Image ID:
Port: 3500/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Volume Mounts:
/var/lib/grafana from grafana-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f1372 (ro)
Environment Variables:
INFLUXDB_URLS: http://$(DEIS_MONITOR_INFLUXAPI_SERVICE_HOST):$(DEIS_MONITOR_INFLUXAPI_SERVICE_PORT_TRANSPORT)
BIND_PORT: 3500
DEFAULT_USER: admin
DEFAULT_USER_PASSWORD: admin
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
grafana-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: deis-monitor-grafana
ReadOnly: false
default-token-f1372:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-f1372
QoS Class: BestEffort
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned deis-monitor-grafana-1674584155-qz1pq to gke-vrd-default-pool-7890514e-p65q
1m 14s 7 {controller-manager } Warning FailedMount Failed to attach volume "pvc-5d2452ce-13e1-11e7-b75d-42010a80013f" on node "gke-vrd-default-pool-7890514e-p65q" with: googleapi: Error 400: The disk resource 'projects/deis-sandbox/zones/us-central1-b/disks/gke-vrd-c820edac-dynam-pvc-5d2452ce-13e1-11e7-b75d-42010a80013f' is already being used by 'projects/deis-sandbox/zones/us-central1-b/instances/gke-vrd-default-pool-7890514e-0xxs'