Productionize the cockroachdb example a little more

Includes:
* A service for clients to use
* Readiness/liveness probes
* An extended graceful termination period
* Easy clean-up of all created resources

examples/cockroachdb/README.md

@@ -67,6 +67,25 @@ steps.
 
 Follow the steps in [minikube.sh](minikube.sh) (or simply run that file).
 
+## Accessing the database
+
+Along with our PetSet configuration, we expose a standard Kubernetes service
+that offers a load-balanced virtual IP for clients to access the database
+with. In our example, we've called this service `cockroachdb-public`.
+
+Start up a client pod and open up an interactive, (mostly) Postgres-flavor
+SQL shell using:
+
+```console
+$ kubectl run -it cockroach-client --image=cockroachdb/cockroach --restart=Never --command -- bash
+root@cockroach-client # ./cockroach sql --host cockroachdb-public
+```
+
+You can see example SQL statements for inserting and querying data in the
+included [demo script](demo.sh), but can use almost any Postgres-style SQL
+commands. Some more basic examples can be found within
+[CockroachDB's documentation](https://www.cockroachlabs.com/docs/learn-cockroachdb-sql.html).
+
 ## Simulating failures
 
 When all (or enough) nodes are up, simulate a failure like this:
@@ -75,14 +94,15 @@ When all (or enough) nodes are up, simulate a failure like this:
 kubectl exec cockroachdb-0 -- /bin/bash -c "while true; do kill 1; done"
 ```
 
-On one of the other pods, run `./cockroach sql --host $(hostname)` and use
-(mostly) Postgres-flavor SQL. The example runs with three-fold replication,
-so it can tolerate one failure of any given node at a time.
-Note also that there is a brief period of time immediately after the creation
-of the cluster during which the three-fold replication is established, and
-during which killing a node may lead to unavailability.
+You can then reconnect to the database as demonstrated above and verify
+that no data was lost. The example runs with three-fold replication, so
+it can tolerate one failure of any given node at a time. Note also that
+there is a brief period of time immediately after the creation of the
+cluster during which the three-fold replication is established, and during
+which killing a node may lead to unavailability.
 
-There is also a [demo script](demo.sh).
+The [demo script](demo.sh) gives an example of killing one instance of the
+database and ensuring the other replicas have all data that was written.
 
 ## Scaling up or down
 
@@ -91,6 +111,15 @@ volume claim first). If you ran `minikube.sh`, there's a spare volume so you
 can immediately scale up by one. Convince yourself that the new node
 immediately serves reads and writes.
 
+## Cleaning up when you're done
+
+Because all of the resources in this example have been tagged with the label `app=cockroachdb`,
+we can clean up everything that we created in one quick command using a selector on that label:
+
+```shell
+kubectl delete petsets,pods,persistentvolumes,persistentvolumeclaims,services -l app=cockroachdb
+```
+
 
 <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
 [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/examples/cockroachdb/README.md?pixel)]()

examples/cockroachdb/cockroachdb-petset.yaml

@@ -1,5 +1,27 @@
 apiVersion: v1
 kind: Service
+metadata:
+  # This service is meant to be used by clients of the database. It exposes a ClusterIP that will
+  # automatically load balance connections to the different database pods.
+  name: cockroachdb-public
+  labels:
+    app: cockroachdb
+spec:
+  ports:
+  # The main port, served by gRPC, serves Postgres-flavor SQL, internode
+  # traffic and the cli.
+  - port: 26257
+    targetPort: 26257
+    name: grpc
+  # The secondary port serves the UI as well as health and debug endpoints.
+  - port: 8080
+    targetPort: 8080
+    name: http
+  selector:
+    app: cockroachdb
+---
+apiVersion: v1
+kind: Service
 metadata:
   annotations:
     # Make sure DNS is resolvable during initialization.
@@ -8,17 +30,17 @@ metadata:
     prometheus.io/scrape: "true"
     prometheus.io/path: "_status/vars"
     prometheus.io/port: "8080"
+  # This service only exists to create DNS entries for each pet in the petset such that they can resolve
+  # each other's IP addresses. It does not create a load-balanced ClusterIP and should not be used
+  # directly by clients in most circumstances.
   name: cockroachdb
   labels:
     app: cockroachdb
 spec:
   ports:
-  # The main port, served by gRPC, serves Postgres-flavor SQL, internode
-  # traffic and the cli.
   - port: 26257
     targetPort: 26257
     name: grpc
-  # The secondary port serves the UI as well as health and debug endpoints.
   - port: 8080
     targetPort: 8080
     name: http
@@ -54,6 +76,16 @@ spec:
           name: grpc
         - containerPort: 8080
           name: http
+        livenessProbe:
+          httpGet:
+            path: /_admin/v1/health
+            port: http
+          initialDelaySeconds: 30
+        readinessProbe:
+          httpGet:
+            path: /_admin/v1/health
+            port: http
+          initialDelaySeconds: 10
         volumeMounts:
         - name: datadir
           mountPath: /cockroach/cockroach-data
@@ -84,6 +116,9 @@ spec:
               CRARGS+=("--join" "cockroachdb")
             fi
             /cockroach/cockroach ${CRARGS[*]}
+      # No pre-stop hook is required, a SIGTERM plus some time is all that's
+      # needed for graceful shutdown of a node.
+      terminationGracePeriodSeconds: 60
       volumes:
       - name: datadir
         persistentVolumeClaim:

examples/cockroachdb/minikube.sh

@@ -28,6 +28,9 @@
 
 set -exuo pipefail
 
+# Clean up anything from a prior run:
+kubectl delete petsets,pods,persistentvolumes,persistentvolumeclaims,services -l app=cockroachdb
+
 # Make persistent volumes and (correctly named) claims. We must create the
 # claims here manually even though that sounds counter-intuitive. For details
 # see https://github.com/kubernetes/contrib/pull/1295#issuecomment-230180894.
@@ -40,6 +43,7 @@ metadata:
   name: pv${i}
   labels:
     type: local
+    app: cockroachdb
 spec:
   capacity:
     storage: 1Gi