Documentation for password management

docs/Examples.md

@@ -18,6 +18,11 @@ deployments.
   - [Creating a LicenseMaster Using a ConfigMap](#creating-a-licensemaster-using-a-configmap)
   - [Using an External License Master](#using-an-external-license-master)
   - [Using an External Indexer Cluster](#using-an-external-indexer-cluster)
+  - [Managing global kubernetes secret object](#managing-global-kubernetes-secret-object)
+    - [Creating global kubernetes secret object](#creating-global-kubernetes-secret-object)
+    - [Reading global kubernetes secret object](#reading-global-kubernetes-secret-object)
+    - [Updating global kubernetes secret object](#updating-global-kubernetes-secret-object)
+    - [Deleting global kubernetes secret object](#deleting-global-kubernetes-secret-object)
 
 Please refer to the [Custom Resource Guide](CustomResources.md) for more
 information about the custom resources that you can use with the Splunk
@@ -39,6 +44,7 @@ metadata:
   - enterprise.splunk.com/delete-pvc
 ```
 
+Please refer [here](#managing-global-kubernetes-secret-object) for instructions on managing Splunk Secret tokens in a global kubernetes secret object.
 
 ### Indexer Clusters
 
@@ -263,6 +269,7 @@ splunk-single-standalone-0         1/1     Running   0          11m
 Similar to indexer clusters, you can easily scale search head clusters
 by just patching the `replicas` parameter.
 
+Please refer [here](#managing-global-kubernetes-secret-object) for instructions on managing Splunk Secret tokens in a global kubernetes secret object.
 
 ### Cluster Services
 
@@ -289,16 +296,17 @@ automatically created for your deployment:
 kubectl port-forward service/splunk-example-search-head-service 8000
 ```
 
-Similar to other examples, the default admin password can be obtained
-from the secrets it generated for your deployment:
+Similar to other examples, the default administrator password can be obtained
+from the global kubernetes secrets object as described here:
 
 ```
-kubectl get secret splunk-example-search-head-secrets -o jsonpath='{.data.password}' | base64 --decode
+kubectl get secret splunk-`<namespace`>-secret -o jsonpath='{.data.password}' | base64 --decode
 ```
 
 Please see [Configuring Ingress](Ingress.md) for guidance on making your
 Splunk clusters accessible outside of Kubernetes.
 
+Please refer [here](#managing-global-kubernetes-secret-object) for instructions on managing Splunk Secret tokens in a global kubernetes secret object.
 
 ### Creating a Cluster with Data Fabric Search (DFS)
 
@@ -339,6 +347,7 @@ spec:
 EOF
 ```
 
+Please refer [here](#managing-global-kubernetes-secret-object) for instructions on managing Splunk Secret tokens in a global kubernetes secret object.
 
 ### Cleaning Up
 
@@ -727,3 +736,78 @@ spec:
         secretName: splunk-cluster-master
   defaultsUrl: /mnt/cluster-master/default.yml
 ```
+## Managing global kubernetes secret object
+
+### Creating global kubernetes secret object
+
+To create the global kubernetes secret object you can use the kubectl command line tool. Here are the steps to the same:
+
+1. Retrieve the namespace in the current context using the following command `kubectl config view --minify --output 'jsonpath={..namespace}'` and make a note of it.
+If the command doesn't display an output it indicates that we are in the `default` namespace.
+
+2. Gather the values for all splunk secret tokens you want to configure on the global kubernetes secret object. All possible Splunk secret tokens along with their corresponding key strings in the global kubernetes secret object are mentioned [here](PasswordManagement.md#splunk-secret-tokens-in-the-global-secret-object).
+
+3. Create a kubernetes secret object whose name is of the format splunk-`<namespace_retrieved_above`>-secret using the command below. In this example we are creating the global kubernetes secret object with default administrator password and pass4symmkey configured.  
+`kubectl create secret generic splunk-<namespace_retrieved_above>-secret --from-literal='password=<admin_password_value>' --from-literal='pass4symmkey=<pass4symmkey_value>'`
+
+### Reading global kubernetes secret object
+
+Once created, all secret tokens in the secret object are base64 encoded. To read the global kubernetes secret object you can use the following command:
+
+`kubectl get secret splunk-<namespace_retrieved_above>-secret -o yaml`
+
+A sample global kubernetes secret object with base64 encoded values looks like:
+
+```
+kubectl get secret splunk-default-secret -o yaml
+apiVersion: v1
+data:
+  hec_token: RUJFQTE4OTMtMDI4My03RkMzLThEQTAtQ0I1RTFGQzgzMzc1
+  idxc_secret: VUY5dWpHU1I4ZmpoZlJKaWNNT2VMSUNY
+  pass4SymmKey: dkFjelZSUzJjZzFWOHZPaVRGZk9hSnYy
+  password: OHFqcnV5WFhHRFJXU1hveDdZMzY5MGRs
+  shc_secret: ZEdHWG5Ob2dzTDhWNHlocDFiYWpiclo1
+kind: Secret
+metadata:
+  creationTimestamp: "2020-10-07T19:42:07Z"
+  name: splunk-default-secret
+  namespace: default
+  ownerReferences:
+  - apiVersion: enterprise.splunk.com/v1beta1
+    controller: false
+    kind: SearchHeadCluster
+    name: example-shc
+    uid: f7264daf-4a3e-4b44-adb7-af52f45b45fe
+  resourceVersion: "11433590"
+  selfLink: /api/v1/namespaces/default/secrets/splunk-default-secret
+  uid: d6c9a59c-1acf-4482-9990-cdb0eed56e87
+type: Opaque
+```
+
+The kubectl command line tool can be used to decode the splunk secret tokens with the following command:
+
+`kubectl get secret splunk-<namespace_retrieved_above>-secret -o go-template=' {{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'`
+
+A sample global kubernetes secret object with tokens decoded looks like:
+
+```
+hec_token: EBEA1893-0283-7FC3-8DA0-CB5E1FC83375
+idxc_secret: UF9ujGSR8fjhfRJicMOeLICX
+pass4SymmKey: vAczVRS2cg1V8vOiTFfOaJv2
+password: 8qjruyXXGDRWSXox7Y3690dl
+shc_secret: dGGXnNogsL8V4yhp1bajbrZ5
+```
+
+### Updating global kubernetes secret object
+
+To update the global kubernetes secret object you can use the kubectl command line tool. Here are the steps to do the same.
+
+1. Base64 encode the plain-text value of the secret token using the following command: `echo -n <plain_text_value> | base64`
+2. Obtain the key string for the secret token you are interested in from [here](PasswordManagement.md#splunk-secret-tokens-in-the-global-secret-object). 
+3. Use the encoded value, key string/s to edit the global kubernetes secret object via a patch using the following command: `kubectl patch secret splunk-<namespace_retrieved_above>-secret -p='{"data":{"<key_string_for_secret_token>": "<encoded_value>"}}' -v=1`
+
+### Deleting global kubernetes secret object
+
+To delete the global kubernetes secret object you can use the kubectl command line tool as shown below:
+
+`kubectl delete secret splunk-<namespace_retrieved_above>-secret`

docs/PasswordManagement.md

@@ -0,0 +1,69 @@
+# Password Management
+
+- [Global kubernetes secret object](#global-kubernetes-secret-object)
+- [Default behavior of global kubernetes secret object](#default-behavior-of-global-kubernetes-secret-object)
+  - [Splunk Secret Tokens in the global secret object](#splunk-secret-tokens-in-the-global-secret-object)
+    - [HEC Token](#hec-token)
+    - [Default administrator password](#default-administrator-password)
+    - [Pass4symmkey](#pass4symmkey)
+    - [IDXC Pass4symmkey](#idxc-pass4symmkey)
+    - [SHC Pass4symmkey](#shc-pass4symmkey)
+- [Information for Splunk Enterprise administrator](#information-for-splunk-enterprise-administrator)
+- [Secrets on Docker Splunk](#secrets-on-docker-splunk)
+
+## Global kubernetes secret object
+A global kubernetes secret object acts as a global source of splunk secret tokens for all Splunk Enterprise CRs in a kubernetes namespace. It's name follows the format "splunk-`<namespace`>-secret" where `<namespace`> represents the namespace we are operating in. The contents of this object are volume mounted on all the pods within a kubernetes namespace. 
+
+This approach:
+  - Eliminates any mismatch between operator-generated secrets and admin provided secrets as all secrets are synced into a common object.
+  - Allows for dynamic adoption and modification of secrets.
+
+## Default behavior of global kubernetes secret object
+
+For a given namespace upon the creation of the first Splunk Enterprise CR, the operator checks for the existence of a global kubernetes secret object:
+
+- If the object does not exist,
+    - Creates the global kubernetes secret object with name splunk-`<namespace`>-secret
+    - Auto-generates(except for S3 specific tokens) & stores all the Splunk secret tokens into the global kubernetes secret object
+- If the object exists,
+    - Reads the configured Splunk secret tokens in the global secret object
+    - Auto-generates(except for S3 specific tokens) & stores only the missing Splunk secret tokens into the global secret object
+
+Note: Before the creation of any Splunk deployments in the kubernetes namespace, the admin can create a global kubernetes secret object with the tokens mentioned below so that the operator uses these values to deploy.
+
+### Splunk Secret Tokens in the global secret object
+The configurable Splunk Secret Tokens include:
+
+#### HEC Token
+**Key String in global kubernetes secret object**: `hec_token`  
+**Description**: hec_token is used to authenticate clients sending data into Splunk Enterprise via HTTP connections.
+
+#### Default administrator password
+**Key String in global kubernetes secret object**: `password`  
+**Description**: password refers to the default administrator password for Splunk. 
+
+#### Pass4symmkey
+**Key String in global kubernetes secret object**: `pass4symmkey`  
+**Description**: pass4symmkey is an authentication token for inter-communication within Splunk Enterprise.
+
+#### IDXC Pass4symmkey
+**Key String in global kubernetes secret object**: `idxc.secret`  
+**Description**: idxc.secret is an authentication token for inter-communication specifically for indexer clustering in Splunk Enterprise.
+
+#### SHC Pass4symmkey
+**Key String in global kubernetes secret object**: `shc.secret`  
+**Description**: shc.secret is an authentication token for inter-communication specifically for search head clustering in Splunk Enterprise.
+
+For specifications on Splunk secret tokens refer [Splunk docs](https://docs.splunk.com)
+
+For examples of performing CRUD operations on the global secrets object refer this [link](Examples.md#managing-global-kubernetes-secret-object). For more information
+on managing kubernetes secret objects refer [here](https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-kubectl/)
+
+## Information for Splunk Enterprise administrator
+- The default administrator account cannot be disabled on any Splunk Enterprise instance. The kubernetes operator uses this account to interact with all Splunk Enterprise instances in the namespace.
+- The passwords managed using the global kubernetes secret object should never be changed using Splunk Enterprise tools (CLI, UI) 
+- The default administrator account must use the global kubernetes secret object for any password changes. See [here](Examples.md#managing-global-kubernetes-secret-object)
+- After initiating a update/delete operation on the global secrets object, the operator will require time to finish setting the changes on the all Splunk Enterprise instances in the namespace during which disruption of splunk services can be expected while the secret updates are happening. A status check on all the Splunk Enterprise cluster tiers is required.
+
+## Secrets on Docker Splunk
+When Splunk Enterprise is deployed on a docker container, ansible playbooks are used to setup Splunk. Ansible playbooks interpret the environment variable SPLUNK_DEFAULTS_URL in the container as the location to read the Splunk Secret Tokens from. The tokens are used to setup Splunk Instances running on containers inside pods.