Splunk Operator: admin password replication issue on searchheadcluster when composed of multiple pods #1554
Description
Please select the type of request
Bug
Tell us more
Describe the request
Today we use the command /opt/splunk/bin/splunk cmd splunkd rest --noauth POST /services/admin/users/admin 'password=%s' searchheadcluster.go#L379
to change the admin user password in a Search Head Cluster deployed on a Kubernetes cluster:
The issue we found out with this approach is that, in situations where the searchhead cluster is composed of multiple searchhead pods (above 5), we can have issues with the replication of the password when the local change done in the mentioned line conflicts with replication operation leaving some of the search heads with the old admin user password.
In our investigation we found out that we could use splunk edit user admin -password %s -auth admin:%s instead, running it on only one search head from the search head cluster. With this change we were able to change the admin password without any issues even on the Kubernetes clusters where the searchhead cluster is composed of multiple searchhead pods (above 5)
Expected behavior
The change of the admin password should happen without issues also when searchhead cluster is composed of multiple searchhead pods (>5)
Splunk setup on K8S
- Splunk Enterprise deployed via the Splunk Operator.
- Splunk Operator version 2.7.1
- The searchhead cluster should be composed of multiple searchhead pods (>5), I recommend to try with 10 pods.
Reproduction/Testing steps
- To trigger the reproduction of the issue the namespace password should change so that this condition in this line https://github.com/splunk/splunk-operator/blob/main/pkg/splunk/enterprise/searchheadcluster.go#L369C18-L369C25 holds true.
K8s environment
- The cluster where we run splunk is managed with Openshift version 4, but I expect the issue to be reproducible with any version of k8s (at least >1.27) both in the client and the server
Proposed changes(optional)
- I will submit a PR proposal soon that we can discuss together
K8s collector data(optional)
- Please provide data collected from the K8s collectors, if any.
Additional context(optional)
These are the documentation references we have used to investigate this issue:
- https://help.splunk.com/en/splunk-enterprise/administer/distributed-search/9.0/update-search-head-cluster-members/how-configuration-changes-propagate-across-the-search-head-cluster
- https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Changeapassword
- https://docs.splunk.com/Documentation/Splunk/7.1.2/Security/Secureyouradminaccount#Reset_a_lost_password