Skip to content
This repository has been archived by the owner on Apr 4, 2023. It is now read-only.

Commit

Permalink
Add CreateNodePool and ScaleOut actions for Cassandra
Browse files Browse the repository at this point in the history
* These become the only changes supported by the Cassandra controller.
* ScaleIn and CassandraUpgrade actions will be implemented in followup branches.
* Some initial documentation on supported configuration changes.

Fixes: #253
  • Loading branch information
wallrj committed Mar 27, 2018
1 parent ff7e744 commit 36b773f
Show file tree
Hide file tree
Showing 15 changed files with 687 additions and 67 deletions.
52 changes: 52 additions & 0 deletions docs/cassandra.rst
Original file line number Diff line number Diff line change
Expand Up @@ -176,3 +176,55 @@ For example, you could use the `Datastax Python driver <http://datastax.github.i
depends on the DNS server and operating system configuration.

.. include:: supplementary-resources.rst

The Life Cycle of a Navigator Cassandra Cluster
-----------------------------------------------

Changes to the configuration of an established Cassandra cluster must be carefully sequenced in order to maintain the health of the cluster.
So Navigator is conservative about the configuration changes that it supports.

Here are the configuration changes that are supported and the configuration changes which are not yet supported.

Supported Configuration Changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Navigator supports the following changes to a Cassandra cluster:

* :ref:`create-cluster-cassandra`: Add all initially configured node pools and nodes.
* :ref:`scale-out-cassandra`: Increase ``CassandraCluster.Spec.NodePools[0].Replicas`` to add more C* nodes to a ``nodepool``.

Navigator does not currently support any other changes to the Cassandra cluster configuration.

Unsupported Configuration Changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The following configuration changes are not currently supported but will be supported in the near future:

* Minor Upgrade: Trigger a rolling Cassandra upgrade by increasing the minor and / or patch components of ``CassandraCluster.Spec.Version``.
* Scale In: Decrease ``CassandraCluster.Spec.NodePools[0].Replicas`` to remove C* nodes from a ``nodepool``.

The following configuration changes are not currently supported:
* Add Rack: Add a ``nodepool`` for a new rack.
* Remove Rack: Remove a ``nodepool``.
* Add Data Center: Add a ``nodepool`` for a new data center.
* Remove Data Center: Remove all the ``nodepools`` in a data center.
* Major Upgrade: Upgrade to a new major Cassandra version.

.. _create-cluster-cassandra:

Create Cluster
~~~~~~~~~~~~~~

When you first create a ``CassandraCluster`` resource, Navigator will add nodes, one at a time,
in order of ``NodePool`` and according to the process described in :ref:`scale-out-cassandra` (below).
The order of node creation is determined by the order of the entries in the ``CassandraCluster.Spec.NodePools`` list.
You can look at ``CassandraCluster.Status.NodePools`` to see the current state.

.. _scale-out-cassandra:

Scale Out
~~~~~~~~~

When you first create a cluster or when you increment the ``CassandraCluster.Spec.NodePools[i].ReplicaCount``,
Navigator will add C* nodes, one at a time, until the desired number of nodes is reached.
You can look at ``CassandraCluster.Status.NodePools[<nodepoolname>].ReadyReplicas`` to see the current number of healthy C* nodes in each ``nodepool``.
14 changes: 13 additions & 1 deletion hack/e2e.sh
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,12 @@ function test_cassandracluster() {
fail_test "Failed to create cassandracluster"
fi

kubectl get cassandraclusters -n "${namespace}" -o yaml
# A NodePool is created
if ! retry TIMEOUT=300 kube_event_exists "${namespace}" \
"navigator-controller:CassandraCluster:Normal:CreateNodePool"
then
fail_test "A CreateNodePool event was not recorded"
fi

# A Pilot is elected leader
if ! retry TIMEOUT=300 kube_event_exists "${namespace}" \
Expand Down Expand Up @@ -345,6 +350,13 @@ function test_cassandracluster() {
'$NAVIGATOR_IMAGE_REPOSITORY:$NAVIGATOR_IMAGE_TAG:$NAVIGATOR_IMAGE_PULLPOLICY:$CASS_NAME:$CASS_REPLICAS:$CASS_VERSION' \
< "${SCRIPT_DIR}/testdata/cass-cluster-test.template.yaml")

# The NodePool is scaled out
if ! retry TIMEOUT=300 kube_event_exists "${namespace}" \
"navigator-controller:CassandraCluster:Normal:ScaleOut"
then
fail_test "A ScaleOut event was not recorded"
fi

if ! retry TIMEOUT=300 stdout_equals 2 kubectl \
--namespace "${namespace}" \
get cassandracluster \
Expand Down
25 changes: 25 additions & 0 deletions internal/test/util/generate/generate.go
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,28 @@ func StatefulSet(c StatefulSetConfig) *apps.StatefulSet {
},
}
}

type CassandraClusterConfig struct {
Name, Namespace string
}

func CassandraCluster(c CassandraClusterConfig) *v1alpha1.CassandraCluster {
return &v1alpha1.CassandraCluster{
ObjectMeta: metav1.ObjectMeta{
Name: c.Name,
Namespace: c.Namespace,
},
}
}

type CassandraClusterNodePoolConfig struct {
Name string
Replicas int32
}

func CassandraClusterNodePool(c CassandraClusterNodePoolConfig) *v1alpha1.CassandraClusterNodePool {
return &v1alpha1.CassandraClusterNodePool{
Name: c.Name,
Replicas: c.Replicas,
}
}
39 changes: 39 additions & 0 deletions pkg/controllers/cassandra/actions/create_nodepool.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
package actions

import (
corev1 "k8s.io/api/core/v1"
k8sErrors "k8s.io/apimachinery/pkg/api/errors"

"github.com/jetstack/navigator/pkg/apis/navigator/v1alpha1"
"github.com/jetstack/navigator/pkg/controllers"
"github.com/jetstack/navigator/pkg/controllers/cassandra/nodepool"
)

type CreateNodePool struct {
Cluster *v1alpha1.CassandraCluster
NodePool *v1alpha1.CassandraClusterNodePool
}

var _ controllers.Action = &CreateNodePool{}

func (a *CreateNodePool) Name() string {
return "CreateNodePool"
}

func (a *CreateNodePool) Execute(s *controllers.State) error {
ss := nodepool.StatefulSetForCluster(a.Cluster, a.NodePool)
_, err := s.Clientset.AppsV1beta1().StatefulSets(ss.Namespace).Create(ss)
if k8sErrors.IsAlreadyExists(err) {
return nil
}
if err != nil {
return err
}
s.Recorder.Eventf(
a.Cluster,
corev1.EventTypeNormal,
a.Name(),
"CreateNodePool: Name=%q", a.NodePool.Name,
)
return nil
}
91 changes: 91 additions & 0 deletions pkg/controllers/cassandra/actions/create_nodepool_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
package actions_test

import (
"testing"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"

"github.com/jetstack/navigator/internal/test/unit/framework"
"github.com/jetstack/navigator/internal/test/util/generate"
"github.com/jetstack/navigator/pkg/controllers/cassandra/actions"
)

func TestCreateNodePool(t *testing.T) {
type testT struct {
kubeObjects []runtime.Object
cluster generate.CassandraClusterConfig
nodePool generate.CassandraClusterNodePoolConfig
expectedStatefulSet *generate.StatefulSetConfig
expectedErr bool
}
tests := map[string]testT{
"A statefulset is created if one does not already exist": {
cluster: generate.CassandraClusterConfig{
Name: "cluster1",
Namespace: "ns1",
},
nodePool: generate.CassandraClusterNodePoolConfig{
Name: "pool1",
},
expectedStatefulSet: &generate.StatefulSetConfig{
Name: "cass-cluster1-pool1",
Namespace: "ns1",
},
},
"Idempotent: CreateNodePool can be executed again without error": {
kubeObjects: []runtime.Object{
generate.StatefulSet(
generate.StatefulSetConfig{
Name: "cass-cluster1-pool1",
Namespace: "ns1",
},
),
},
cluster: generate.CassandraClusterConfig{Name: "cluster1", Namespace: "ns1"},
nodePool: generate.CassandraClusterNodePoolConfig{
Name: "pool1",
},
expectedStatefulSet: &generate.StatefulSetConfig{
Name: "cass-cluster1-pool1",
Namespace: "ns1",
},
expectedErr: false,
},
}

for name, test := range tests {
t.Run(
name,
func(t *testing.T) {
fixture := &framework.StateFixture{
T: t,
KubeObjects: test.kubeObjects,
}
fixture.Start()
defer fixture.Stop()
state := fixture.State()
a := &actions.CreateNodePool{
Cluster: generate.CassandraCluster(test.cluster),
NodePool: generate.CassandraClusterNodePool(test.nodePool),
}
err := a.Execute(state)
if !test.expectedErr && err != nil {
t.Errorf("Unexpected error: %s", err)
}
if test.expectedErr && err == nil {
t.Errorf("Expected an error")
}
if test.expectedStatefulSet != nil {
_, err = fixture.KubeClient().
AppsV1beta1().
StatefulSets(test.expectedStatefulSet.Namespace).
Get(test.expectedStatefulSet.Name, metav1.GetOptions{})
if err != nil {
t.Fatalf("Unexpected error retrieving statefulset: %v", err)
}
}
},
)
}
}
59 changes: 59 additions & 0 deletions pkg/controllers/cassandra/actions/scaleout.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
package actions

import (
corev1 "k8s.io/api/core/v1"

"github.com/golang/glog"

"github.com/jetstack/navigator/pkg/apis/navigator/v1alpha1"
"github.com/jetstack/navigator/pkg/controllers"
"github.com/jetstack/navigator/pkg/controllers/cassandra/nodepool"
"github.com/jetstack/navigator/pkg/util/ptr"
)

type ScaleOut struct {
Cluster *v1alpha1.CassandraCluster
NodePool *v1alpha1.CassandraClusterNodePool
}

var _ controllers.Action = &ScaleOut{}

func (a *ScaleOut) Name() string {
return "ScaleOut"
}

func (a *ScaleOut) Execute(s *controllers.State) error {
baseSet := nodepool.StatefulSetForCluster(a.Cluster, a.NodePool)
existingSet, err := s.StatefulSetLister.
StatefulSets(baseSet.Namespace).Get(baseSet.Name)
if err != nil {
return err
}
newSet := existingSet.DeepCopy()
if *existingSet.Spec.Replicas == a.NodePool.Replicas {
return nil
}
if *existingSet.Spec.Replicas > a.NodePool.Replicas {
glog.Errorf(
"ScaleOut error:"+
"The StatefulSet.Spec.Replicas value (%d) "+
"is greater than the desired value (%d)",
*existingSet.Spec.Replicas, a.NodePool.Replicas,
)
return nil
}
newSet.Spec.Replicas = ptr.Int32(*newSet.Spec.Replicas + 1)
_, err = s.Clientset.AppsV1beta1().
StatefulSets(newSet.Namespace).Update(newSet)
if err != nil {
return err
}
s.Recorder.Eventf(
a.Cluster,
corev1.EventTypeNormal,
a.Name(),
"ScaleOut: NodePool=%q, ReplicaCount=%d, TargetReplicaCount=%d",
a.NodePool.Name, *newSet.Spec.Replicas, a.NodePool.Replicas,
)
return nil
}
Loading

0 comments on commit 36b773f

Please sign in to comment.