longhorn/longhorn#5832

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Don't depend on $GOPATH being set already
- Export the directory containing the tools in the $PATH, so they are
  found later on by the shell
- Export the $GOPATH, so the subshell can use it
- Show a diff of the CRDs, so it's easier to tell if things go south

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Add some skeleton code for the Object Endpoint Controller

- Add Lister and Informer:
  - Add an object endpoint lister to the datastore
  - Add an object endpoint informer to the datastore
  - Add an event-handler to the object endpoint informer

- CRD: Make properties optional

- Propagate image settings from the command line into the object store
  controller.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Make the object endpoint controller actually deploy resources when an
object endpoint API object is created.

Object Endpoint: Update CRDs

- Update CRDs:
  - Delete Image and Volume properties
  - Add StorageClass and Size properties
  - Add descriptions

- Generate accurate and useful labes for resource selection and
  identification

- Move constants to types/object_endpoint.go

- Remove debug print statements

- Add state tracking in the object endpoint resource

- Add delete hook for cleanup when the object endpoint resource is
  deleted from the cluster

- Accept the size setting in the ObjectEndpoint CRD in any of the usual
  formats for volume sizes, e.g. `10Gi`.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Implement a lifecycle state machine for the object endpoint resource.
This machine is used by the controller to track the creation of
resources and application of labels and annotations where necessary to
control the lifecycle of the dependant object correctly

Implement uninstall hooks via the uninstall controller. This ensures
that when Longhorn is uninstalled no object endpoint resources are left
lingering in the cluster.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Add internal API endpoints for management from the UI

Add internal API endpoint implementations for Create and Delete actions

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Remove diff of CRDs not modified for the purpose of the object
  endpoint controller
- Disable s3gw debug options
- Revert broken up long line
- Clarifying comments

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Rename function `handleError` to `handleErr`, which is more descriptive
of its purpose of handling the return value `err` from the synchronizing
function.
Fill out errors with descriptive messages indicating what the controller
decided to do.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Incorporate Review Comments:
  Flatten conditional scopes for easier understanding of code,
  inline function to dealing with work queue handling in the controller

- Add unit tests:
  Add simple unit tests for the object endpoint controller

- Update CRDs

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Add controlled creation of PV and Volume resources, directly binding the
PVC to the underlying storage and tightly controlling the creation
process of the filesystem. This allows for more control over filesystem
parameters, enforcing the use of an XFS with reflinks enabled.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Ensure clean shutdown by using correct ownerships in the K8s API.
Since the CSI driver is circumvented to allow for precise filesystem
parameters, the object endpoint controller needs to manage the lifecycle
of the persistent volume and longhorn volume that fulfiill the
persistent volume claim.
The lifecycles must also be syncronized with the lifecycle of the pod to
prevent storage from disappearing underneath the mount point but at the
same time the resources must be cleaned up when the object endpoint is
deleted. To achieve this, K8s owner references are utilized as follows:

 ┌────────────────┐
 │ ObjectEndpoint │
 └─┬──────────────┘
   │
   │owns
   │
   │     ┌───────────────────────┐
   ├────►│ Service               │
   │     └───────────────────────┘
   │
   │     ┌───────────────────────┐
   ├────►│ Secret                │
   │     └───────────────────────┘
   │
   │     ┌───────────────────────┐
   ├────►│ Deployment            ├──────┐
   │     └───────────────────────┘      │owns
   │                                    ▼
   │     ┌───────────────────────┐    ┌────────────────┐
   └────►│ PersistentVolumeClaim │    │ ReplicaSet     │
         └─┬─────────────────────┘    └─┬──────────────┘
           │owns                        │owns
           │                            │
           ▼                            ▼
         ┌───────────────────────┐    ┌────────────────┐
         │ LonghornVolume        │    │ Pod            │
         └───────────────────────┘    └────────────────┘
                                        ▲
                                        │
                                        │waits for
         ┌───────────────────────┐      │shutdown
         │ PersistentVolume      ├──────┘
         └───────────────────────┘

The respective ownerships propagate the deletion of the object endpoint
and allow K8s to use its finalizers to avoid accidental deletion in the
wrong order.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Add s3gw UI container command line option to the manager
- Add s3gw UI container to deployment created by the object endpoint
  controller

- Add unittests for the object endpoint state machine. The unittests
  ensure that the object endpoint controller handles all states an
  ObjectEndpoint K8s object can be in gracefully, assuming the
  respective resources are as expected

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Rename CRD from ObjectEndpoint to ObjectStore, update generated code
- Receive volume parameters in the ObjectStore CRD and propagate through
  to the static volume provisioning in the ObjectStore controller
- Implement unit testing the ObjectStore controller's state machine for
  the happy path
- Deploy a UI container along side the object gateway container

- update state machine with separate stopping/stopped states for
  temporarily shutting down an object store
- update state machine with dedicated terminatting state
- add additional properties to object store spec and API models

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Replace Credentials propery of ObjectStore type with a secret
  reference
- Replace labeling mechanism to match the existing resource labeling
  mechanisms better

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Rename informer and lister in datastore to match the other informer and
lister properties and make them less cryptic

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fixup manager API after changes to the credentials property of the
ObjectStore

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Consolidate creation functions. This simplifies resource creation as
  there is no longer two code paths through which the K8s objects may be
  created

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Simplify deletion by reducing the amount of actions the controller
  does.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fix spelling mistakes found by codespell

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fix unit tests of object store controller by adding the new states and
test them.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Only initialize the .spec.Image and .spec.uiImage fields of the
  ObjectStore if they don't already contain values
- Test that .spec.Image, .spec.UiImage and .spec.TargetState are
  correctly initialized for new object stores
- Test that new object stores successfully transition into "starting"
  state when first being picked up by the controller

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Deploy ingress for accessing the s3gw UI

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Fix calling multiple updates within the same reconciliation
- Move the controller to an event driven model, monitoring events on
  resources rather than periodically re-syncing.
- Make only the Longhorn manager which is responsible for the Volume
  responsible for the ObjectStore. This avoids multiple controllers
  posting updates to the ObjectStore resources with conflicting contents

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Add mutating webhook for initializing the object store
- Add ownership to ensure only one controller modifies an object store
  CR
- Add ingress configurations for S3

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Sync list of tls secrets to UI to enable selecting a secret when
creating an endpoint for an object store

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fix some problems in the object store controller:
- possible nil pointer dereference in handleStopping function
- more usage of datastore.GetResourceRO functions, in cases where the
  returned object is not modified
- less lookups in the enqueue*() functions

Frontend API:
- Add endpoints for listing TLS secrets
- Fix object store creation never having any endpoints

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Rebase on master and fixup broken things

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fix review remarks

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Remove ingres for the object store UI. This is no longer needed when
using the Longhorn UI's nginx as reverse proxy.

Fix deleting an object store by removing the finalizer without waiting
on resources to be deleted. Since OwnerReferences are used for automatic
cleanup, the finalizer can be removed immediately.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fix websocket controller:
  Relay secret, object store and settings information as expected by
  the UI frontend and as used by the object store tab in the UI

Error handling:
  Remove custom error variables from the object store controller, since
  in Go, errors created with the errors.New function can not be compared
  with the errors.Is function as expected
  This mechanism is replaced by just wrapping normal errors with
  errors.Wrapf

Error handling:
  Fix some conditions in checkDeployment. When a deployment is scaled
  up, there is a brief period where it has 0 replicas and 0 unavailable
  replicas. Therefore it is insufficient to just check the count of
  unavailable replicas to make sure the deployment is ready, the total
  count of replicas has to checked too.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Expose size and usage information of the longhorn volume associated with
an object store via the internal API to the UI. This allows the Longhorn
UI to receive information about the actual size and free space in an
object store.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fix event trigger for PVC events. This had misakenly been wired up to
call the event handler for Service events, but now it's fixed to call
the right event handler.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fix creating a secret when given credentials from the UI. Instead of
over-writing an existing secret or similar antics, generate a name
without a conflict for a new secret to use.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fix volume expansion:
- ensure a volume expansion is only tried when the new size is larger
  than the old size
- propagate the new size to the Longhorn volume, thereby expanding it,
  when an object store is updated

Fix update:
- Propagate the container image from the UI to the deployment, thereby
  allowing users to update the s3gw container images from the Longhorn
  UI even for existing object stores and even to newer versions than
  those that are originally shipped

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Fix s3gw status page container port
- Force setting the `.spec.image` and `.spec.uiImage` properties of the
  object store via webhook on creation
- Don't transition the object store to error state when creating
  resources fails, instead retry
- Check PV and Longhorn Volume health
- Don't check service health as a K8s Service resource doesn't have any
  status indicating healthyness or not. It either exists or not,
  readiness is dealt with on the deployment/pod level

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Update CRDs, move size property under .spec, rename .spec.storage to
  .spec.volumeParameters
- Improve webhook to make storage attributes immutable if they don't
  also change the backing volume
- Improve error handling in the controller

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Update CRDs
Safe image modifications when updating images

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fix secret creation with logic that supports reclaiming managed secrets
and errors out when user-created secrets are conflicting with
to-be-created ones

Adapt the kubernetes config map controller to be able to create and
manage multiple storage classes.

Add default settings to allow the kubernetes config map controller to
maanger the stub storage class for the volumes of object stores

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Stop propagating image settings to the object store controller. Applying
default image settings to object stores is handled in the webhooks.
Therefore the object store controller no longer needs access to this
information, besides it would have to be fetched from the appropriate
setting anyways.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Fix the race condition on start of a new object store. As long as there
is just the object store and no longhorn volume yet some controller
needs to be responsible for the object store. Usually it would be the
one responsible for the volume, this doesn't exist yet and there are no
provisions to track the owner another way. To fix this, just let the
controller on the first node handle it.

Fix error handling by setting the object store state to error only once
and otherwise propagate errors up to the reconcile function.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Adjust container launch options for the s3gw to match the new launch
  options for s3gw v0.23.0 and up

- Cleanup access credential secrets when they have been created through
  the longhorn manager and the object store is deleted

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Fix potential nil pointer dereference when receiving API errors other
  than ErrNotFound

- Export Object Store size specs in the prometheus exporter

- Send the number of deployed object stores to the telemetry endpoint
  (if sending telemetry is enabled)
  longhorn/longhorn#6720

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Add admission webhook for validating object store resources

- Force foreground cascading deleteion of object stores. This ensures
  that the volume and PV are cleaned up by the object store controller
  and resources aren't leaked before removing the finalizer on the
  object store.

- Consolidate starting state. The object store's state is only set to
  starting in the `initializeObjectStore` function, not whenever
  resources are created

- Error conditions: All errors except server-side API errors will cause
  the object store state to become `error`.
  There is only one place in the code where the state can be set to
  `error`, which is at the end of the reconcile function.

- Telemetry setting: Depending on the Longhorn setting allowing for
  sending telemetry, the telemetry for the s3gw is switchen on or off.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Reorder the health checks during the creation of an object store

When creating an object store, the responsibility for managing the
resources making up that object store rests with the controller that's
also responsible for the longhorn volume.
There is a brief moment while the object store is starting when the
information which controller is responsible for the longhorn volume has
not yet been propagated to all controllers. Therefore the controller
must first wait until the longhorn volume is healthy and has an owner
before it can determine for sure if it's responsible or not. Only then
the other resources can be created without running into race conditions
with other controller who think they are responsible.

- Clean up utility functions and constants

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Rebase on master and update vendored modules

Fix logic in test script. To ensure only the necessary unit tests are
executed (or of nothing has been changed, all unit tests are executed),
the logic needs to make sure not to interprete vendored modules as
changed source since it will fail to find unit tests for them.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Simplify deletes by removing preceeding gets and dealing with errors.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>