-
Notifications
You must be signed in to change notification settings - Fork 147
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Object Storage (LEP 20230430) #2136
Commits on Nov 28, 2023
-
longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 9b58eef - Browse repository at this point
Copy the full SHA 9b58eefView commit details -
improve code generation script
- Don't depend on $GOPATH being set already - Export the directory containing the tools in the $PATH, so they are found later on by the shell - Export the $GOPATH, so the subshell can use it - Show a diff of the CRDs, so it's easier to tell if things go south Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for bd79b49 - Browse repository at this point
Copy the full SHA bd79b49View commit details -
Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for e20c3cf - Browse repository at this point
Copy the full SHA e20c3cfView commit details -
object store: Add controller skeleton
- Add some skeleton code for the Object Endpoint Controller - Add Lister and Informer: - Add an object endpoint lister to the datastore - Add an object endpoint informer to the datastore - Add an event-handler to the object endpoint informer - CRD: Make properties optional - Propagate image settings from the command line into the object store controller. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com> Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for c86937b - Browse repository at this point
Copy the full SHA c86937bView commit details -
object store: deploy resources
Make the object endpoint controller actually deploy resources when an object endpoint API object is created. Object Endpoint: Update CRDs - Update CRDs: - Delete Image and Volume properties - Add StorageClass and Size properties - Add descriptions - Generate accurate and useful labes for resource selection and identification - Move constants to types/object_endpoint.go - Remove debug print statements - Add state tracking in the object endpoint resource - Add delete hook for cleanup when the object endpoint resource is deleted from the cluster - Accept the size setting in the ObjectEndpoint CRD in any of the usual formats for volume sizes, e.g. `10Gi`. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for c17f81b - Browse repository at this point
Copy the full SHA c17f81bView commit details -
object store: Implement Delete
Implement a lifecycle state machine for the object endpoint resource. This machine is used by the controller to track the creation of resources and application of labels and annotations where necessary to control the lifecycle of the dependant object correctly Implement uninstall hooks via the uninstall controller. This ensures that when Longhorn is uninstalled no object endpoint resources are left lingering in the cluster. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 2f371cd - Browse repository at this point
Copy the full SHA 2f371cdView commit details -
object store: Add internal API endpoints
Add internal API endpoints for management from the UI Add internal API endpoint implementations for Create and Delete actions Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 919f962 - Browse repository at this point
Copy the full SHA 919f962View commit details -
object endpoint controller: address review
- Remove diff of CRDs not modified for the purpose of the object endpoint controller - Disable s3gw debug options - Revert broken up long line - Clarifying comments Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 84d7447 - Browse repository at this point
Copy the full SHA 84d7447View commit details -
object endpoint: error handling
Rename function `handleError` to `handleErr`, which is more descriptive of its purpose of handling the return value `err` from the synchronizing function. Fill out errors with descriptive messages indicating what the controller decided to do. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 7b387b5 - Browse repository at this point
Copy the full SHA 7b387b5View commit details -
Object Endpoint: Unittests, CRDs, Review
- Incorporate Review Comments: Flatten conditional scopes for easier understanding of code, inline function to dealing with work queue handling in the controller - Add unit tests: Add simple unit tests for the object endpoint controller - Update CRDs Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 91d5cef - Browse repository at this point
Copy the full SHA 91d5cefView commit details -
object endpoint: manage filesystem parameters
Add controlled creation of PV and Volume resources, directly binding the PVC to the underlying storage and tightly controlling the creation process of the filesystem. This allows for more control over filesystem parameters, enforcing the use of an XFS with reflinks enabled. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 5d40040 - Browse repository at this point
Copy the full SHA 5d40040View commit details -
object endpoint: K8s owner relationships
Ensure clean shutdown by using correct ownerships in the K8s API. Since the CSI driver is circumvented to allow for precise filesystem parameters, the object endpoint controller needs to manage the lifecycle of the persistent volume and longhorn volume that fulfiill the persistent volume claim. The lifecycles must also be syncronized with the lifecycle of the pod to prevent storage from disappearing underneath the mount point but at the same time the resources must be cleaned up when the object endpoint is deleted. To achieve this, K8s owner references are utilized as follows: ┌────────────────┐ │ ObjectEndpoint │ └─┬──────────────┘ │ │owns │ │ ┌───────────────────────┐ ├────►│ Service │ │ └───────────────────────┘ │ │ ┌───────────────────────┐ ├────►│ Secret │ │ └───────────────────────┘ │ │ ┌───────────────────────┐ ├────►│ Deployment ├──────┐ │ └───────────────────────┘ │owns │ ▼ │ ┌───────────────────────┐ ┌────────────────┐ └────►│ PersistentVolumeClaim │ │ ReplicaSet │ └─┬─────────────────────┘ └─┬──────────────┘ │owns │owns │ │ ▼ ▼ ┌───────────────────────┐ ┌────────────────┐ │ LonghornVolume │ │ Pod │ └───────────────────────┘ └────────────────┘ ▲ │ │waits for ┌───────────────────────┐ │shutdown │ PersistentVolume ├──────┘ └───────────────────────┘ The respective ownerships propagate the deletion of the object endpoint and allow K8s to use its finalizers to avoid accidental deletion in the wrong order. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for ed333cc - Browse repository at this point
Copy the full SHA ed333ccView commit details -
object endpoint: configure s3gw ui container
- Add s3gw UI container command line option to the manager - Add s3gw UI container to deployment created by the object endpoint controller - Add unittests for the object endpoint state machine. The unittests ensure that the object endpoint controller handles all states an ObjectEndpoint K8s object can be in gracefully, assuming the respective resources are as expected Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for c9876e3 - Browse repository at this point
Copy the full SHA c9876e3View commit details -
object store: static provisioning, tests, ui
- Rename CRD from ObjectEndpoint to ObjectStore, update generated code - Receive volume parameters in the ObjectStore CRD and propagate through to the static volume provisioning in the ObjectStore controller - Implement unit testing the ObjectStore controller's state machine for the happy path - Deploy a UI container along side the object gateway container - update state machine with separate stopping/stopped states for temporarily shutting down an object store - update state machine with dedicated terminatting state - add additional properties to object store spec and API models Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for da08917 - Browse repository at this point
Copy the full SHA da08917View commit details -
object store: expose UI, volume fixes
Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for ef38d5f - Browse repository at this point
Copy the full SHA ef38d5fView commit details -
object store: add struct fields for backup
Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 7095924 - Browse repository at this point
Copy the full SHA 7095924View commit details -
object store: replace credentials and labels
- Replace Credentials propery of ObjectStore type with a secret reference - Replace labeling mechanism to match the existing resource labeling mechanisms better Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 16f3b6a - Browse repository at this point
Copy the full SHA 16f3b6aView commit details -
object store: rename informer and lister
Rename informer and lister in datastore to match the other informer and lister properties and make them less cryptic Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for ed4a92d - Browse repository at this point
Copy the full SHA ed4a92dView commit details -
object store: Fixup manager API
Fixup manager API after changes to the credentials property of the ObjectStore Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for bac0280 - Browse repository at this point
Copy the full SHA bac0280View commit details -
object store: consolidate creation functions
- Consolidate creation functions. This simplifies resource creation as there is no longer two code paths through which the K8s objects may be created Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 51bbe25 - Browse repository at this point
Copy the full SHA 51bbe25View commit details -
object store: simplify deletion
- Simplify deletion by reducing the amount of actions the controller does. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for f7df6f0 - Browse repository at this point
Copy the full SHA f7df6f0View commit details -
Fix spelling mistakes found by codespell Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 56633a7 - Browse repository at this point
Copy the full SHA 56633a7View commit details -
Fix unit tests of object store controller by adding the new states and test them. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 3f4216c - Browse repository at this point
Copy the full SHA 3f4216cView commit details -
object store: unit test initialization
- Only initialize the .spec.Image and .spec.uiImage fields of the ObjectStore if they don't already contain values - Test that .spec.Image, .spec.UiImage and .spec.TargetState are correctly initialized for new object stores - Test that new object stores successfully transition into "starting" state when first being picked up by the controller Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for ac4ba92 - Browse repository at this point
Copy the full SHA ac4ba92View commit details -
Deploy ingress for accessing the s3gw UI Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for c0ed496 - Browse repository at this point
Copy the full SHA c0ed496View commit details -
object store: Move to an event driven model, fixes
- Fix calling multiple updates within the same reconciliation - Move the controller to an event driven model, monitoring events on resources rather than periodically re-syncing. - Make only the Longhorn manager which is responsible for the Volume responsible for the ObjectStore. This avoids multiple controllers posting updates to the ObjectStore resources with conflicting contents Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for c5125e1 - Browse repository at this point
Copy the full SHA c5125e1View commit details -
object store: webhook, ownership, ingresses
- Add mutating webhook for initializing the object store - Add ownership to ensure only one controller modifies an object store CR - Add ingress configurations for S3 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 7162d79 - Browse repository at this point
Copy the full SHA 7162d79View commit details -
object store: sync list of tls secrets
Sync list of tls secrets to UI to enable selecting a secret when creating an endpoint for an object store Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for e41dddb - Browse repository at this point
Copy the full SHA e41dddbView commit details -
object store: fixes, secrets API
Fix some problems in the object store controller: - possible nil pointer dereference in handleStopping function - more usage of datastore.GetResourceRO functions, in cases where the returned object is not modified - less lookups in the enqueue*() functions Frontend API: - Add endpoints for listing TLS secrets - Fix object store creation never having any endpoints Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 57195cd - Browse repository at this point
Copy the full SHA 57195cdView commit details -
Rebase on master and fixup broken things Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 833d9ab - Browse repository at this point
Copy the full SHA 833d9abView commit details -
object store: fixes for review
Fix review remarks Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 8dfdfb4 - Browse repository at this point
Copy the full SHA 8dfdfb4View commit details -
object store: fix deletes, remove ingress
Remove ingres for the object store UI. This is no longer needed when using the Longhorn UI's nginx as reverse proxy. Fix deleting an object store by removing the finalizer without waiting on resources to be deleted. Since OwnerReferences are used for automatic cleanup, the finalizer can be removed immediately. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 3e915bb - Browse repository at this point
Copy the full SHA 3e915bbView commit details -
object store: web socket fixes, error handling
Fix websocket controller: Relay secret, object store and settings information as expected by the UI frontend and as used by the object store tab in the UI Error handling: Remove custom error variables from the object store controller, since in Go, errors created with the errors.New function can not be compared with the errors.Is function as expected This mechanism is replaced by just wrapping normal errors with errors.Wrapf Error handling: Fix some conditions in checkDeployment. When a deployment is scaled up, there is a brief period where it has 0 replicas and 0 unavailable replicas. Therefore it is insufficient to just check the count of unavailable replicas to make sure the deployment is ready, the total count of replicas has to checked too. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 72e07be - Browse repository at this point
Copy the full SHA 72e07beView commit details -
object store: usage information to the UI
Expose size and usage information of the longhorn volume associated with an object store via the internal API to the UI. This allows the Longhorn UI to receive information about the actual size and free space in an object store. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 8e1fc25 - Browse repository at this point
Copy the full SHA 8e1fc25View commit details -
object store: fix event trigger
Fix event trigger for PVC events. This had misakenly been wired up to call the event handler for Service events, but now it's fixed to call the right event handler. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 9bf594a - Browse repository at this point
Copy the full SHA 9bf594aView commit details -
object store: fix secret creation
Fix creating a secret when given credentials from the UI. Instead of over-writing an existing secret or similar antics, generate a name without a conflict for a new secret to use. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for fad0bd0 - Browse repository at this point
Copy the full SHA fad0bd0View commit details -
object store: fix volume expansion, update
Fix volume expansion: - ensure a volume expansion is only tried when the new size is larger than the old size - propagate the new size to the Longhorn volume, thereby expanding it, when an object store is updated Fix update: - Propagate the container image from the UI to the deployment, thereby allowing users to update the s3gw container images from the Longhorn UI even for existing object stores and even to newer versions than those that are originally shipped Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 87ae4b5 - Browse repository at this point
Copy the full SHA 87ae4b5View commit details -
- Fix s3gw status page container port - Force setting the `.spec.image` and `.spec.uiImage` properties of the object store via webhook on creation - Don't transition the object store to error state when creating resources fails, instead retry - Check PV and Longhorn Volume health - Don't check service health as a K8s Service resource doesn't have any status indicating healthyness or not. It either exists or not, readiness is dealt with on the deployment/pod level Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for c905be8 - Browse repository at this point
Copy the full SHA c905be8View commit details -
object store: update CRDs, webhooks controller
- Update CRDs, move size property under .spec, rename .spec.storage to .spec.volumeParameters - Improve webhook to make storage attributes immutable if they don't also change the backing volume - Improve error handling in the controller Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 95899d5 - Browse repository at this point
Copy the full SHA 95899d5View commit details -
object store: safe image modifications
Update CRDs Safe image modifications when updating images Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 53750c6 - Browse repository at this point
Copy the full SHA 53750c6View commit details -
object store: fix secret creation, sc controller
Fix secret creation with logic that supports reclaiming managed secrets and errors out when user-created secrets are conflicting with to-be-created ones Adapt the kubernetes config map controller to be able to create and manage multiple storage classes. Add default settings to allow the kubernetes config map controller to maanger the stub storage class for the volumes of object stores Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for a7689ec - Browse repository at this point
Copy the full SHA a7689ecView commit details -
object store: stop propagating image settings
Stop propagating image settings to the object store controller. Applying default image settings to object stores is handled in the webhooks. Therefore the object store controller no longer needs access to this information, besides it would have to be fetched from the appropriate setting anyways. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 11f5677 - Browse repository at this point
Copy the full SHA 11f5677View commit details -
object store: fix race condition, error handling
Fix the race condition on start of a new object store. As long as there is just the object store and no longhorn volume yet some controller needs to be responsible for the object store. Usually it would be the one responsible for the volume, this doesn't exist yet and there are no provisions to track the owner another way. To fix this, just let the controller on the first node handle it. Fix error handling by setting the object store state to error only once and otherwise propagate errors up to the reconcile function. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 32358c6 - Browse repository at this point
Copy the full SHA 32358c6View commit details -
Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 7065db2 - Browse repository at this point
Copy the full SHA 7065db2View commit details -
object store: launch options, cleanup secrets
- Adjust container launch options for the s3gw to match the new launch options for s3gw v0.23.0 and up - Cleanup access credential secrets when they have been created through the longhorn manager and the object store is deleted Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 0f7dae7 - Browse repository at this point
Copy the full SHA 0f7dae7View commit details -
object store: fix, metrics, telemetry
- Fix potential nil pointer dereference when receiving API errors other than ErrNotFound - Export Object Store size specs in the prometheus exporter - Send the number of deployed object stores to the telemetry endpoint (if sending telemetry is enabled) longhorn/longhorn#6720 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for a55f3e5 - Browse repository at this point
Copy the full SHA a55f3e5View commit details -
object store: foreground deletion, validation
- Add admission webhook for validating object store resources - Force foreground cascading deleteion of object stores. This ensures that the volume and PV are cleaned up by the object store controller and resources aren't leaked before removing the finalizer on the object store. - Consolidate starting state. The object store's state is only set to starting in the `initializeObjectStore` function, not whenever resources are created - Error conditions: All errors except server-side API errors will cause the object store state to become `error`. There is only one place in the code where the state can be set to `error`, which is at the end of the reconcile function. - Telemetry setting: Depending on the Longhorn setting allowing for sending telemetry, the telemetry for the s3gw is switchen on or off. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 519ddad - Browse repository at this point
Copy the full SHA 519ddadView commit details -
object store: reorder checks ; clean up
- Reorder the health checks during the creation of an object store When creating an object store, the responsibility for managing the resources making up that object store rests with the controller that's also responsible for the longhorn volume. There is a brief moment while the object store is starting when the information which controller is responsible for the longhorn volume has not yet been propagated to all controllers. Therefore the controller must first wait until the longhorn volume is healthy and has an owner before it can determine for sure if it's responsible or not. Only then the other resources can be created without running into race conditions with other controller who think they are responsible. - Clean up utility functions and constants Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 5a6d625 - Browse repository at this point
Copy the full SHA 5a6d625View commit details
Commits on Nov 29, 2023
-
object store: rebase and update vendored modules
Rebase on master and update vendored modules Fix logic in test script. To ensure only the necessary unit tests are executed (or of nothing has been changed, all unit tests are executed), the logic needs to make sure not to interprete vendored modules as changed source since it will fail to find unit tests for them. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for 1a93355 - Browse repository at this point
Copy the full SHA 1a93355View commit details
Commits on Dec 1, 2023
-
object store: simplify deletes
Simplify deletes by removing preceeding gets and dealing with errors. Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>
Configuration menu - View commit details
-
Copy full SHA for cbaa80e - Browse repository at this point
Copy the full SHA cbaa80eView commit details