Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic volume resize for VMSingle/VMCluster without pods restart #161

Closed
vpedosyuk opened this issue Feb 15, 2021 · 3 comments
Closed

Dynamic volume resize for VMSingle/VMCluster without pods restart #161

vpedosyuk opened this issue Feb 15, 2021 · 3 comments
Assignees
Labels
enhancement New feature or request

Comments

@vpedosyuk
Copy link

We use Helm charts to deploy VM, sometimes we need to increase storage size. For example, this includes cases when:

  • the VM load always grows instead of being constant
  • there's a need to change the retention period for the existing cluster
  • there's a need to add some spare storage for background merges

With Helm we can't just increase PVC template size for vmstorages because of statefulset constraints in k8s. In order to increase our disks, we either add another replica pod of vmstorage or manually edit resizable PVC in GKE, which is very inconvenient for many reasons.

Some database operators have solved the issue: as I understood their operator patches PVC, then deletes the statefulset with --cascade=false so that pods remain untouched, and then creates the statefulset with renewed storage requests so that new replicas get created with new storage size.

A few examples:
elastic/cloud-on-k8s#3752
zalando/postgres-operator#958

P.S. Further improvements could be, for example, a kind of watcher that automatically adjusts PVCs depending on the "free storage threshold" that a user specifies for VMSingle/VMCluster.

@f41gh7
Copy link
Collaborator

f41gh7 commented Feb 16, 2021

Hello, it looks really interesting, need to implement it, thanks for suggestion.

@f41gh7 f41gh7 added the enhancement New feature or request label Feb 16, 2021
@f41gh7 f41gh7 self-assigned this Feb 26, 2021
f41gh7 added a commit that referenced this issue Mar 23, 2021
adds recreation for statefulset if storage changed
implements #161
@f41gh7
Copy link
Collaborator

f41gh7 commented Mar 23, 2021

Will be included to the next release.

f41gh7 added a commit that referenced this issue Mar 24, 2021
* fixes vmselect storage naming,
* adds recreation for statefulset if storage changed
* implements storage grows #161
@f41gh7
Copy link
Collaborator

f41gh7 commented Mar 29, 2021

Included to 0.12.0 release

@f41gh7 f41gh7 closed this as completed Mar 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants