✨ Adding storage pool support #254

mhkarimi1383 · 2025-11-23T07:15:05Z

Allows users to create multiple storage pools and use them in Storage Class via parameters

resolves #234

mergify · 2025-11-23T07:24:21Z

🧪 CI Insights

Here's what we observed from your CI run for 8e97515.

🟢 All jobs passed!

But CI Insights is watching 👀

mhkarimi1383 · 2025-11-30T19:10:37Z

@tiagolobocastro

Can you review the structure of the configuration?

rawfile/config/model.py

mhkarimi1383 · 2025-12-07T12:48:00Z

@mergify rebase

mergify · 2025-12-07T12:48:18Z

rebase

✅ Branch has been successfully rebased

Copilot

Pull request overview

This PR adds support for multiple storage pools to the rawfile-localpv CSI driver, allowing users to configure different storage locations with individual capacity settings and filesystem defaults. The implementation includes metadata schema migration from version 5 to 6 to track which storage pool each volume belongs to, and updates the configuration model, volume management, and Helm charts to support the new multi-pool architecture.

Key Changes:

Introduced storage pool configuration with per-pool settings for path, reserved capacity, capacity override, and default filesystem
Added metadata schema version 6 that includes a storage_pool field for each volume
Updated all volume operations to use storage pool-specific paths and configurations

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 20 comments.

Show a summary per file

File	Description
rawfile/volume_schema.py	Adds migration from schema v5 to v6, adding storage_pool field to metadata
rawfile/utils/volume_manager.py	Updates volume path resolution and all operations to use storage pool parameter
rawfile/utils/task_manager.py	Updates tasks file path migration to use default storage pool
rawfile/utils/storage_pool.py	Refactors get_capacity to accept storage pool parameter and use pool-specific configuration
rawfile/utils/snapshot_manager.py	Updates snapshot operations to use storage pool from volume metadata
rawfile/utils/rawfile.py	Updates metadata functions to accept storage_pool parameter for permission updates
rawfile/rawfile_servicer.py	Adds storagePool parameter extraction from storage class and passes to volume creation
rawfile/rawfile.py	Updates COW support detection to per-pool basis and adds pool path validation
rawfile/metrics.py	Adds aggregation of capacity across all storage pools for metrics
rawfile/internal_svc.py	Updates volume expansion to use pool-specific capacity checks
rawfile/consts.py	Replaces single COW_SUPPORTED flag with COW_SUPPORT_MAP dictionary for per-pool tracking
rawfile/config/model.py	Adds StoragePool model and validates pool configuration in CSIDriverCmd
rawfile/bd2fs.py	Updates volume staging to use pool-specific default filesystem configuration
deploy/helm/rawfile-localpv/values.yaml	Adds storagePools configuration and default pool settings, marks old fields as deprecated
deploy/helm/rawfile-localpv/templates/storageclass.yaml	Adds optional storagePool parameter to storage class
deploy/helm/rawfile-localpv/templates/node-plugin/daemonset.yaml	Replaces single data-dir volume with dynamic pool volumes and adds storage pools env config
deploy/helm/rawfile-localpv/templates/_helpers.tpl	Adds helper templates for pool volumes and volume mounts
deploy/helm/rawfile-localpv/README.md	Updates documentation to reflect new storage pool configuration
.ci/deployer.sh	Updates test infrastructure paths to match new default pool structure

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

rawfile/config/model.py

rawfile/rawfile_servicer.py

rawfile/utils/volume_manager.py

rawfile/bd2fs.py

rawfile/utils/rawfile.py

rawfile/config/model.py

rawfile/utils/volume_manager.py

rawfile/bd2fs.py

rawfile/utils/snapshot_manager.py

rawfile/config/model.py

mhkarimi1383 · 2025-12-16T08:44:58Z

@mergify rebase

mergify · 2025-12-16T08:45:13Z

rebase

✅ Branch has been successfully rebased

mhkarimi1383 · 2025-12-17T08:12:53Z

What happens on upgrade from previous version? If we keep dataDirPath would we be able to handle upgrades gracefully?

Upgrade requires manual actions right now, but we can make the dataDirPath available, and if you provide it we will set that as the default pool and then will make templates to fail if both dataDirPath and storagePools are set

tiagolobocastro · 2025-12-17T09:22:28Z

What happens on upgrade from previous version? If we keep dataDirPath would we be able to handle upgrades gracefully?

Upgrade requires manual actions right now, but we can make the dataDirPath available, and if you provide it we will set that as the default pool and then will make templates to fail if both dataDirPath and storagePools are set

Any way to make it an error if user is upgrading from previous version without making any changes?

mhkarimi1383 · 2025-12-17T16:26:19Z

Any way to make it an error if user is upgrading from previous version without making any changes?

I will comment out new pools and and keep older config, So it will keep current behavior, Then I will make it to create a single default pool if they are provided and use pool configurations if they are not provided

If that's OK I will apply this

* 🚧 Adding storage pool support Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🚧 Remove use of static DATA_DIR Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Make CoW Support dynamic Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Add Pool support to helm chart Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix metric collection problem Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Node Plugin config validation Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix storage pool handling in meta patch Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix default fs handling Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Connect storage pool to the storage class in helm chart Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Fix snapshot list problem with storage pool Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Validate Storage Pool in Storage Class Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update rawfile/config/model.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/config/model.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/utils/snapshot_manager.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/bd2fs.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Use `classmethod` on field validator Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Metadata getter fallback Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/utils/rawfile.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev>

mhkarimi1383 · 2025-12-21T12:23:31Z

Any way to make it an error if user is upgrading from previous version without making any changes?

I will comment out new pools and and keep older config, So it will keep current behavior, Then I will make it to create a single default pool if they are provided and use pool configurations if they are not provided

If that's OK I will apply this

@tiagolobocastro

* 🚧 Adding storage pool support Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🚧 Remove use of static DATA_DIR Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Make CoW Support dynamic Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Add Pool support to helm chart Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix metric collection problem Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Node Plugin config validation Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix storage pool handling in meta patch Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix default fs handling Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Connect storage pool to the storage class in helm chart Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Fix snapshot list problem with storage pool Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Validate Storage Pool in Storage Class Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update rawfile/config/model.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/config/model.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/utils/snapshot_manager.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/bd2fs.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Use `classmethod` on field validator Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Metadata getter fallback Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/utils/rawfile.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev>

tiagolobocastro · 2025-12-27T18:54:10Z

Any way to make it an error if user is upgrading from previous version without making any changes?

I will comment out new pools and and keep older config, So it will keep current behavior, Then I will make it to create a single default pool if they are provided and use pool configurations if they are not provided
If that's OK I will apply this

@tiagolobocastro

sounds good to me

* 🚧 Adding storage pool support Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🚧 Remove use of static DATA_DIR Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Make CoW Support dynamic Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Add Pool support to helm chart Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix metric collection problem Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Node Plugin config validation Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix storage pool handling in meta patch Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix default fs handling Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Connect storage pool to the storage class in helm chart Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Fix snapshot list problem with storage pool Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Validate Storage Pool in Storage Class Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update rawfile/config/model.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/config/model.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/utils/snapshot_manager.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/bd2fs.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Use `classmethod` on field validator Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Metadata getter fallback Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/utils/rawfile.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev>

mhkarimi1383 · 2025-12-28T08:45:22Z

@tiagolobocastro

Done.

* 🚧 Adding storage pool support Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🚧 Remove use of static DATA_DIR Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Make CoW Support dynamic Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Add Pool support to helm chart Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix metric collection problem Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Node Plugin config validation Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix storage pool handling in meta patch Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix default fs handling Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Connect storage pool to the storage class in helm chart Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Fix snapshot list problem with storage pool Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Validate Storage Pool in Storage Class Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update rawfile/config/model.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/config/model.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/utils/snapshot_manager.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/bd2fs.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Use `classmethod` on field validator Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Metadata getter fallback Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/utils/rawfile.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev>

mhkarimi1383 · 2025-12-28T10:06:55Z

@mergify rebase

* 🚧 Adding storage pool support Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🚧 Remove use of static DATA_DIR Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Make CoW Support dynamic Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Add Pool support to helm chart Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix metric collection problem Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Node Plugin config validation Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix storage pool handling in meta patch Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * 🐛 Fix default fs handling Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✨ Connect storage pool to the storage class in helm chart Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Fix snapshot list problem with storage pool Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Validate Storage Pool in Storage Class Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update rawfile/config/model.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/config/model.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/utils/snapshot_manager.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/bd2fs.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Use `classmethod` on field validator Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * ✏️ Metadata getter fallback Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev> * Update rawfile/utils/rawfile.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Muhammed Hussain Karimi <info@karimi.dev>

mergify · 2025-12-28T10:07:01Z

rebase

✅ Branch has been successfully rebased

mhkarimi1383 · 2025-12-28T10:37:32Z

The only notice will be that this will create a storage pool called data-dir and it will connect all of the volumes to that pool and changing this will require manual actions (Since we don't have rename functionality for the Storage Pools

tiagolobocastro reviewed Dec 2, 2025

View reviewed changes

rawfile/config/model.py Outdated Show resolved Hide resolved

rawfile/config/model.py Outdated Show resolved Hide resolved

mhkarimi1383 force-pushed the storage-pool branch from fcf7f6b to bbe03f4 Compare December 2, 2025 14:24

mhkarimi1383 force-pushed the storage-pool branch 11 times, most recently from af3e911 to 013931c Compare December 7, 2025 17:55

mhkarimi1383 force-pushed the storage-pool branch 3 times, most recently from bd99bfa to 927431e Compare December 16, 2025 06:50

mhkarimi1383 marked this pull request as ready for review December 16, 2025 06:50

mhkarimi1383 requested a review from a team as a code owner December 16, 2025 06:50

Copilot AI review requested due to automatic review settings December 16, 2025 06:50

Copilot started reviewing on behalf of mhkarimi1383 December 16, 2025 06:50 View session

Copilot AI reviewed Dec 16, 2025

View reviewed changes

mhkarimi1383 force-pushed the storage-pool branch 2 times, most recently from 4ec1701 to fd31fa2 Compare December 16, 2025 08:42

mhkarimi1383 force-pushed the storage-pool branch from 14ae546 to 21cf993 Compare December 16, 2025 08:45

mhkarimi1383 force-pushed the storage-pool branch from 7c8f410 to 03d42fa Compare December 17, 2025 08:06

mhkarimi1383 force-pushed the storage-pool branch from 03d42fa to de0637a Compare December 17, 2025 18:19

mhkarimi1383 force-pushed the storage-pool branch from de0637a to 9b66816 Compare December 21, 2025 17:31

mhkarimi1383 force-pushed the storage-pool branch from 9b66816 to 9d8a307 Compare December 28, 2025 08:44

mhkarimi1383 force-pushed the storage-pool branch from 9d8a307 to ebd6944 Compare December 28, 2025 08:55

mhkarimi1383 force-pushed the storage-pool branch from ebd6944 to 9a75171 Compare December 28, 2025 09:05

mhkarimi1383 force-pushed the storage-pool branch from 9a75171 to d31f56e Compare December 28, 2025 09:20

mhkarimi1383 force-pushed the storage-pool branch from d31f56e to a12735e Compare December 28, 2025 09:31

mhkarimi1383 force-pushed the storage-pool branch from a12735e to 963b34b Compare December 28, 2025 09:37

mhkarimi1383 force-pushed the storage-pool branch from 963b34b to 46b4a73 Compare December 28, 2025 09:52

mhkarimi1383 force-pushed the storage-pool branch from 46b4a73 to 8e97515 Compare December 28, 2025 10:07

✨ Adding storage pool support #254

Are you sure you want to change the base?

✨ Adding storage pool support #254

Uh oh!

Conversation

mhkarimi1383 commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 CI Insights

🟢 All jobs passed!

Uh oh!

mhkarimi1383 commented Nov 30, 2025

Uh oh!

Uh oh!

Uh oh!

mhkarimi1383 commented Dec 7, 2025

Uh oh!

mergify bot commented Dec 7, 2025

✅ Branch has been successfully rebased

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mhkarimi1383 commented Dec 16, 2025

Uh oh!

mergify bot commented Dec 16, 2025

✅ Branch has been successfully rebased

Uh oh!

mhkarimi1383 commented Dec 17, 2025

Uh oh!

tiagolobocastro commented Dec 17, 2025

Uh oh!

mhkarimi1383 commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhkarimi1383 commented Dec 21, 2025

Uh oh!

tiagolobocastro commented Dec 27, 2025

Uh oh!

mhkarimi1383 commented Dec 28, 2025

Uh oh!

mhkarimi1383 commented Dec 28, 2025

Uh oh!

mergify bot commented Dec 28, 2025

✅ Branch has been successfully rebased

Uh oh!

mhkarimi1383 commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mhkarimi1383 commented Nov 23, 2025 •

edited

Loading

mergify bot commented Nov 23, 2025 •

edited

Loading

mhkarimi1383 commented Dec 17, 2025 •

edited

Loading

mhkarimi1383 commented Dec 28, 2025 •

edited

Loading