Consider options for persisting metadata

[gman0]

Currently, both RBD and CephFS CSI plugins suffer from relying on metadata stored locally on nodes - this will cause problems if e.g. the node goes down and the plugin gets relocated, losing the persisted state. This issue is intended to open a discussion and weigh the options to solve this problem.

[clkao]

What are the current metadata stored that are supposed to be persistent and are not node-specific, for rbd and cephfs, respectively? from the linked PR the rbd snapshot info seems to be one of them.

[gman0]

Yes, snapshot info and both rbd and cephfs CSI plugins store volume attributes.

[gman0]

I'm playing around with having etcd as a metadata store. Is it ok to have it as a dependency @rootfs ?

[dillaman]

Note that RBD allows you to associate arbitrary key/value pairs to images via "rbd image-meta get/set".

[rootfs]

@gman0 i would like a cli option to persist meta data in kubernetes api server (e.g. as a configmap). This option can disable meta data persist if the driver runs in mesos.

@dillaman thanks for help. this case is a bit tricky since we want to persist both rbd as well as ceph cluster meta data.

[kfox1111]

Is this just for snapshotting, or are other uses also affected?

[pohly]

Also other uses.

[JrCs]

Also if snapshotter container is not start in the same node as provisionner container, trying to create a snapshot result in: Source Volume ID csi-xxx-xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx cannot found cannot found

[mickymiek]

Made a PR to address this issue (still WIP), please give me your feedbacks :)

[mickymiek]

@gman0 I think this can be closed?