Object store based metadata store

[tillrohrmann]

With S3's latest addition to support conditional writes, we can consider building our metadata store based on an object store. The metadata store implementation could store all its state in a single blob on S3. When accessing the data, it could fetch the blob and extract the requested parts.

[Xuanwo]

Hi, I'm from the opendal community and want to promote opendal for implementing an object store-based metadata store. I personally believe it would be a great fit.

OpenDAL offers a unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services.

It's now support nearly all object storage services like S3, Gcs, Azblob, Aliyun OSS, ...

Welcome to check it out; I'm eager to collaborate.

[tillrohrmann]

Thanks a lot for your suggestion @Xuanwo. Opendal looks really interesting and it has come up earlier in some internal discussions as well. Once we get started with this issue, we'll get back to you on it.