[RFC] Refactor Metadata management in Cluster Manager code as separate lib

[shwetathareja]

Is your feature request related to a problem? Please describe

Today, Cluster Manager metadata management like create/update/delete of an index, alias, template, data stream etc. are tightly coupled with cluster state or cluster management. Hence, any mutable action is tied to enqueuing a task in CM pending_tasks queue and then ClusterStateExecutor updates the ClusterState object. This make it difficult to evolve Cluster Manager to be more cloud native provider friendly, And, also eventually run the CM in stateless mode.

Describe the solution you'd like

Refactor the Cluster Manager code to separate Metadata management as a lib for core entities and there are already custom data types which are pluggable from plugins perspective. This will ensure both cluster and cloud native implementation can build on top of it.

Related component

Cluster Manager

Describe alternatives you've considered

No response

Additional context

No response

[Vikasht34]

@shwetathareja I would really appreciate your insights on this issue. Since we are working on improving Cluster Metadata, I thought it would be valuable to have a detailed discussion comparing cloud-native solutions with more cost-efficient alternatives which aligns with making co-ordinator stateless. Looking forward to your thoughts!