Can we handle restore model-system index from snapshot gracefully?

[jmazanec15]

Description

Right now, I am not sure what will happen if the model system index is restored from snapshot. If the model-metadata from the cluster metadata is different, then the cluster will be in an inconsistent state. The only way to get the cluster out of this inconsistent state will be to manually patch the cluster metadata with the opensearch-node command: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/cluster/coordination/RemoveCustomsCommand.java. Even with this, I am not sure how it would be done except for deleting all metadata and then deleting system index.

I am wondering if we can come up with a graceful solution to this.