[Segment Replication] Review existing snapshot and restore compatibility with SegRep

[dreamer-89]

As part of this issue, we want to identify if SegRep enabled cluster have any issues with snapshot and recovery. In case there are issues, identify changes needed to make it compatible. Concretely, we want to test snapshot/restore in below scenarios:

1. Snapshot & restore on segrep disabled cluster.
2. Snapshot & restore on segrep enabled cluster.
3. Snapshot on docrep cluster and restore on segrep cluster & vice-versa.

Segment Replication

• Design Proposal [Segment Replication] Design Proposal #2229
• Meta Issue [META] Segment Replication Issue list #2194

[dreamer-89]

Wrote segment replication integration tests for snapshot in commit below
dreamer-89@91aa303. The initial integration tests shows that segment replication works with snapshot & restore feature. Added tests as mentioned in the issue description.

This is built on top of commit mch2@f996894. Waiting for a working segrep changes before trying out a manual snapshot test.

[Rishikesh1159]

With opensearch 2.3 version, we have Segment replication feature as an experimental release. I have tested the latest 2.3 version with all the 3 above mentioned testing scenarios. The snapshot and restore process is compatible with segment replication in all above mentioned scenarios. I used this doc to understand the process of snapshot and restore. Here are the steps I used to test these:

Steps to Verify:
-> First I run a single node cluster.
-> Then I registered a repository (locally [file system] not on s3/remote store) for storing snapshots
-> Then I created an index of size 18gb with replicas and segment replication enabled.
-> Then I took a snapshot of the cluster.
-> Then I deleted the index.
-> Then I restore cluster from snapshot and i see deleted index came up again with segment replication enabled and entire 18gb size.
-> Later I tested in different scenarios by creating a segment replication enabled index and taking snapshot of cluster. Then I deleted the index and while restoring I restored the index with document replication enabled and entire process was successful. Similar result with vice-versa scenarios.

[dreamer-89]

Snapshot & Restore changes are tested manually as mentioned in last comment. Integration tests are also added via #3993 to ensure snapshot/restore flow is not broken with segrep enabled.