[Remote Translog] Remote Translog Truncation and Deletions

[gbbafna]

Is your feature request related to a problem? Please describe.

Pasting relevant content from design doc

Each operation gets logged into transaction logs and are uploaded to remote store before an acknowledgement is sent back to the client with sync durability. Also at every refresh and commit, the uncommitted data in the form of segments gets uploaded to the remote segment store. When we need to recover data at any point in time we piggy back on segment store for the vast majority of data set and only try to recover uncommitted operations using remote translog store, to ensure we optimise recovery time and leverage physical segments that are pre-created. Now this means that the amount of data on remote translog store ever keeps on increasing and we need to ensure we purge this data.
OpenSearch uses a flush threshold interval to minimise translog operations on the disk at any point, post which the engine fires up a flush to commit segments. The flush triggers a rollover for translog generation and then trims older translog generational files(readers) that do not reference any sequence numbers below the local checkpoint of the new safe commit. The unreferenced translog generations are purged as per a deletion policy . We can purge older generation files in remote translogs in this flow before the sequence number durably persisted using a remote translog deletion policy. This can run in background , not affecting the translog operations latency.

In addition to this, the minReferencedGeneration for remote translog needs to depend on segments uploaded succesfully to the remote segment store . Currently it happens with every flush.

Describe solution you like
r

1. We would need to communicate the last refresh/flush durably persisted in remote , to safely delete the files in Remote Translog.
2. We need to truncate the remote translogs based on last uploaded segments
3. In Remote Translog, we can delete the older translog gens which have sequence number below than (current - x) flush.