Open
Description
Currently the topic_manifest.json
file is uploaded once. Only when the topic is created in the S3 bucket.
The problem is that now the data inside the topic config can change so we might want to check the content of the topic periodically and update it if it's different. The check might be as follows:
- generate the json for the topic manifest, compute md5 from it
- use S3 ObjectHead request to get md5 hash of the topic manifest in S3 bucket
- if hashes are different - re-upload the manifest
The method that uploads the topic manifest is invoked asynchronously. We can add the loop that checks the manifest hash every 10 minutes or so there. Need to make sure that it will stop when the archival subsystem is being stopped.
JIRA Link: CORE-706