Skip to content

archival: Re-upload topic manifest #1973

Open
@Lazin

Description

Currently the topic_manifest.json file is uploaded once. Only when the topic is created in the S3 bucket.
The problem is that now the data inside the topic config can change so we might want to check the content of the topic periodically and update it if it's different. The check might be as follows:

  • generate the json for the topic manifest, compute md5 from it
  • use S3 ObjectHead request to get md5 hash of the topic manifest in S3 bucket
  • if hashes are different - re-upload the manifest

The method that uploads the topic manifest is invoked asynchronously. We can add the loop that checks the manifest hash every 10 minutes or so there. Need to make sure that it will stop when the archival subsystem is being stopped.

JIRA Link: CORE-706

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions