Implement Storage Synchronizer

[eneshoxha]

Storage Synchronizer is a standalone process that will consumer .bin files written from the Producer Node and will store into topics.

The storage will be done into Ledgers, each ledger will have around 5000 messages stored.

The payload of the message will be stored as binary.

Discussion: Should we make Ledgers created every one hour or should we have it done by the numbers of messages written into the Ledger.

• For each Ledger Storage Synchronizer will store the status into ledger_logs and will do snapshots.
• Should ledger_logs be stored as .json configuration file or should me use SQLite to store the configuration.
  • Recommandation is using SQLite db.

ledger_logs should provide table
ledgers for topic
| ledger_id | ledger_location | entries| createdDate | status

ledger_id	ledger_location	entries	createdDate	status	size
10001	root/data/topic/msg_10001_date.andx	5000	2022-05-23	Closed	100MB
10002	root/data/topic/msg_10002_date.andx	6500	2022-05-23	Closed	100MB
10003	root/data/topic/msg_10003_date.andx	75	2022-05-23	Opened	100MB