Use new higher-level abstractions

[the-mikedavis]

This updates the log reader and log manifest to use the newer versions of the behaviors in osiris. The new log reader behaviour is at a higher level with callbacks like chunk_iterator/3, iterator_next/1 and send_file/3 for reading. This means we need to include more logic from osiris_log, but the new reader is ultimately much cleaner and it makes the manifest behaviour less complex.

Since the new behaviors are much different from the old ones, I have made new branches in rabbitmq-server (streams-tiered-storage) and osiris (tiered-storage-abstractions) and updated the README here to point to them.

There are a lot of outstanding TODOs especially in the manifest module. The basic "kick the tires" workflow should work:

ADDITIONAL_PLUGINS=rabbitmq_stream_s3 \
dep_rabbitmq_stream_s3="git git@github.com:amazon-mq/rabbitmq-stream-s3.git md/new-abstractions" \
dep_osiris="git https://github.com/amazon-mq/upstream-to-osiris tiered-storage-abstractions" \
RABBITMQ_CONFIG_FILES=../tiered-storage.conf \
TEST_TMPDIR=/src/tiered-storage-data/ \
NON_DETERMINISTIC=true \
make run-broker

With tiered-storage.conf like so:

s3.region = <region where the bucket is>
s3.bucket = <bucket name>
s3.account_id = <account ID>
s3.access_key_id = <access key ID>
s3.secret_key = <secret key>

streams.log_reader = rabbitmq_stream_s3_log_reader
streams.log_manifest = rabbitmq_stream_s3_log_manifest

Then publishing and consuming from a stream:

stream-perf-test --producers 1 --consumers 0 --streams sq
stream-perf-test --producers 0 --consumers 1 --streams sq --offset first

Note that to see reading from the remote tier, you need to delete some segment and index files from the head of the log and then stop rabbit and start rabbit. In order for this to work well (i.e. no reboot) we will need to get retention working - that's a TODO in the manifest.