[Discuss] High level plan for multistream downloads

[kotwanikunal]

Overview

The purpose of this issue is to walk through a high level plan for an enhanced file download/copy framework for index files which should accommodate for parallel, multi-stream downloads.

Background

• A previous design for multi-stream file downloads for segment files was implemented as a part of Remote Store
• During the implementation phase, there were limitations around the concurrency framework as well as support for the crypto SDKs. The feature was instead limited the change to enable parallel file downloads using existing mechanisms.
• Currently, the above multi-stream APIs and implementation exists within the OpenSearch codebase here.
• The following sections walk through the plan to utilize an existing approach and add the missing pieces along with any redesign needed to cover multiple downloads paths (remote store/remote index/snapshot-restore) within the cluster.

Problems with the previous approach

• Concurrency framework incapabilities
  • Parallelization led to an explosion in the number of jobs for segment downloads, eventually leading to a contention between resources at the repository level.
  • All file downloads were initialized in parallel, along with the multiple streams for each file and the handoff between compute threads and I/O threads wasn’t well controlled - leading to thread blocking when the stream was being fetched (I/O wasn’t purely async)
  • There was also an added layer of compute during the decryption process for crypto where the data needed to be buffered until a crypto frame was ready.
• Crypto SDK interworking with multi-stream segment downloads
  • Support for crypto was challenging as the APIs required the frames to align with the part fetches
  • The frame offset data was not stored with the segment files requiring assumptions about offsets to be hard-coded in the implementation

Considerations / Goals

The new approach will be designed with the following goals -

• Performance better than or equal to the current approach
• Reusable across various file download use cases
  • Snapshot/Restore
  • Remote index (Searchable Snapshot/Shasta)
  • Remote store segment replication
• Robust framework for parallelization
  • Enables fairness for competing resources
  • Configurable concurrency limits (Streams/Files)
  • Per request status tracking
  • Retry support
  • Telemetry/Metrics
• Pure Async I/O support
• Client side crypto support
• File integrity verification capability

High Level Design

The following diagram covers the high level design and flow which covers the above goals.

Changes at the Repository layer

Plan A:

• Async IO support will be added using ReactiveStreams from the Java SDKs (Outlined here)
• AsyncMultiStreamBlobContainer will be enhanced with the following two new APIs that will accept a subscriber which will be notified when the data is ready to be consumed.
• The APIs will have the following signature to read the entire or partial file respectively -
  • readBlobAsync(blobName, Subscriber<ByteBuffer>, ...)
  • readBlobAsync(blobName, position, length, Subscriber<ByteBuffer>, ...)
• T.B.D: ThreadPool handoff between compute and I/O threads - including crypto operations

Plan B:

• Add support for virtual threads as outlined by the issue: [Feature Request] [Parallel downloads] Investigate the use of virtual threads as the new I/O mechanism for segment downloads #11708
• Virtual threads simplify the code maintainability while also being as performant as async I/O since the complexities are handled by the JVM

TBD: Feasibility and choice between the above two approaches as a part of https://github.com/opensearch-project/OpenSearch/issues/11708

Introduction of a new, unified DownloadManager

• All the use cases currently utilize their own implementations of an orchestrator/mediator (TransferManager for Searchable Snapshots, RemoteStoreFileDownloader for Remote Store)
• The new implementation will pull out and unify the functionality into a unified DownloadManager class

Changes required in Remote Store metadata

• The segment metadata stored for Remote Store will need enhancements/additions to store the necessary part specific information required to fetch the appropriate parts -
  • Start location of the decrypted file part
  • Length of the decrypted part
• In case of missing metadata i.e the original Remote Store data in the repository, downloads will fallback to the previous mechanism.

Changes at the crypto layer

• The crypto layer will define a wrapper for the Subscriber<ByteBuffer> class which will be passed to the repository and will perform the necessary transformation of data as needed before returning the data to the caller i.e the new unified DownloadManager