Skip to content

Task: Enable multi-node dataset routing  #204

@orlandohohmeier

Description

@orlandohohmeier

Summary

Update scheduler-to-worker dataset routing logic to support multiple data node targets per fetch.

  • Allow a Fetch request to include multiple peer IDs (data nodes) with a random selection strategy.
  • Adjust the scheduler to return a list of data nodes for each slice.
  • Ensure fallback behavior is graceful: if one data node fails, retries will hit another peer.
  • Reuse and extend the existing Reference::Peers variant and SelectionStrategy::Random.

Background

Currently, a worker fetches a slice from only one data node, which is a single point of failure. This change enables fault-tolerant fetching and prepares the system for distributed data delivery at scale.

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions