Skip to content

Use of Gossipsub .PublishBatch #16129

@MarcoPolo

Description

@MarcoPolo

🚀 Feature Request

Use Gossipsub's new PublishBatch method to broadcast things that can be batched, such as DataColumnSidecars.

Description

Without this, network writes tend to prioritize sending data for one column to all peers before sending data for later columns (e.g for two columns and 4 peers per column it would send A,A,A,A,B,B,B,B*). With batch publishing we can change the write order to round robin across columns (e.g. A,B,A,B,A,B,A,B).

In cases where the process is sending at a rate over the network limit, this approach allows at least some copies of every column to propagate through the network early. In early simulations with bandwidth limits of 50mbps for the publisher, this improved network dissemination by ~20-30%.

Describe the solution you'd like

I have a draft PR that updates the relevant call sites. There are no tests for it, help appreciated for how to test this in Prysm.

Describe alternatives you've considered

An alternative is to no change. With partial messages on data columns, this isn't needed as the data we send out initially is fairly small, and the order we send out cells is somewhat random (as it depends on how our peers respond). That said, partial messages is a backwards compatible change and this may still benefit the node if peers do not support partial messages.

  • This isn't exactly correct as the writes may be interleaved in slightly different ways due to goroutine scheduling, but it tends to this pattern.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions