fix(chain): add rpc retry logic

[gacevicljubisa]

Checklist

• I have read the coding guide.
• My change requires a documentation update, and I have done it.
• I have added tests to cover my changes.
• I have filled out the description and linked the related issues.

Description

The bee node was experiencing intermittent connection failures when communicating with blockchain RPC endpoints (especially HAProxy-backed services), resulting in errors like:

"could not get block number" "error"="Post \"http://rpc-sepolia-haproxy...\": EOF"

These EOF errors occur due to short timeout configurations on HAProxy which can prematurely close connections during RPC communication. Without retry logic, these transient network errors (EOF, connection resets) would cause operations to fail immediately, impacting node stability and reliability.

Implemented a custom retryRoundTripper that wraps the HTTP transport with retry logic

Key Features:

• Automatic retries: Up to 3 retry attempts for transient errors
• Exponential backoff: 100ms → 200ms → 400ms between retries (capped at 5s)
• Error detection: Retries on EOF, connection reset, connection refused, and broken pipe errors
• Request body handling: Rewinds request bodies between retries using GetBody() to avoid content-length mismatches
• Uses http.DefaultTransport wrapped with retry logic

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

[janos]

Instead of sleep, use time.After and req.Context in select to return as soon as possible if context is done.

[janos]

This may be problematic if the Body is already read or closed. There is a check below for non-rewritable body, so it should be ok, I suppose.

In general, only idempotent requests should be retried, but in case of rpc, identifying idempotent requests are not easy. Getting information is ok, but posting information may present a problem by duplicating them.

[gacevicljubisa]

Maybe it is the simpliest to increase the values for the timeouts on the RPC side?
These align with HAProxy timeouts to prevent errors, but they look low?

func newHTTPClient() *http.Client {
    return &http.Client{
        Transport: &http.Transport{
            DialContext: (&net.Dialer{
                Timeout:   9 * time.Second, // < HAProxy timeout connect (10s)
                KeepAlive: 5 * time.Second, // Keep connections alive but close before idle timeout
            }).DialContext,
            IdleConnTimeout:       8 * time.Second, // < HAProxy timeout client (10s)
            TLSHandshakeTimeout:   9 * time.Second, // < HAProxy timeout connect (10s)
            ResponseHeaderTimeout: 9 * time.Second, // < HAProxy timeout server (10s)
        },
    }
}

[janos]

I agree, even reties would make the rpc endpoint do more work in this case. Increasing the timeout would be better in this case.

[akrem-chabchoub]

Are these checks for error codes compatible with windows ?
AFAIK that windows have different error codes, maybe some unit tests can reveal it ?

[akrem-chabchoub]

The loop above runs for attempt in 0 to maxRetries, for example with maxRetries = 3, there are 4 attempts, but the error says 3 retries.

[akrem-chabchoub]

Take a look on this package, it may help simplifying the implementation:
https://github.com/hashicorp/go-retryablehttp