solanalabs/solana:v1.16.x Docker image cannot deploy programs

[andreisilviudragnea]

Problem

No solanalabs/solana v1.16.x Docker image can successfully execute solana program deploy commands.

I reproduced the bug by running in one terminal:

docker run --ulimit nofile=1000000 --network host solanalabs/solana:v1.16.13

and deployed the example-helloworld Solana program from another terminal:

cargo build-sbf
solana airdrop 100
solana program deploy target/deploy/helloworld.so

The output of the failing command is:

Blockhash expired. 4 retries remaining
Blockhash expired. 3 retries remaining
Blockhash expired. 2 retries remaining
Blockhash expired. 1 retries remaining
Blockhash expired. 0 retries remaining
=================================================================================
Recover the intermediate account's ephemeral keypair file with
`solana-keygen recover` and the following 12-word seed phrase:
=================================================================================
recycle garage galaxy aisle rebuild detail coconut insane shiver exile space pave
=================================================================================
To resume a deploy, pass the recovered keypair as the
[BUFFER_SIGNER] to `solana program deploy` or `solana program write-buffer'.
Or to recover the account's lamports, pass it as the
[BUFFER_ACCOUNT_ADDRESS] argument to `solana program close`.
=================================================================================
Error: Data writes to account failed: Custom error: Max retries exceeded

The logs of the running validator on Solana v1.16.13 are also pretty different and way fewer than when running validator v1.14.27.

When deploying the same Solana program to v1.14.27 validator, it works successfully.

The bug has been introduced in #29992.

Proposed Solution

Adding --allow-private-addr parameter to SOLANA_RUN_SH_VALIDATOR_ARGS environment variable can fix the issue.

I believe --allow-private-addr should be provided by default when running Docker validators, since their primary purpose is CI.