op-proposer: Unable to create the L2 Output Submitter, execution reverted

[lyfsn]

Describe the bug

I started the op-proposer in a development environment and encountered the following error:

~/w/f/o/optimism-priv/op-proposer v1.2.0-devnet ❯ ./bin/op-proposer \                 ▼ wangyufsn
    --poll-interval=12s \
    --rpc.port=8560 \
    --rollup-rpc=http://*********:8547 \
    --l2oo-address=$L2OO_ADDR \
    --private-key=$PROPOSER_KEY \
    --l1-eth-rpc=$L1_RPC \
    --log.level=debug
INFO [10-20|14:16:19.394] Initializing L2 Output Submitter 
ERROR[10-20|14:16:20.474] Unable to create the L2 Output Submitter error="execution reverted"
CRIT [10-20|14:16:20.474] Application failed                       message="execution reverted"

To Reproduce

I'm certain that the issue isn't with l1_eth_rpc or that the L2OutputOracle contract hasn't been deployed to the network. I've modified the source code as follows:

// NewL2OutputSubmitter creates a new L2 Output Submitter
func NewL2OutputSubmitter(cfg Config, l log.Logger, m metrics.Metricer) (*L2OutputSubmitter, error) {
	ctx, cancel := context.WithCancel(context.Background())

	l2ooContract, err := bindings.NewL2OutputOracleCaller(cfg.L2OutputOracleAddr, cfg.L1Client)
	if err != nil {
		cancel()
		return nil, fmt.Errorf("failed to create L2OO at address %s: %w", cfg.L2OutputOracleAddr, err)
	}

	cCtx, cCancel := context.WithTimeout(ctx, cfg.NetworkTimeout)
	defer cCancel()

	//======= my code there ========
	id, err := cfg.L1Client.ChainID(cCtx)
	fmt.Println("===chainID: ", id, err) // l1 eth client is correct
	var code string
	err = cfg.L1Client.Client().Call(&code, "eth_getCode", cfg.L2OutputOracleAddr, "latest")
	fmt.Println("===code: ", code, err) // contract is deployed

	version, err := l2ooContract.Version(&bind.CallOpts{Context: cCtx})
	fmt.Println("===version: ", version, err) // but it's not working
	if err != nil {
		cancel()
		return nil, err
	}
	log.Info("Connected to L2OutputOracle", "address", cfg.L2OutputOracleAddr, "version", version)

...

and then I received the following result:

~/w/f/o/optimism-priv/op-proposer v1.2.0-devnet !1 ❯ ./bin/op-proposer \              ▼ wangyufsn
    --poll-interval=12s \
    --rpc.port=8560 \
    --rollup-rpc=http://*******:8547 \
    --l2oo-address=$L2OO_ADDR \
    --private-key=$PROPOSER_KEY \
    --l1-eth-rpc=$L1_RPC \
    --log.level=debug
INFO [10-20|14:46:07.348] Initializing L2 Output Submitter 
===chainID: ******** <nil>
===code:  0x60806040526004361061005e5760003560e01c80635c60da1b116100435780635c60da1b146100be5780638f283970146100f8578063f851a440146101185761006d565b80633659cfe6146100755780634f1ef286146100955761006d565b3661006d5761006b61012d565b005b61006b61012d565b34801561008157600080fd5b5061006b6100903660046106dd565b610224565b6100a86100a33660046106f8565b610296565b6040516100b5919061077b565b60405180910390f35b3480156100ca57600080fd5b506100d3610419565b60405173ffffffffffffffffffffffffffffffffffffffff90911681526020016100b5565b34801561010457600080fd5b5061006b6101133660046106dd565b6104b0565b34801561012457600080fd5b506100d3610517565b60006101577f360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc5490565b905073ffffffffffffffffffffffffffffffffffffffff8116610201576040517f08c379a000000000000000000000000000000000000000000000000000000000815260206004820152602560248201527f50726f78793a20696d706c656d656e746174696f6e206e6f7420696e6974696160448201527f6c697a656400000000000000000000000000000000000000000000000000000060648201526084015b60405180910390fd5b3660008037600080366000845af43d6000803e8061021e573d6000fd5b503d6000f35b7fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035473ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff16148061027d575033155b1561028e5761028b816105a3565b50565b61028b61012d565b60606102c07fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035490565b73ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff1614806102f7575033155b1561040a57610305846105a3565b6000808573ffffffffffffffffffffffffffffffffffffffff16858560405161032f9291906107ee565b600060405180830381855af49150503d806000811461036a576040519150601f19603f3d011682016040523d82523d6000602084013e61036f565b606091505b509150915081610401576040517f08c379a000000000000000000000000000000000000000000000000000000000815260206004820152603960248201527f50726f78793a2064656c656761746563616c6c20746f206e657720696d706c6560448201527f6d656e746174696f6e20636f6e7472616374206661696c65640000000000000060648201526084016101f8565b91506104129050565b61041261012d565b9392505050565b60006104437fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035490565b73ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff16148061047a575033155b156104a557507f360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc5490565b6104ad61012d565b90565b7fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035473ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff161480610509575033155b1561028e5761028b8161060c565b60006105417fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035490565b73ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff161480610578575033155b156104a557507fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035490565b7f360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc81815560405173ffffffffffffffffffffffffffffffffffffffff8316907fbc7cd75a20ee27fd9adebab32041f755214dbc6bffa90cc0225b39da2e5c2d3b90600090a25050565b60006106367fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035490565b7fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61038381556040805173ffffffffffffffffffffffffffffffffffffffff80851682528616602082015292935090917f7e644d79422f17c01e4894b5f4f588d331ebfa28653d42ae832dc59e38c9798f910160405180910390a1505050565b803573ffffffffffffffffffffffffffffffffffffffff811681146106d857600080fd5b919050565b6000602082840312156106ef57600080fd5b610412826106b4565b60008060006040848603121561070d57600080fd5b610716846106b4565b9250602084013567ffffffffffffffff8082111561073357600080fd5b818601915086601f83011261074757600080fd5b81358181111561075657600080fd5b87602082850101111561076857600080fd5b6020830194508093505050509250925092565b600060208083528351808285015260005b818110156107a85785810183015185820160400152820161078c565b818111156107ba576000604083870101525b50601f017fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe016929092016040019392505050565b818382376000910190815291905056fea164736f6c634300080f000a <nil>
===version:   execution reverted
ERROR[10-20|14:46:08.596] Unable to create the L2 Output Submitter error="execution reverted"
CRIT [10-20|14:46:08.596] Application failed                       message="execution reverted"

I can use ether.js to call the same contract address and retrieve the version value.

Additionally, when I use the older release version of optimism to build the op-proposer and start it with the same command line and environment variables, it works. Therefore, I suspect there's a bug in version v1.2.0.

Expected behavior
"Hope it works fine.

System Specs:

• OS: macOs or Linux
• Package Version (or commit hash): v1.2.0

[ajsutton]

v1.2.0 is proposing successfully in production so it's not entirely broken - we'll need more information to understand why its failing in your situation.

What is the "older release version" you said works?
What does the old version print for its Connected to L2OutputOracle log message?

[lyfsn]

The "older version" refers to this commit version (it's not a released version; that was my mistake): https://github.com/ethereum-optimism/optimism/tree/b2d75bf0c61508333a6e6569d3ac25e7d037c7dc

Use the older version, with the same source code as above, the same environment variable, and the same start command line. Only the op-proposer was make with a different version, and its log is:

/w/f/o/optimism-priv/op-proposer fsn-devnet-2 *1 ~1 ❯ ./bin/op-proposer \             ▼ smallyunet
    --poll-interval=12s \
    --rpc.port=8560 \
    --rollup-rpc=http://*********:8547 \
    --l2oo-address=$L2OO_ADDR \
    --private-key=$PROPOSER_KEY \
    --l1-eth-rpc=$L1_RPC \
    --log.level=debug
INFO [10-23|09:40:25.820] Initializing L2 Output Submitter 
===chainID:  ********* <nil>
===code:  0x60806040526004361061005e5760003560e01c80635c60da1b116100435780635c60da1b146100be5780638f283970146100f8578063f851a440146101185761006d565b80633659cfe6146100755780634f1ef286146100955761006d565b3661006d5761006b61012d565b005b61006b61012d565b34801561008157600080fd5b5061006b6100903660046106dd565b610224565b6100a86100a33660046106f8565b610296565b6040516100b5919061077b565b60405180910390f35b3480156100ca57600080fd5b506100d3610419565b60405173ffffffffffffffffffffffffffffffffffffffff90911681526020016100b5565b34801561010457600080fd5b5061006b6101133660046106dd565b6104b0565b34801561012457600080fd5b506100d3610517565b60006101577f360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc5490565b905073ffffffffffffffffffffffffffffffffffffffff8116610201576040517f08c379a000000000000000000000000000000000000000000000000000000000815260206004820152602560248201527f50726f78793a20696d706c656d656e746174696f6e206e6f7420696e6974696160448201527f6c697a656400000000000000000000000000000000000000000000000000000060648201526084015b60405180910390fd5b3660008037600080366000845af43d6000803e8061021e573d6000fd5b503d6000f35b7fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035473ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff16148061027d575033155b1561028e5761028b816105a3565b50565b61028b61012d565b60606102c07fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035490565b73ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff1614806102f7575033155b1561040a57610305846105a3565b6000808573ffffffffffffffffffffffffffffffffffffffff16858560405161032f9291906107ee565b600060405180830381855af49150503d806000811461036a576040519150601f19603f3d011682016040523d82523d6000602084013e61036f565b606091505b509150915081610401576040517f08c379a000000000000000000000000000000000000000000000000000000000815260206004820152603960248201527f50726f78793a2064656c656761746563616c6c20746f206e657720696d706c6560448201527f6d656e746174696f6e20636f6e7472616374206661696c65640000000000000060648201526084016101f8565b91506104129050565b61041261012d565b9392505050565b60006104437fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035490565b73ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff16148061047a575033155b156104a557507f360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc5490565b6104ad61012d565b90565b7fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035473ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff161480610509575033155b1561028e5761028b8161060c565b60006105417fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035490565b73ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff161480610578575033155b156104a557507fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035490565b7f360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc81815560405173ffffffffffffffffffffffffffffffffffffffff8316907fbc7cd75a20ee27fd9adebab32041f755214dbc6bffa90cc0225b39da2e5c2d3b90600090a25050565b60006106367fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61035490565b7fb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d61038381556040805173ffffffffffffffffffffffffffffffffffffffff80851682528616602082015292935090917f7e644d79422f17c01e4894b5f4f588d331ebfa28653d42ae832dc59e38c9798f910160405180910390a1505050565b803573ffffffffffffffffffffffffffffffffffffffff811681146106d857600080fd5b919050565b6000602082840312156106ef57600080fd5b610412826106b4565b60008060006040848603121561070d57600080fd5b610716846106b4565b9250602084013567ffffffffffffffff8082111561073357600080fd5b818601915086601f83011261074757600080fd5b81358181111561075657600080fd5b87602082850101111561076857600080fd5b6020830194508093505050509250925092565b600060208083528351808285015260005b818110156107a85785810183015185820160400152820161078c565b818111156107ba576000604083870101525b50601f017fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe016929092016040019392505050565b818382376000910190815291905056fea164736f6c634300080f000a <nil>
===version:  1.6.0 <nil>
INFO [10-23|09:40:26.630] Connected to L2OutputOracle              address=0x63BaB31275Ea2f3093A4E45CC3cC7575Bf949A48 version=1.6.0
INFO [10-23|09:40:26.631] Starting L2 Output Submitter 
INFO [10-23|09:40:26.631] L2 Output Submitter started 

Additionally, my Go version is: go version go1.21.0 darwin/arm64.

[ajsutton]

There are quite a few contract changes between that old version and the latest, including changes to how the version is stored. However 1.6.0 is the latest version of the contract so it doesn't appear to be an out of date contract. I'm not too familiar with the changes to how versions are managed by the smart contracts so will have to wait for someone with more knowledge to help out here I suspect.

[lyfsn]

I think this might not be a problem with the contract; it's a bug on the client side.

The code above suggests that there is no issue with the contract, but the Ethereum SDK can't query values from it.

[ajsutton]

The info above just shows that there is some incompatibility between the contract and the go bindings such that the call to version is failing. We don't currently have any info on why that transaction is failing, but the go code is successfully creating and sending the transaction and then for some reason it reverts in the EVM. The same go code is working correctly in production against op-goerli, op-sepolia and op-mainnet. It's also working in CI's end-to-end tests.

We need to work out what's different about your deployment to those environments and to do that we need to know more about what's going on inside the EVM where the failure is occurring.

When you're rolling back to the older version, are you continuing to use the same chain and just changing op-proposer or are you also redeploying the chain? And what version was this chain originally deployed with?

[lyfsn]

I've noticed something:

In the old version, the low-level request parameters are as follows:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "eth_call",
  "params": [
    {
      "data": "0x54fd4d50",
      "from": "0x0000000000000000000000000000000000000000",
      "to": "0x63bab31275ea2f3093a4e45cc3cc7575bf949a48"
    },
    "latest"
  ]
}

In the v1.2.0 version, the request is like this:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "eth_call",
  "params": [
    {
      "from": "0x0000000000000000000000000000000000000000",
      "input": "0x54fd4d50",
      "to": "0x63bab31275ea2f3093a4e45cc3cc7575bf949a48"
    },
    "pending"
  ]
}

The difference is in one parameter: the old version uses data, while the latest version uses input.

[lyfsn]

I found the key commit with the change: ethereum-optimism/op-geth@5cf53f5

Why was this done?

[trianglesphere]

input is the newer term. I'm surprised this broke. We'll definitely need to look into this

[ajsutton]

Geth supports both input and data and has done for a while I believe. The execution specs standard says input should be used.

What L1 client are you using and what version?

[ajsutton]

Also for background that change is from upstream geth: ethereum/go-ethereum#28078 That PR has links to further details on why it was changed.

[lyfsn]

I am using an older version of geth (1.1.8-0914c5e8-20221031), so I encountered this problem. It doesn't seem to be an issue with optimism, and I now know how to fix it. Thank you very much for your help!