debug_TraceBlockByNumber reports lower gasUsed than eth_getTransactionReceipt with Avalanchego v0.12.0

[jwelch-qn]

Describe the bug
eth_getTransactionReceipt shows gasUsed for a specific transaction as: 1,014,929 while this debug_TraceBlockByNumber shows gasUsed: 970,713.

To Reproduce
Compare gasUsed for tx:0xd69a271d948cf87fc8d8e99568e82a5f87be11eb98b6ef25d01f90920613a54c
returned by:

curl <ENDPOINT_URL>/ext/bc/C/rpc \
 -X POST \
 -H "Content-Type: application/json" \
 --data '{"method":"debug_traceBlockByNumber","params":["0x1DD60D5", {"tracer": "callTracer"}],"id":1,"jsonrpc":"2.0"}' | jq . 

vs. gasUsed for that same tx with:

curl -s <ENDPOINT_URL>/ext/bc/C/rpc \
 -X POST \
 -H "Content-Type: application/json" \
 --data '{"method":"eth_getTransactionReceipt","params":["0xd69a271d948cf87fc8d8e99568e82a5f87be11eb98b6ef25d01f90920613a54c"],"id":1,"jsonrpc":"2.0"}' | jq .

Expected behavior
I would expect that gasUsed for a specific tx would be the same by any method that returns this data.

Operating System
Mac OS Ventura

[StephenButtolph]

Here is a good resource to understand what is happening here: https://dashboard.tenderly.co/tx/ava/0xd69a271d948cf87fc8d8e99568e82a5f87be11eb98b6ef25d01f90920613a54c/gas-usage

970,713 is the gas consumed by the calls - but there is the additional 44,216 gas that is charged due to the size of the transaction.

I'm going to close this - but if something else seems off please feel free to re-open.

[chaoyaji-cb]

Hi @StephenButtolph, thanks for replying and that makes sense to me, but we have another endpoint with Avalanchego v0.12.3 returned gasUsed 1,014,929 for this transaction from debug call. See below:

curl "https://api-eece15.avax.network/ext/bc/C/rpc?token=APIKEY" \
 -X POST \
 -H "Content-Type: application/json" \
 --data '{"method":"debug_traceBlockByNumber","params":["0x1DD60D5", {"tracer": "callTracer"}],"id":1,"jsonrpc":"2.0"}' | jq .
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "result": {
        "from": "0x21f87cfb3218176557273eeaed15f33ac5b4b68d",
        "gas": "0x1e5c02",
        "gasUsed": "0xf7c91",
        "to": "0x040993fbf458b95871cd2d73ee2e09f4af6d56bb",
        "input": "...

Could you please take another look?

[StephenButtolph]

curl --location 'https://{host}/ext/bc/C/rpc \
--header 'Content-Type: application/json' \
--data '{
    "jsonrpc":"2.0",
    "id":1,
    "method":"debug_traceBlockByNumber",
    "params":[
        "0x1DD60D5", 
        {
            "tracer": "callTracer"
        }
    ]
}'

returns:

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": [
        {
            "result": {
                "from": "0x21f87cfb3218176557273eeaed15f33ac5b4b68d",
                "gas": "0x1e5c02",
                "gasUsed": "0xf7c91",
                "to": "0x040993fbf458b95871cd2d73ee2e09f4af6d56bb",
                "input": ...
                "calls": [
                    ...
                ],
                "value": "0x0",
                "type": "CALL"
            }
        },
        ...
    ]
}

Gas = 0xf7c91

---

curl --location 'https://api.avax.network/ext/bc/C/rpc' \
--header 'Content-Type: application/json' \
--data '{
    "jsonrpc": "2.0",
    "method": "eth_getTransactionReceipt",
    "params": [
        "0xd69a271d948cf87fc8d8e99568e82a5f87be11eb98b6ef25d01f90920613a54c"
    ],
    "id": 1
}'

returns:

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
        "blockHash": "0xb4d75f2331e32f734cdcb579a7d643dfc189238ac0f9e37baf0a36f57dd3ee32",
        "blockNumber": "0x1dd60d5",
        "contractAddress": null,
        "cumulativeGasUsed": "0xf7c91",
        "effectiveGasPrice": "0x8d8f9fc00",
        "from": "0x21f87cfb3218176557273eeaed15f33ac5b4b68d",
        "gasUsed": "0xf7c91",
        "logs": [
            ...
        ],
        "logsBloom": ...
        "status": "0x1",
        "to": "0x040993fbf458b95871cd2d73ee2e09f4af6d56bb",
        "transactionHash": "0xd69a271d948cf87fc8d8e99568e82a5f87be11eb98b6ef25d01f90920613a54c",
        "transactionIndex": "0x0",
        "type": "0x0"
    }
}

Gas = 0xf7c91

---

These gas values are the same afaict. Which number do you think is inconsistent? (And are you sure that number isn't referring to only the gas spent while performing the calls rather than the total gas usage?)

[chaoyaji-cb]

@StephenButtolph Let me clarify. We have two node providers that on different client version, and they return different gasUsed results for a same call({"method":"debug_traceBlockByNumber","params":["0x1DD60D5", {"tracer": "callTracer"}],"id":1,"jsonrpc":"2.0"}). See examples below:

Node on v0.12.0

❯ curl https://{host}/API_KEY/ext/bc/C/rpc \
 -X POST \
 -H "Content-Type: application/json" \
 --data '{"method":"debug_traceBlockByNumber","params":["0x1DD60D5", {"tracer": "callTracer"}],"id":1,"jsonrpc":"2.0"}' | jq
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "result": {
        "type": "CALL",
        "from": "0x21f87cfb3218176557273eeaed15f33ac5b4b68d",
        "to": "0x040993fbf458b95871cd2d73ee2e09f4af6d56bb",
        "value": "0x0",
        "gas": "0x1e5c02",
        "gasUsed": "0xecfd9",

gasUsed is "0xecfd9"

Node on v0.12.3

curl "https://api-eece15.avax.network/ext/bc/C/rpc?API_KEY" \
 -X POST \
 -H "Content-Type: application/json" \
 --data '{"method":"debug_traceBlockByNumber","params":["0x1DD60D5", {"tracer": "callTracer"}],"id":1,"jsonrpc":"2.0"}' | jq .

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "result": {
        "from": "0x21f87cfb3218176557273eeaed15f33ac5b4b68d",
        "gas": "0x1e5c02",
        "gasUsed": "0xf7c91",
        "to": "0x040993fbf458b95871cd2d73ee2e09f4af6d56bb",

gasUsed is "0xf7c91"

According to your previous reply:

Here is a good resource to understand what is happening here: https://dashboard.tenderly.co/tx/ava/0xd69a271d948cf87fc8d8e99568e82a5f87be11eb98b6ef25d01f90920613a54c/gas-usage. 970,713 is the gas consumed by the calls - but there is the additional 44,216 gas that is charged due to the size of the transaction.

I believe "0xecfd9" -> 970,713 should be the correct gasUsed value returned from debug_traceBlockByNumber since debug calls return the gas consumed by the calls, instead of the total gasUsed.

Please let me know if there's any wrong takes and also please help take a look on that parity issues in different avalanche clients. Thanks!

[aaronbuchwald]

Hey @chaoyaji-cb , thanks for re-opening and reporting this.

Funny timing on this one, TLDR v0.12.3 fixed the originally reported weirdness of this issue, so this is an expected change in behavior that we migrated in from geth.

When you reported this issue prior to AvalancheGo v1.10.3, the behavior that @StephenButtolph described was correct ie. the debug tracer would NOT count the IntrinsicGas from the transaction, which resulted in traced transactions reporting a lower amount of gas used than transaction receipts.

This discrepancy was fixed upstream in go-ethereum in v1.11.0: https://github.com/ethereum/go-ethereum/releases/tag/v1.11.0 as mentioned in the release notes section on tracing. This change was made in this PR: ethereum/go-ethereum#27029.

We migrated the changes through v1.11.4 (including v1.11.0) in v0.12.3 release of Coreth, which changed the behavior and is actually the fix for the originally reported issue.

The actual change here is that the tracer now uses CaptureTxStart invoked from https://github.com/ava-labs/coreth/blob/v0.12.3/core/state_transition.go#L343 in order to capture the amount of gas available at the start of the transaction, which sets the gasLimit of the callTracer here: https://github.com/ava-labs/coreth/blob/v0.12.3/eth/tracers/native/call.go#L239. Since CaptureTxStart is invoked with the actual gas limit prior to subtracting IntrinsicGas this results in the top-level call including the IntrinsicGas of the transaction.

CaptureTxEnd then uses the originally set gasLimit to calculate the amount of gas that was actually used: https://github.com/ava-labs/coreth/blob/v0.12.3/eth/tracers/native/call.go#L243.

Previously, the callTracer used CaptureEnd to set the amount of gasUsed by the top-level call: https://github.com/ava-labs/coreth/blob/v0.12.2/eth/tracers/native/call.go#L104.

CaptureEnd is called from within the EVM where the IntrinsicGas has already been subtracted by checking if a CALL operation is the top-level call (ie depth == 0): https://github.com/ava-labs/coreth/blob/v0.12.2/core/vm/evm.go#L276.

[chaoyaji-cb]

Thank you for the in depth explanations! @aaronbuchwald I think for now the gasUsed would be stayed identical for both debug call and transactionReceipt call then. We will let the node provider on 0.12.0 to upgrade to the latest version to resolve this.

[aaronbuchwald]

Sounds good, thanks @chaoyaji-cb !