eth_subscriptionLogs with topics filter problem

[d-polikhranidi]

System information

Erigon version: erigon version 2022.04.3-alpha-87421417

OS & Version: Linux (Alpine 3.15.4 in Docker)

Commit hash : 8742141

Expected behaviour

When I'm subscribe to erigon node with topics filter - I must see transactions that corresponds to topics.

Actual behaviour

I see no any transactions at all, but they are (according to ETH/BSC explorer). If I don't use any filters (no topics) - I see all transactions, but I need only transactions with topics I'm interested in.

Steps to reproduce the behaviour

const Web3 = require("web3");
const web3 = new Web3(new Web3.providers.WebsocketProvider("wss://NODE_IP:NODE_PORT"));

let options = {
    topics: [
        null,
        '0x0000000000000000000000002faf487a4414fe77e2327f0bf4ae2a264a776ad2',
        null
    ]
};

let subscription = web3.eth.subscribe('logs', options, (err, event) => {
    if (err) throw err;
});

subscription.on('data', event => console.log(event))
subscription.on('changed', changed => console.log(changed))
subscription.on('error', err => { throw err })
subscription.on('connected', nr => console.log(nr))

The minimal code above works well with gEth nodes and getblock.io node.

So, what I miss?

[AskAlexSharov]

I think subscription to logs not supported: https://github.com/ledgerwatch/erigon/tree/devel/cmd/rpcdaemon#rpc-implementation-status
eth_getLogs - works and fast

[d-polikhranidi]

I think subscription to logs not supported: https://github.com/ledgerwatch/erigon/tree/devel/cmd/rpcdaemon#rpc-implementation-status
eth_getLogs - works and fast

But subscription to logs without topics - works great

const Web3 = require("web3");
const web3 = new Web3(new Web3.providers.WebsocketProvider("wss://NODE_IP:NODE_PORT"));

let subscription = web3.eth.subscribe('logs', {}, (err, event) => {
    if (err) throw err;
});

subscription.on('data', event => console.log(event))
subscription.on('changed', changed => console.log(changed))
subscription.on('error', err => { throw err })
subscription.on('connected', nr => console.log(nr))

This works. But when I add topics to options it's shows nothing((

[AskAlexSharov]

Don’t know, maybe bug. Need somebody to check.

[primalcs]

Could you please try
let options = { topics: [ '0x0000000000000000000000002faf487a4414fe77e2327f0bf4ae2a264a776ad2' ] };

instead of
let options = { topics: [ null, '0x0000000000000000000000002faf487a4414fe77e2327f0bf4ae2a264a776ad2', null ] };
?

[dan-nathan]

I'm running similar issues with my erigon node running on Ubuntu 22.04 on commit aa79853. If I only have one log subscription it works fine, but running multiple at once with different topic/address queries causes unexpected behaviour. I'm testing using go-ethereum with this simple program, which subscribes to logs with the option to filter by address 0xdac17f958d2ee523a2206206994597c13d831ec7 (USDT) using the flag -usdt, and the option to filter by topics ["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"] (ERC20 Transfer event hash) using the flag -erc20

package main

import (
	"context"
	"flag"
	"fmt"
	"os"

	ethereum "github.com/ethereum/go-ethereum"
	ethCommon "github.com/ethereum/go-ethereum/common"
	"github.com/ethereum/go-ethereum/core/types"
	ethClient "github.com/ethereum/go-ethereum/ethclient"
)

const (
	usdtAddress         = "0xdac17f958d2ee523a2206206994597c13d831ec7"
	erc20TransferTopic0 = "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
)

func main() {
	websocketAddress := os.Getenv("ERIGON_WEBSOCKET_URL")
	if websocketAddress == "" {
		fmt.Println("Please set ERIGON_WEBSOCKET_URL environment variable")
		return
	}
	client, err := ethClient.Dial(websocketAddress)
	if err != nil {
		fmt.Printf("Failed to connect to websocket: %v\n", err)
		return
	}

	erc20, uniswap := parseCliArgs()

	var filter ethereum.FilterQuery
	if erc20 {
		filter.Topics = [][]ethCommon.Hash{{ethCommon.HexToHash(erc20TransferTopic0)}}
	}
	if uniswap {
		filter.Addresses = []ethCommon.Address{ethCommon.HexToAddress(usdtAddress)}
	}

	receivedLogs := make(chan types.Log)
	subscription, err := client.SubscribeFilterLogs(context.Background(), filter, receivedLogs)
	if err != nil {
		fmt.Printf("Unable to create subscription: %v\n", err)
		return
	}
	defer subscription.Unsubscribe()
	fmt.Println("Subscription successful")
	errChan := subscription.Err()

	for {
		select {
		case log := <-receivedLogs:
			fmt.Printf("Received log %d of block %d from address %s and topic0 %s\n",
				log.Index,
				log.BlockNumber,
				log.Address.String(),
				log.Topics[0].String(),
			)
		case err := <-errChan:
			fmt.Printf("Subscription error encountered: %v\n", err)
			return
		}
	}
}

// Parses command line flags, and returns whether or not to filter by the ERC20
// transfer topic 0 hash, and whether or not to filter by USDT address
func parseCliArgs() (bool, bool) {
	erc20TransferFlag := flag.Bool("erc20", false, "Whether to filter logs by topic 0 hash "+erc20TransferTopic0)
	usdtFlag := flag.Bool("usdt", false, "Whether to filter logs by address "+usdtAddress)
	flag.Parse()
	return *erc20TransferFlag, *usdtFlag
}

When I run this program multiple time concurrently, I get the following behaviour:

• Running one with no flags and one with the -erc20 flag, neither receive any logs
• Running one with no flags and one with the -usdt flag, neither receive any logs
• Running one with no flags, one with the -erc20 flag, and one with both flags, the former two erroneously only receive logs from the usdt address
• Running one with no flags, one with the -usdt flag, and one with both flags, the former two erroneously only receive logs with the erc20 transfer topic 0 hash
• Running one with no flags, one with the -usdt flag, one with the -erc20 flag, and one with both flags, all four only receive logs with the erc20 transfer topic 0 hash from the usdt address
• Running one with the -usdt flag, one with the -erc20 flag, and one with both flags, all behave as expected
• Running one with no flags, and one with both flags, both behave as expected

[samsondav]

This is a blocking bug for us to use Chainlink with Erigon. We require topic filtering on log streams.

[revitteth]

Closing issue as fix is merged.

[rcastellaj]

I'm not sure this is fixed, as we continue to see this issue. I cannot use eth_getLogs with any filters (address, topics). I can only get the full bloom from a log range. It's not the end of the world as I can filter in the client, but it adds a lot of latency and bandwidth requirements which in turn make it lose its advantage against geth.