gas trasfer failure

[vkarak1]

neofs-node displays the following error when service is started

Dec 28 13:40:45 az neofs-ir[598]: 2022-12-28T13:40:45.821Z        error        innerring/settlement.go:240        basic income: could not send transfer        {"sender": "NcHgKZPCG7rFJgAni2y23WSgDQSahnxUHT", "recipient": "NL1H4he1ggRajT1ZVqn23zveMRaH9jDvfh", "amount (GASe-12)": "302", "details": "412500000000000000", "error": "could not invoke method (transferX): neofs error: chain/client: contract execution finished with state FAULT; exception: at instruction 1720 (THROW): unhandled exception: \"can't transfer assets\""}

Attached logs

Your Environment

Linux az 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
NeoFS Storage node
Version: v0.34.0-142-g8744d6ab-dirty
GoVersion: go1.18.4

[vkarak1]

Getting the same errors in the following build:

NeoFS Storage node
Version: v0.34.0-144-g7c3ab4d1-dirty
GoVersion: go1.18.4

NeoFS Inner Ring node
Version: v0.34.0-144-g7c3ab4d1-dirty
GoVersion: go1.18.4

Jan 10 07:14:31 az neofs-ir[578]: 2023-01-10T07:14:31.892Z        info        netmap/handlers.go:48        notification        {"type": "add peer"}
Jan 10 07:14:31 az neofs-ir[578]: 2023-01-10T07:14:31.895Z        info        netmap/handlers.go:48        notification        {"type": "add peer"}
Jan 10 07:19:28 az neofs-ir[578]: 2023-01-10T07:19:28.043Z        info        settlement/calls.go:55        start basic income collection        {"epoch": 84}
Jan 10 07:19:28 az neofs-ir[578]: 2023-01-10T07:19:28.046Z        error        innerring/settlement.go:240        basic income: could not send transfer        {"sender": "NMSatKxHPq3h79CiWgH3ah3gRXSYpiqpwz", "recipient": "NL1H4he1ggRajT1ZVqn23zveMRaH9jDvfh", "amount (GASe-12)": "302", "details": "415400000000000000", "error": "could not invoke method (transferX): neofs error: chain/client: contract execution finished with state FAULT; exception: at instruction 1720 (THROW): unhandled exception: \"can't transfer assets\""}
Jan 10 07:20:56 az neofs-ir[578]: 2023-01-10T07:20:56.101Z        warn        event/listener.go:366        skip expired main TX notary event        {"chain": "morph", "error": "received main tx has expired"}
Jan 10 07:20:56 az neofs-ir[578]: 2023-01-10T07:20:56.104Z        warn        event/listener.go:366        skip expired main TX notary event        {"chain": "morph", "error": "received main tx has expired"}
Jan 10 07:20:56 az neofs-ir[578]: 2023-01-10T07:20:56.104Z        warn        event/listener.go:366        skip expired main TX notary event        {"chain": "morph", "error": "received main tx has expired"}
Jan 10 07:22:00 az neofs-ir[578]: 2023-01-10T07:22:00.142Z        info        settlement/calls.go:100        start basic income distribution        {"epoch": 84}
Jan 10 07:24:32 az neofs-ir[578]: 2023-01-10T07:24:32.240Z        info        netmap/handlers.go:15        tick        {"type": "epoch"}
Jan 10 07:24:40 az neofs-node[585]: 2023-01-10T07:24:40.243Z        info        neofs-node/morph.go:221        new epoch event from sidechain        {"number": 86}
Jan 10 07:24:40 az neofs-ir[578]: 2023-01-10T07:24:40.247Z        info        netmap/handlers.go:29        notification        {"type": "new epoch", "value": 86}
Jan 10 07:24:40 az neofs-ir[578]: 2023-01-10T07:24:40.254Z        info        netmap/handlers.go:91        tick        {"type": "netmap cleaner"}
Jan 10 07:24:40 az neofs-ir[578]: 2023-01-10T07:24:40.254Z        info        audit/handlers.go:13        new round of audit        {"epoch": 86}
Jan 10 07:24:40 az neofs-ir[578]: 2023-01-10T07:24:40.254Z        info        settlement/calls.go:20        new audit settlement event        {"epoch": 86}
Jan 10 07:24:40 az neofs-ir[578]: 2023-01-10T07:24:40.254Z        info        settlement/handlers.go:14        process audit settlements        {"epoch": 86}
Jan 10 07:24:40 az neofs-ir[578]: 2023-01-10T07:24:40.254Z        info        audit/calculate.go:65        calculate audit settlements        {"current epoch": 86}
Jan 10 07:24:40 az neofs-ir[578]: 2023-01-10T07:24:40.258Z        info        audit/process.go:39        select containers for audit        {"epoch": 86, "amount": 0}
Jan 10 07:24:40 az neofs-ir[578]: 2023-01-10T07:24:40.258Z        info        settlement/handlers.go:18        audit processing finished        {"epoch": 86}
Jan 10 07:25:12 az neofs-ir[578]: 2023-01-10T07:25:12.266Z        info        alphabet/handlers.go:11        tick        {"type": "alphabet gas emit"}
Jan 10 07:29:36 az neofs-ir[578]: 2023-01-10T07:29:36.435Z        info        settlement/calls.go:55        start basic income collection        {"epoch": 85}
Jan 10 07:29:36 az neofs-ir[578]: 2023-01-10T07:29:36.440Z        error        innerring/settlement.go:240        basic income: could not send transfer        {"sender": "NMSatKxHPq3h79CiWgH3ah3gRXSYpiqpwz", "recipient": "NL1H4he1ggRajT1ZVqn23zveMRaH9jDvfh", "amount (GASe-12)": "302", "details": "415500000000000000", "error": "could not invoke method (transferX): neofs error: chain/client: contract execution finished with state FAULT; exception: at instruction 1720 (THROW): unhandled exception: \"can't transfer assets\""}
Jan 10 07:32:08 az neofs-ir[578]: 2023-01-10T07:32:08.529Z        info        settlement/calls.go:100        start basic income distribution        {"epoch": 85}
Jan 10 07:34:40 az neofs-ir[578]: 2023-01-10T07:34:40.624Z        info        netmap/handlers.go:15        tick        {"type": "epoch"}
Jan 10 07:34:48 az neofs-node[585]: 2023-01-10T07:34:48.628Z        info        neofs-node/morph.go:221        new epoch event from sidechain        {"number": 87}
Jan 10 07:34:48 az neofs-node[585]: 2023-01-10T07:34:48.630Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.629Z        info        netmap/handlers.go:29        notification        {"type": "new epoch", "value": 87}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.639Z        info        netmap/handlers.go:91        tick        {"type": "netmap cleaner"}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.639Z        info        audit/handlers.go:13        new round of audit        {"epoch": 87}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.640Z        info        settlement/calls.go:20        new audit settlement event        {"epoch": 87}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.640Z        info        settlement/handlers.go:14        process audit settlements        {"epoch": 87}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.640Z        info        audit/calculate.go:65        calculate audit settlements        {"current epoch": 87}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.641Z        info        audit/process.go:39        select containers for audit        {"epoch": 87, "amount": 1}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.641Z        info        settlement/handlers.go:18        audit processing finished        {"epoch": 87}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.656Z        info        audit/process.go:85        select storage groups for audit        {"epoch": 87, "cid": "29hBa4JBovvZgS4tneL6G4MfmF7R2Tnj73gy3xAwaLxH", "amount": 0}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.656Z        info        audit/process.go:91        filter expired storage groups for audit        {"epoch": 87, "cid": "29hBa4JBovvZgS4tneL6G4MfmF7R2Tnj73gy3xAwaLxH", "amount": 0}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.660Z        info        netmap/handlers.go:48        notification        {"type": "add peer"}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.668Z        info        netmap/handlers.go:48        notification        {"type": "add peer"}
Jan 10 07:34:48 az neofs-ir[578]: 2023-01-10T07:34:48.670Z        info        netmap/handlers.go:48        notification        {"type": "add peer"}
Jan 10 07:39:45 az neofs-ir[578]: 2023-01-10T07:39:45.963Z        info        settlement/calls.go:55        start basic income collection        {"epoch": 86}
Jan 10 07:39:46 az neofs-ir[578]: 2023-01-10T07:39:46.685Z        error        innerring/settlement.go:240        basic income: could not send transfer        {"sender": "NMSatKxHPq3h79CiWgH3ah3gRXSYpiqpwz", "recipient": "NL1H4he1ggRajT1ZVqn23zveMRaH9jDvfh", "amount (GASe-12)": "302", "details": "415600000000000000", "error": "could not invoke method (transferX): neofs error: chain/client: contract execution finished with state FAULT; exception: at instruction 1720 (THROW): unhandled exception: \"can't transfer assets\""}

[carpawell]

That installation uses ContainerFee: 0 that allows generate-and-create container creations (no need to have positive balance) but basic income knows nothing about missing economics in such installations so it tries to send some (missing) NeoFS tokens to the nodes.

We can either turn off any transfers if withoud_mainnet is true or change BasicIncomeRate to zero.

Anyway, basic income with withoud_mainnet==false looks weird to me but can't say it is a bug for sure. @cthulhu-rider, @fyrchik.

[roman-khimov]

Theoretically, internal token can be used for some billing in case the system is used to provide a service. But this may require a bit different economics anyway. I think BasicIncomeRate=0 is OK for now as a setting, maybe it can be made a default one for standalone setups (along with 0 other fees).

[carpawell]

However, config changes did not solve the problem completely. Needed to be investigated one more time.