IR fails to distribute basic income in some cases

[roman-khimov]

Expected Behavior

Containers are being paid for.

Current Behavior

May 19 13:57:42 tenes2 neofs-ir[1413]: info        settlement/calls.go:63        start basic income collection        {"epoch": 21816}
May 19 13:57:43 tenes2 neofs-ir[1413]: error        innerring/settlement.go:220        basic income: could not send transfer        {"sender": "NWtk9HYWsf1njtSzA3XNgwZXRtriACcJ9G", "recipient": "NL1H4he1ggRajT1ZVqn23zveMRaH9jDvfh", "amount (GASe-12)": "712052094", "details": "413855000000000000", "error": "could not invoke method (transferX): chain/client: contract execution finished with state FAULT; exception: at instruction 1314 (THROW): unhandled exception: \"can't transfer assets\""}
May 19 13:57:43 tenes2 neofs-ir[1413]: error        innerring/settlement.go:220        basic income: could not send transfer        {"sender": "NZJtHKNpaCg178caTwUjEaj5QYASQ5eNeQ", "recipient": "NL1H4he1ggRajT1ZVqn23zveMRaH9jDvfh", "amount (GASe-12)": "5251183949", "details": "413855000000000000", "error": "could not invoke method (transferX): chain/client: contract execution finished with state FAULT; exception: at instruction 1314 (THROW): unhandled exception: \"can't transfer assets\""}

This leads to

May 19 13:57:43 tenes2 neofs-ir[1413]: info        settlement/calls.go:66        start basic income distribution        {"epoch": 21816}
May 19 13:57:43 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 3315033667, expected: 100351356615", "retry-after": 1.40718345}
May 19 13:57:45 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 3315033667, expected: 100351356615", "retry-after": 1.380891264}
May 19 13:57:46 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 3675917808, expected: 100351356615", "retry-after": 1.664829643}
May 19 13:57:48 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 4.755013948}
May 19 13:57:53 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 3.05119816}
May 19 13:57:56 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 9.968390533}
May 19 13:58:06 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 16.017736536}
May 19 13:58:22 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 17.040460179}
May 19 13:58:39 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 25.342676669}
May 19 13:59:04 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 29.5030309}
May 19 13:59:33 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 38.191679394}
May 19 14:00:12 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 45.338615893}
May 19 14:00:57 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 77.403874944}
May 19 14:02:14 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 38.456892376}
May 19 14:02:53 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 59.666466105}
May 19 14:03:53 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 54.358809853}
May 19 14:04:47 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 65.18532265}
May 19 14:05:52 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 62.346612445}
May 19 14:06:54 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 32.35222142}
May 19 14:07:27 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 61.801507482}
May 19 14:08:29 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 61.098490704}
May 19 14:09:30 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 42.07181251}
May 19 14:10:12 tenes2 neofs-ir[1413]: info        basic/distribute.go:46        waiting for basic income bank        {"error": "bank balance: 6697745956, expected: 100351356615", "retry-after": 84.872440702}
May 19 14:11:37 tenes2 neofs-ir[1413]: warn        basic/distribute.go:50        failed to get expected bank balance for distribution        {"error": "bank balance: 6697745956, expected: 100351356615"}

Failed collection phase transfer -> not enough GAS to distribute -> total distribution failure.

Also,

1. We have N+M transactions where N is container owners and M is storage nodes.
2. It's all because of "bank" and it's all fun, but it doesn't really work because we have shared payment responsibility suddenly, we also have additional complexity and practical inability to trace payments (everyone is interested in exact money for exact containers, where is it?)

Possible Solution

1. One transaction per container.
2. Starting from the container contract, like Pay(id, []nodeList), it has basic income rate in the netmap, it has estimations, it can do the math.
3. This will make transfers exact, here is a payment for container X from account A to nodes N, that's it.
4. If it fails, mark container as unpaid (or store the last payment epoch), so that SNs could check for it.
5. But Basic income doesn't follow data distribution #3053.
6. There is also a rounding error currently, 1 picoGAS can't be distributed for N > 1 nodes, btw, so any estimations must be normalized.

Requires contract modifications as well.

Context

Testnet. #3053. #774 node-side.

Your Environment

• Version used: 0.46.1

[roman-khimov]

Unfortunately, 77d7397 has a bug, it compares balance to size which doesn't really work. Luckily, it only affects distribution phase, we still have collection done properly and that's the primary thing we care about now. So here we also need to check for the bank balance and distribute it evenly between participating nodes so that the bank balance would be zero.

[roman-khimov]

Oct 07 11:26:13 tenes4 neofs-ir[552469]: error settlement/calls.go:161 could not send transfer {"component": "basicIncomeProcessor", "epoch": 24612, "sender": "NWtk9HYWsf1njtSzA3XNgwZXRtriACcJ9G", "recipient": "NTFWBaa1NxhJR1VjAqvLpkZidXgc3oefmj", "amount (GASe-12)": "1261083", "paymentDetails": "QiRgAAAAAAAA", "error": "could not invoke method (transferX): chain/client: contract execution finished with state FAULT; exception: at instruction 1321 (THROW): unhandled exception: "can't transfer assets""}
Oct 07 11:26:13 tenes4 neofs-ir[552469]: error settlement/calls.go:161 could not send transfer {"component": "basicIncomeProcessor", "epoch": 24612, "sender": "NZJtHKNpaCg178caTwUjEaj5QYASQ5eNeQ", "recipient": "NTFWBaa1NxhJR1VjAqvLpkZidXgc3oefmj", "amount (GASe-12)": "501033753", "paymentDetails": "QiRgAAAAAAAA", "error": "could not invoke method (transferX): chain/client: contract execution finished with state FAULT; exception: at instruction 1321 (THROW): unhandled exception: "can't transfer assets""}
Oct 07 11:26:13 tenes4 neofs-ir[552469]: error settlement/calls.go:161 could not send transfer {"component": "basicIncomeProcessor", "epoch": 24612, "sender": "NWtk9HYWsf1njtSzA3XNgwZXRtriACcJ9G", "recipient": "NaorVa9TJaUwKrW58f9uNZAvD8g15t92yg", "amount (GASe-12)": "236081239", "paymentDetails": "QiRgAAAAAAAA", "error": "could not invoke method (transferX): chain/client: contract execution finished with state FAULT; exception: at instruction 1321 (THROW): unhandled exception: "can't transfer assets""}
Oct 07 11:26:13 tenes4 neofs-ir[552469]: error settlement/calls.go:161 could not send transfer {"component": "basicIncomeProcessor", "epoch": 24612, "sender": "NZJtHKNpaCg178caTwUjEaj5QYASQ5eNeQ", "recipient": "NaorVa9TJaUwKrW58f9uNZAvD8g15t92yg", "amount (GASe-12)": "1355057660", "paymentDetails": "QiRgAAAAAAAA", "error": "could not invoke method (transferX): chain/client: contract execution finished with state FAULT; exception: at instruction 1321 (THROW): unhandled exception: "can't transfer assets""}
Oct 07 11:26:13 tenes4 neofs-ir[552469]: error settlement/calls.go:161 could not send transfer {"component": "basicIncomeProcessor", "epoch": 24612, "sender": "NWtk9HYWsf1njtSzA3XNgwZXRtriACcJ9G", "recipient": "NgzLFRNrb9yfAWXteYo1v1s8QGRx2VhLHG", "amount (GASe-12)": "86731072", "paymentDetails": "QiRgAAAAAAAA", "error": "could not invoke method (transferX): chain/client: contract execution finished with state FAULT; exception: at instruction 1321 (THROW): unhandled exception: "can't transfer assets""}
Oct 07 11:26:13 tenes4 neofs-ir[552469]: error settlement/calls.go:161 could not send transfer {"component": "basicIncomeProcessor", "epoch": 24612, "sender": "NZJtHKNpaCg178caTwUjEaj5QYASQ5eNeQ", "recipient": "NgzLFRNrb9yfAWXteYo1v1s8QGRx2VhLHG", "amount (GASe-12)": "1380057660", "paymentDetails": "QiRgAAAAAAAA", "error": "could not invoke method (transferX): chain/client: contract execution finished with state FAULT; exception: at instruction 1321 (THROW): unhandled exception: "can't transfer assets""}
Oct 07 11:26:13 tenes4 neofs-ir[552469]: error settlement/calls.go:161 could not send transfer {"component": "basicIncomeProcessor", "epoch": 24612, "sender": "NWtk9HYWsf1njtSzA3XNgwZXRtriACcJ9G", "recipient": "NQGRF8uepLQhLvNFpAa8b7eDNX4jpxmnTK", "amount (GASe-12)": "141293102", "paymentDetails": "QiRgAAAAAAAA", "error": "could not invoke method (transferX): chain/client: contract execution finished with state FAULT; exception: at instruction 1321 (THROW): unhandled exception: "can't transfer assets""}
Oct 07 11:26:13 tenes4 neofs-ir[552469]: error settlement/calls.go:161 could not send transfer {"component": "basicIncomeProcessor", "epoch": 24612, "sender": "NZJtHKNpaCg178caTwUjEaj5QYASQ5eNeQ", "recipient": "NQGRF8uepLQhLvNFpAa8b7eDNX4jpxmnTK", "amount (GASe-12)": "1380057660", "paymentDetails": "QiRgAAAAAAAA", "error": "could not invoke method (transferX): chain/client: contract execution finished with state FAULT; exception: at instruction 1321 (THROW): unhandled exception: "can't transfer assets""}