fix: tzkt api may return more delegators than numDelegators

[TPXP]

---

name: fix: tzkt api may return more delegators than numDelegators
about: Create a pull request to make a contribution
labels: bugfix, tzkt_api

---

This PR resolves the issue . The following steps were performed:

• Analysis: TZTK seems to return more delegators than numDelegators in some cases on the rewards/split endpoint. Handle this gracefully and move on. See tz3LV9aGKHDnAZHCtC9SjNtTrKRu678FqSki at cycle 751 for an example: https://api.tzkt.io/v1/rewards/split/tz3LV9aGKHDnAZHCtC9SjNtTrKRu678FqSki/751?offset=0&limit=10000

• Solution: Update the length condition to stop if we fetched at least as much delegators as numDelegators. Also, add another safeguard by stopping the loop if we're not fetching any more validators (indicating we've reached the end of the list) - this way we will stop if TzKT returns less validators than numDelegators.

• Implementation: Rough description/explanation of the implementation choices.

• Performed tests: Run TRD for baker tz3LV9aGKHDnAZHCtC9SjNtTrKRu678FqSki for cycle 751, confirm it works

• Documentation: N/A

• Check list:

• I tested my changes and confirm they fix the issue I was facing

Work effort: 2 hours

Possibly fixes #616

[nicolasochem]

thanks @TPXP , can you clarify what happens when tzkt returns more delegators than numDelegators? Who are the extra delegators in this case? Are they duplicates? what's happening with the payouts? Is the program crashing?

[TPXP]

Hello @nicolasochem , thanks for the reply.

When TzKT replies more delegators, the len(res["delegators"]) == res["numDelegators"] condition never triggers so we make more and more calls with increasing offset. TRD ends up failing to calculate rewards for the cycle with the following exception.

producer  - ERROR - tzkt error at payment producer loop: 'Max sequent calls number exceeded (256)', will try again.

I couldn't find any duplicates in the example I'm giving - maybe a few weird addresses but that's all. It seems that numDelegators is just wrong. Here's how I checked:

$ wget https://api.tzkt.io/v1/rewards/split/tz3LV9aGKHDnAZHCtC9SjNtTrKRu678FqSki/751\?offset\=0\&limit\=10000 -O rewards.json
$ jq '.numDelegators' ./rewards.json
3224
$ jq '.delegators[].address' ./rewards.json | sort | uniq | wc -l
3225
$ jq '.delegators[].address' ./rewards.json | sort | less
"KT1JBkpjozcKXWpHJLVhU67ZpXFhzH9Hekzg"
"tz1KeBQPzzGA7rBDcUY7biCKEKTSnxX291Hm"
...
"tz1iyXtwz3bKa6r8KEh6CnQU6CayU4msz4zj"
"tz2VJ6pJnnb9rYHFSQzUiZHX1M3uGnvGu4pz"
# Maybe 0 balances? (No)
$ jq '.delegators[].delegatedBalance' ./rewards.json | sort -h | less
154
[...]
6025854993128

[nicolasochem]

@Groxan can you please check on your end why this would be happening?

[Groxan]

That indeed may happen, when an account re-delegates to another baker, but still has unstake requests with the previous baker. I.e. the account has delegated balance with two different bakers at the same time, but counted as delegator only with the actual one.

In other words, numDelegators doesn't include those who actually delegated to a different baker, but still has pending unstake requests.

[Groxan]

The best way to check for end of data is response.length < limit.

[TPXP]

Thanks for confirming @Groxan , I've updated the PR to fetch pages until we get less delegators than limit.

[darrensteele]

Looks good to me - works fine in my bakery

[nicolasochem]

thanks everyone