Skip to content

Conversation

@knst
Copy link
Collaborator

@knst knst commented Aug 1, 2023

Issue being fixed or feature implemented

Each Asset Unlock transaction requires an unique index. This index can't be re-used so far as I know because no quorum can re-sign message with same requestID, but 2 txes with same indexes will have same requestID.

Normally, most of asset-unlock txes should be mined, but some of them won't be mined, for example, by various reasons:fee is too low, trying to unlock more that current limits for block, or any other reason. Asset unlock transactions also permanently expires by design when height of blockchain achieves some numbers since this unlock-tx is signed. But the specific index is already occupied.
It means, that amount of gaps in a datastructure that keep all indexes will be increased over time continuously and earlier or later limit in 10k gaps in indexes could be achieved. Maybe it will take 10 years to achieve this day, may be it will take just a day due to some bug in platform.
Let's take a precaution and update storage by this PR.

Instead of old way with limited storage capacity (no more than 10k for total size of all gaps), let's keep indexes as an ordered list of ranges. For example, the set of these indexes: {1, 2, 3, 5, 7, 8, 9, 1000, 1001} would be presented like this in memory and in evo db:
{[1, 3], [5, 5], [7, 9], [1000, 1001]}.

For reference, CSkipList keep this list like: {all indexes below 1001 is mined except: 4, 6, 10, 11, 12, 13, 14, ... , 999}

NOTE: this changes may become useless if platform will generate Asset Unlock transaction by other way when indexes of expired transactions can be re-used. For now it is impossible so far as I know.

What was done?

Implemented a new datastructure CRangesSet to replace CSkipSet and used in CreditPool's impementation in core.
Also updated related unit test.

How Has This Been Tested?

  1. Run the old version of feature_asset_locks.py -> it failed as expected because transaction that should generate too big gap successfully mined.
  2. Updated functional test accordingly new behavior -> it succeed now.
  3. Added new Unit-Test that is copied from CSkipList

Breaking Changes

It changes behavior of Credit Pool for case of big gaps. Won't affect anything at the moment, because it is theoretical situation in future.

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation
  • I have assigned this pull request to a milestone

@knst knst added this to the 20 milestone Aug 1, 2023
@knst knst requested review from PastaPastaPasta and UdjinM6 August 1, 2023 19:00
@UdjinM6
Copy link

UdjinM6 commented Aug 1, 2023

also, pls rebase on top of the most recent develop to fix ERROR: Uploading artifacts as "archive" to coordinator... 413 Payload Too Large id=4784634216 responseStatus=413 Payload Too Large status=413 token=64_Cify2 in tsan test job

@knst knst force-pushed the asset-lock-ranges branch from 8a03767 to 1e98cdf Compare August 2, 2023 05:57
@knst knst requested a review from UdjinM6 August 2, 2023 07:43
@knst knst mentioned this pull request Aug 3, 2023
5 tasks
@UdjinM6
Copy link

UdjinM6 commented Aug 4, 2023

pls see 1c9e21e

@knst knst force-pushed the asset-lock-ranges branch from 1e98cdf to 3531e8c Compare August 5, 2023 10:28
@knst
Copy link
Collaborator Author

knst commented Aug 5, 2023

pls see 1c9e21e

I updated based on your diff and added couple more changes:

  • shortened Remove() method similarly to Add()'s improvements.
  • comments are edited
  • re-written operator<() (same logic)

UdjinM6
UdjinM6 previously approved these changes Aug 5, 2023
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, utACK.

@github-actions
Copy link

github-actions bot commented Aug 8, 2023

This pull request has conflicts, please rebase.

@knst
Copy link
Collaborator Author

knst commented Aug 8, 2023

resolved conflict due to #5526 by rebasing to top of develop

UdjinM6
UdjinM6 previously approved these changes Aug 8, 2023
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-utACK

@PastaPastaPasta
Copy link
Member

Couldn't we just also remove the limit or something like that ish? I don't understand why ranges are better than "skipped"?

Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asked questions

@knst
Copy link
Collaborator Author

knst commented Aug 17, 2023

Couldn't we just also remove the limit or something like that ish? I don't understand why ranges are better than "skipped"?

We can't just remove limit. If by any reason the platform will issue Asset Unlock transaction with index 1e9 (or -1_32bit = 0xFFFFFFFF or 0xFFFFFFFFFFFFFFFF), all Dash Core clients will immediately allocate 8Gb RAM (or more depends which disruptive value) to enumerate billions of meaningless continuous numbers. For range will be allocated only a pair of 2 integers.

It looks like Denial-of-Service for me with too high consequence: most nodes may die after that. That's why limit of capacity has been introduced at first, to prevent dangerous consequence.

Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-utACK

Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK for merge via merge commit

knst added 4 commits August 21, 2023 10:19
This data structure provide efficient storage for numbers if amount of gaps between them is not too big
It works similarly to CSkipSet but amount of gaps now in unlimited
By design we can have more and more and more gaps in indexes list so far as
we can not re-sign expired transaction of asset-unlock. CRangesList is protected from this situation
@PastaPastaPasta PastaPastaPasta merged commit 84f2c29 into dashpay:develop Aug 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants