sync -- explain algorithm in readme
#1600
Conversation
StephenButtolph
added
the
documentation
…chego into sync-fill-readme
x/sync/README.md
Outdated
|
|
||
| When the client starts syncing, it requests from a server a range proof for the entire database. The server replies with a range proof, which the client verifies. If it's valid, the key-value pairs in the proof are written to the database. If it's not, the client drops the proof and requests the proof from another server. | ||
|
|
||
| A range proof sent by a server may be valid but not contain all of the key-value pairs in the requested range. For example, a client might request all the key-value pairs in [`start`, `end`] but only receive those in range [`start`, `end'`] where `end'` < `end`. There might be too many key-value pairs to include in one message, or the server may be too busy to provide any more in its response. Unless the database is very small, this means that the range proof the client receives in response to its range proof request for the entire database will not contain all of the key-value pairs in the database. |
There was a problem hiding this comment.
We need to write somewhere the word continuous.
A range proof sent by a server may be valid but not contain all of key-value pairs in the requested range.
sounds wrong.
A range proof sent by a server must return a continuous range of the key-value pairs, but may not return the full range that was requested.
Would be more clear imo
There was a problem hiding this comment.
+1 to @StephenButtolph 's comments -- a Range Proof, by definition, must be valid.
Can you describe the current algorithm for how large of a Range Proof it currently returns? Is it complicated?
There was a problem hiding this comment.
I included the bit about validity since it's possible for the server to send the client an invalid range proof since it's a byzantine environment. Edited to mention that the client includes the maximum number of key-value pairs and maximum response size in its request.
Can you describe the current algorithm for how large of a Range Proof it currently returns? Is it complicated?
This is an implementation detail, so I'm not sure it makes sense to include it here, but basically the server tries to respond with the entire key-value range, but if the proof ends up being too large, it includes only a portion of the range.
x/sync/README.md
Outdated
|
|
||
| A range proof sent by a server may be valid but not contain all of the key-value pairs in the requested range. For example, a client might request all the key-value pairs in [`start`, `end`] but only receive those in range [`start`, `end'`] where `end'` < `end`. There might be too many key-value pairs to include in one message, or the server may be too busy to provide any more in its response. Unless the database is very small, this means that the range proof the client receives in response to its range proof request for the entire database will not contain all of the key-value pairs in the database. | ||
|
|
||
| If a client requests a range proof for range [`start`, `end`] but only receives a range proof for [`start`, `end'`] where `end'` < `end` it recognizes that it must still fetch all of the keys in [`end`, `end'`]. It repeatedly requests range proofs for chunks of the remaining key range until it has all of the key-value pairs in [`start`, `end`]. Note that the client may split the remaining key range into chunks and fetch chunks of key-value pairs in parallel. |
There was a problem hiding this comment.
| If a client requests a range proof for range [`start`, `end`] but only receives a range proof for [`start`, `end'`] where `end'` < `end` it recognizes that it must still fetch all of the keys in [`end`, `end'`]. It repeatedly requests range proofs for chunks of the remaining key range until it has all of the key-value pairs in [`start`, `end`]. Note that the client may split the remaining key range into chunks and fetch chunks of key-value pairs in parallel. | |
| If a client requests a range proof for range [`start`, `end`] but only receives a range proof for [`start`, `end'`] where `end'` < `end` it recognizes that it must still fetch all of the keys in [`end'`, `end`]. It repeatedly requests range proofs for chunks of the remaining key range until it has all of the key-value pairs in [`start`, `end`]. Note that the client may split the remaining key range into chunks and fetch chunks of key-value pairs in parallel. |
There was a problem hiding this comment.
nit: [start, end] -> [requested_start, requested_end], and the other [start, end] -> [proof_start, proof_end].
It was confusing to see end < end which seemed tautologically false, and makes reading this paragraph a lot harder.
I'd like to see more of an explanation of the "may" statements in here -- when does it actually split the remaining key ranges into chunks and when does it actually fetch chunks of key-value pairs in parallel?
There was a problem hiding this comment.
Edited to use requested_end and proof_end instead of end and end', which I could see being hard to read.
I'd like to see more of an explanation of the "may" statements in here -- when does it actually split the remaining key ranges into chunks and when does it actually fetch chunks of key-value pairs in parallel?
Again, this is an implementation detail, so I'm not sure it makes sense to explain it in more detail here. I could just omit mention of this altogether. What do you think?
x/sync/README.md
Outdated
|
|
||
| If a client requests a range proof for range [`start`, `end`] but only receives a range proof for [`start`, `end'`] where `end'` < `end` it recognizes that it must still fetch all of the keys in [`end`, `end'`]. It repeatedly requests range proofs for chunks of the remaining key range until it has all of the key-value pairs in [`start`, `end`]. Note that the client may split the remaining key range into chunks and fetch chunks of key-value pairs in parallel. | ||
|
|
||
| The database may be changing as the client is syncing. The sync client can be notified that the root ID of the database it's trying to sync to has changed. Detecting that the root ID to sync to has changed is done outside this package. If this occurs, the key-value pairs the client has learned about via range proofs may no longer be valid. |
There was a problem hiding this comment.
valid seems too harsh here (like an error has occurred)... super nit-picky.
| The database may be changing as the client is syncing. The sync client can be notified that the root ID of the database it's trying to sync to has changed. Detecting that the root ID to sync to has changed is done outside this package. If this occurs, the key-value pairs the client has learned about via range proofs may no longer be valid. | |
| The database may be changing as the client is syncing. The sync client can be notified that the root ID of the database it's trying to sync to has changed. Detecting that the root ID to sync to has changed is done outside this package. If this occurs, the key-value pairs the client has learned about via range proofs may no longer be up to date. |
There was a problem hiding this comment.
Change "[t]he database may be changing" (which implies a specific Revision is changing, which is impossible) to "additional Commits may occur while the client is syncing"
Also root ID -> Root Hash (or Revision in some cases depending on what you're talking about)
If I want more information about how the new Revision detection happens, what package does that?
There was a problem hiding this comment.
Change "[t]he database may be changing" (which implies a specific Revision is changing, which is impossible) to "additional Commits may occur while the client is syncing"
Done
Also root ID -> Root Hash (or Revision in some cases depending on what you're talking about)
Change "root ID" to "root hash"
If I want more information about how the new Revision detection happens, what package does that?
This is an implementation detail -- this package and the merkledb don't need to be used for a blockchain, but I added an example (i.e. the root ID is updated when a new block is accepted if this is being used for a blockchain.)
| 1. `SyncGetRangeProofRequest` | ||
| 2. `RangeProof` | ||
| 3. `SyncGetChangeProofRequest` | ||
| 4. `SyncGetChangeProofResponse` |
There was a problem hiding this comment.
I hate how much RangeProof sticks out here
There was a problem hiding this comment.
Would you hate it if we made a response struct that just had the range proof in it?
There was a problem hiding this comment.
We could rename it in the proto if you want, but if we do let's do it in a separate PR
x/sync/README.md
Outdated
|
|
||
| ## Algorithm | ||
|
|
||
| For each key range, the sync client keeps track of the root ID of the database revision for which it has downloaded that key range. For example, it will store information that says something like, "I have all of the key-value pairs that were in range [`start`, `end`] when the database's root was `rootID`" for some keys `start` and `end`, and some database `rootID`. Note that `rootID` is the root ID that the client is trying to sync to, not the root ID of its own incomplete database. |
There was a problem hiding this comment.
Let's use the terms as defined here: https://github.com/ava-labs/firewood/blob/main/README.md#termimology
rootID -> Root Hash
database revision -> Revision
The last sentence should be rephrased to something like:
The hash of the incomplete database on a client is never sent anywhere, because it does not represent a Root Hash of any Revision.
There was a problem hiding this comment.
rootID -> Root Hash
Done
The hash of the incomplete database on a client is never sent anywhere, because it does not represent a Root Hash of any Revision.
Added
x/sync/README.md
Outdated
|
|
||
| For each key range, the sync client keeps track of the root ID of the database revision for which it has downloaded that key range. For example, it will store information that says something like, "I have all of the key-value pairs that were in range [`start`, `end`] when the database's root was `rootID`" for some keys `start` and `end`, and some database `rootID`. Note that `rootID` is the root ID that the client is trying to sync to, not the root ID of its own incomplete database. | ||
|
|
||
| When the client starts syncing, it requests from a server a range proof for the entire database. The server replies with a range proof, which the client verifies. If it's valid, the key-value pairs in the proof are written to the database. If it's not, the client drops the proof and requests the proof from another server. |
There was a problem hiding this comment.
How is the Root Hash first determined by the client (assuming it knows nothing except the how to reach peers)? Is there an API that says "what is the most recent Root Hash"? Is that the first round trip?
There was a problem hiding this comment.
Added "When the client is created, it is given the root hash of the revision to sync to" to clarify.
x/sync/README.md
Outdated
|
|
||
| When the client starts syncing, it requests from a server a range proof for the entire database. The server replies with a range proof, which the client verifies. If it's valid, the key-value pairs in the proof are written to the database. If it's not, the client drops the proof and requests the proof from another server. | ||
|
|
||
| A range proof sent by a server may be valid but not contain all of the key-value pairs in the requested range. For example, a client might request all the key-value pairs in [`start`, `end`] but only receive those in range [`start`, `end'`] where `end'` < `end`. There might be too many key-value pairs to include in one message, or the server may be too busy to provide any more in its response. Unless the database is very small, this means that the range proof the client receives in response to its range proof request for the entire database will not contain all of the key-value pairs in the database. |
There was a problem hiding this comment.
+1 to @StephenButtolph 's comments -- a Range Proof, by definition, must be valid.
Can you describe the current algorithm for how large of a Range Proof it currently returns? Is it complicated?
x/sync/README.md
Outdated
|
|
||
| A range proof sent by a server may be valid but not contain all of the key-value pairs in the requested range. For example, a client might request all the key-value pairs in [`start`, `end`] but only receive those in range [`start`, `end'`] where `end'` < `end`. There might be too many key-value pairs to include in one message, or the server may be too busy to provide any more in its response. Unless the database is very small, this means that the range proof the client receives in response to its range proof request for the entire database will not contain all of the key-value pairs in the database. | ||
|
|
||
| If a client requests a range proof for range [`start`, `end`] but only receives a range proof for [`start`, `end'`] where `end'` < `end` it recognizes that it must still fetch all of the keys in [`end`, `end'`]. It repeatedly requests range proofs for chunks of the remaining key range until it has all of the key-value pairs in [`start`, `end`]. Note that the client may split the remaining key range into chunks and fetch chunks of key-value pairs in parallel. |
There was a problem hiding this comment.
nit: [start, end] -> [requested_start, requested_end], and the other [start, end] -> [proof_start, proof_end].
It was confusing to see end < end which seemed tautologically false, and makes reading this paragraph a lot harder.
I'd like to see more of an explanation of the "may" statements in here -- when does it actually split the remaining key ranges into chunks and when does it actually fetch chunks of key-value pairs in parallel?
x/sync/README.md
Outdated
|
|
||
| If a client requests a range proof for range [`start`, `end`] but only receives a range proof for [`start`, `end'`] where `end'` < `end` it recognizes that it must still fetch all of the keys in [`end`, `end'`]. It repeatedly requests range proofs for chunks of the remaining key range until it has all of the key-value pairs in [`start`, `end`]. Note that the client may split the remaining key range into chunks and fetch chunks of key-value pairs in parallel. | ||
|
|
||
| The database may be changing as the client is syncing. The sync client can be notified that the root ID of the database it's trying to sync to has changed. Detecting that the root ID to sync to has changed is done outside this package. If this occurs, the key-value pairs the client has learned about via range proofs may no longer be valid. |
There was a problem hiding this comment.
Change "[t]he database may be changing" (which implies a specific Revision is changing, which is impossible) to "additional Commits may occur while the client is syncing"
Also root ID -> Root Hash (or Revision in some cases depending on what you're talking about)
If I want more information about how the new Revision detection happens, what package does that?
x/sync/README.md
Outdated
|
|
||
| We use change proofs to correct the out of date key-value pairs. When the sync client is notified that the root ID to sync to has changed, it requests a change proof from a server for a given key range. For example, if a client has the key-value pairs in range [`start`, `end`] that were in the database when it had `rootID`, then it will request a change proof that provides all of the key-value changes in range [`start`, `end`] from the database version with root ID `rootID` to the database version with root ID `newRootID`. The client verifies the change proof, and if it's valid, it applies the changes to its database. If it's not, the client drops the proof and requests the proof from another server. | ||
|
|
||
| A server needs to have history in order to serve a change proof. Namely, it needs to know all of the database changes between two roots. If the server does not have sufficient history to generate a change proof, it will send a range proof instead. The client will verify and apply the range proof. Note that change proofs, like range proofs, may not contain all of the key-value pairs in the requested range. This is OK because as mentioned above, the client tracks the root ID associated with each range of key-value pairs it has, so it knows which key-value pairs are out of date. Similar to range proofs, if a client requests the changes in range [`start`, `end`], but the server replies with all of the changes in [`start`, `end'`] for some `end'` < `end`, the client will repeatedly request change proofs until it gets remaining key-value pairs (namely in [`end'`, `end`]). |
There was a problem hiding this comment.
Does the client know that it sent a RangeProof? Seems like it has to know, because it should implicitly delete anything that isn't present in the RangeProof that it currently has.
I think you can avoid the repetition here and just mention that the ChangeProof might be incomplete, and the same algorithm is applied to fetch more changes as mentioned earlier. Actually it looks like you talk about that in the next paragraph.
There was a problem hiding this comment.
Seems like it has to know, because it should implicitly delete anything that isn't present in the RangeProof that it currently has.
It doesn't necessarily need to delete everything in the old range proof, only the key-value pairs that are covered by the new range proof.
For example, if the client has a range proof for [start_1, end_1] at root_hash_1, and it gets a range proof for [start_1, end_2] at root_hash_2 where end_2 < end_1 and root_hash_2 is after root_hash_1, then it can still keep the key-values in [end_2, end_1] from the revision at root_hash_1. It'll just have to update those via change proof as well, since they're (potentially) stale.
x/sync/README.md
Outdated
|
|
||
| ### `RangeProof` | ||
|
|
||
| This message is sent from the server to the client in response to a `SyncGetRangeProofRequest`. It contains the key-value pairs that were in the requested key range when the database had the requested root, as well as a proof that the key-value pairs are correct. If a server can't serve the entire requested key range in one response, its response will omit keys from the end of the range rather than the start. For example, if a client requests a range proof for range [`start`, `end`] but the server can't fit all the key-value pairs in one response, it'll send a range proof for [`start`, `end'`] where `end'` < `end`, as opposed to sending a range proof for [`start'`, `end`] where `start'` > `start`. |
There was a problem hiding this comment.
Will the server will send the following response when the client ack that it applied the previous partial key-val pairs?
There was a problem hiding this comment.
Sorry I don't understand the question -- would you mind clarifying? The client never sends any acks to the server.
x/sync/README.md
Outdated
|
|
||
| We use change proofs to correct the out of date key-value pairs. When the sync client is notified that the root ID to sync to has changed, it requests a change proof from a server for a given key range. For example, if a client has the key-value pairs in range [`start`, `end`] that were in the database when it had `rootID`, then it will request a change proof that provides all of the key-value changes in range [`start`, `end`] from the database version with root ID `rootID` to the database version with root ID `newRootID`. The client verifies the change proof, and if it's valid, it applies the changes to its database. If it's not, the client drops the proof and requests the proof from another server. | ||
|
|
||
| A server needs to have history in order to serve a change proof. Namely, it needs to know all of the database changes between two roots. If the server does not have sufficient history to generate a change proof, it will send a range proof instead. The client will verify and apply the range proof. Note that change proofs, like range proofs, may not contain all of the key-value pairs in the requested range. This is OK because as mentioned above, the client tracks the root ID associated with each range of key-value pairs it has, so it knows which key-value pairs are out of date. Similar to range proofs, if a client requests the changes in range [`start`, `end`], but the server replies with all of the changes in [`start`, `end'`] for some `end'` < `end`, the client will repeatedly request change proofs until it gets remaining key-value pairs (namely in [`end'`, `end`]). |
There was a problem hiding this comment.
If the server does not have sufficient history to generate a change proof, it will send a range proof instead. The client will verify and apply the range proof.
In such case, the client needs to delete all previous key-value pairs stored?
There was a problem hiding this comment.
No, the client doesn't need to do this. It will just update the key-value pairs in the range covered by the range proof.
x/sync/README.md
Outdated
|
|
||
| ## Overview | ||
|
|
||
| This package implements a client and server that allows for the syncing of a [MerkleDB](../merkledb/README.md). The servers have an up to date version of the database, and the clients have an out of date version of the database or an empty database. |
There was a problem hiding this comment.
Here in terms of syncing, we always refer to finalized blocks, right?
There was a problem hiding this comment.
The merkledb and sync packages don't have any notion of blocks. i.e. they could be used to store data for any application, not just blockchain state. However, in practice, one way we intend to use these packages is to do just that. In this case, we will only want to sync state this is actually part of the blockchain state, i.e. finalized blocks.
|
Add: Assuming you have sequenceDiagram
Note right of Client: Normal sync flow
box Client/Server
participant Client
participant Server
end
box New Revision Notifier
participant Notifier
end
Notifier->>Client: CurrentRoot(r1)
Client->>Server: RangeProofRequest(r1, all)
Server->>Client: RangeProofResponse(r1, ..k25)
Client->>Server: RangeProofRequest(r1, k25..)
Server->>Client: RangeProofResponse(r1, k25..k75)
Notifier-)Client: NewRootHash(r2)
Client->>Server: ChangeProofRequest(r1, r2, 0..k75)
Server->>Client: ChangeProofResponse(r1, r2, 0..k50)
Client->>Server: ChangeProofRequest(r1, r2, k50..k75)
Server->>Client: ChangeProofResponse(r1, r2, k50..k75)
Note right of Client: client is @r2 through (..k75)
Client->>Server: RangeProofRequest(r2, k75..)
Server->>Client: RangeProofResponse(r2, k75..k100)
|
| Unless the database is very small, this means that the range proof the client receives in response to | ||
| its range proof request for the entire database will not contain all of the key-value pairs in the database. | ||
|
|
||
| If a client requests a range proof for range [`requested_start`, `requested_end`] but only receives |
There was a problem hiding this comment.
Isn't it the case that range proofs during initial sync do not have a requested_end? And the first one doesn't have a requested_start either. This is alluded to when you say "[the client] requests from a server a range proof for the entire database".
Might be worth a little discussion about (a) requested starts and ends are optional; and (b) the client can determine if there are more keys because the computed partial hash does not match the Root Hash for this Revision.
There was a problem hiding this comment.
Isn't it the case that range proofs during initial sync do not have a requested_end? And the first one doesn't have a requested_start either. This is alluded to when you say "[the client] requests from a server a range proof for the entire database".
Added a blurb that explains the omission of the lower/upper bounds.
the client can determine if there are more keys because the computed partial hash does not match the Root Hash for this Revision.
Sorry I'm not sure what you mean by this -- would you mind elaborating? The root hash calculated from the trie resulting from the range proof has to match the root hash of the requested revision. Otherwise it's not valid.
There was a problem hiding this comment.
the client can determine if there are more keys because the computed partial hash does not match the Root Hash for this Revision.
Sorry I'm not sure what you mean by this -- would you mind elaborating? The root hash calculated from the trie resulting from the range proof has to match the root hash of the requested revision. Otherwise it's not valid.
What I mean is that if the client computes the root hash of what they have received so far and it does not match the Root Hash of the Revision, we know something is wrong -- we haven't received all the keys yet. Of course, this is an oversimplification of what actually happens, but logically it is still correct.
| its range proof request for the entire database will not contain all of the key-value pairs in the database. | ||
|
|
||
| If a client requests a range proof for range [`requested_start`, `requested_end`] but only receives | ||
| a range proof for [`requested_start`, `proof_end`] where `proof_end` < `requested_end` |
There was a problem hiding this comment.
Our previous discussion might require some amendments here, since proof_end probably isn't provided so the comparison isn't possible in most range proof cases.
There was a problem hiding this comment.
Well, I suppose it depends on what your definition of proof_end is. If your definition of proof_end is "the greatest key such that the key's value or non-existence is shown by the proof" then all range proofs have proof_end. However, not all range proofs necessarily have and end proof (as defined by the RangeProof type).
commit a056efc Author: Patrick O'Grady <prohb125@gmail.com> Date: Fri Jun 23 06:08:09 2023 -0700 [x/merkledb] Remove useless `err` check (ava-labs#1650) commit 607489d Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Thu Jun 22 11:27:41 2023 -0400 Update coreth to v0.12.4-rc.0 (ava-labs#1646) commit d2899e6 Author: Stephen Buttolph <stephen@avalabs.org> Date: Wed Jun 21 23:50:41 2023 -0400 Remove GetTx from the DAGVM interface (ava-labs#1642) commit 2fc0d3b Author: Stephen Buttolph <stephen@avalabs.org> Date: Wed Jun 21 20:33:03 2023 -0400 Remove PendingTxs from the DAGVM interface (ava-labs#1641) commit cc73cd5 Author: Stephen Buttolph <stephen@avalabs.org> Date: Wed Jun 21 19:39:44 2023 -0400 Update all AVM tests for post-linearization (ava-labs#1631) commit 0073875 Author: Stephen Buttolph <stephen@avalabs.org> Date: Wed Jun 21 19:10:10 2023 -0400 Remove `dagState` and `GetUTXOFromID` (ava-labs#1632) commit 94302bc Author: Dan Laine <daniel.laine@avalabs.org> Date: Wed Jun 21 10:35:38 2023 -0400 Add daily fuzzing action (ava-labs#1635) commit ac3f2b4 Author: Sam Batschelet <sam.batschelet@avalabs.org> Date: Tue Jun 20 10:59:43 2023 -0400 Remove MaxConnectionAge gRPC StreamID overflow mitigation (ava-labs#1388) commit c7e1c6a Author: felipemadero <felipe.madero@gmail.com> Date: Mon Jun 19 10:56:57 2023 -0300 Improve delegation error message to specify invalid times or over delegated (ava-labs#1606) commit b85b31c Author: Ikko Eltociear Ashimine <eltociear@gmail.com> Date: Sun Jun 18 23:32:49 2023 +0900 Fix typo in binary_snowflake.go (ava-labs#1630) commit 9725fe9 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Thu Jun 15 20:06:56 2023 -0700 Ban usage of `t.Fatal` and `t.Error` (ava-labs#1453) commit df6228b Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Thu Jun 15 14:41:23 2023 -0700 Improve `database/` tests with `require` (ava-labs#1506) commit 94b9ce6 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Thu Jun 15 13:08:15 2023 -0700 Improve `vms/` tests with `require` (ava-labs#1505) commit f458045 Author: Dan Laine <daniel.laine@avalabs.org> Date: Thu Jun 15 11:28:19 2023 -0400 Fix flaky `TestFindNextKeyRandom` test (ava-labs#1624) commit 689aec6 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Thu Jun 15 06:59:32 2023 -0700 Improve `x/` tests with `require` (ava-labs#1454) commit 764c456 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Wed Jun 14 23:04:19 2023 -0700 Improve `snow/` tests with `require` (ava-labs#1503) commit bddbbec Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Jun 15 01:47:10 2023 -0400 Reduce resource log level (ava-labs#1622) commit 5317357 Author: marun <maru.newby@avalabs.org> Date: Wed Jun 14 20:03:37 2023 -0700 e2e: Support testing on MacOS without requiring firewall exceptions (ava-labs#1613) commit eb6e797 Author: Stephen Buttolph <stephen@avalabs.org> Date: Wed Jun 14 17:35:26 2023 -0400 Update versions for v1.10.3 (ava-labs#1605) Co-authored-by: Aaron Buchwald <aaron.buchwald56@gmail.com> commit 9374b56 Author: Stephen Buttolph <stephen@avalabs.org> Date: Tue Jun 13 22:25:38 2023 -0400 Fix `--http-host` flag to support IPv6 (ava-labs#1620) commit cc69f03 Author: Stephen Buttolph <stephen@avalabs.org> Date: Tue Jun 13 21:21:54 2023 -0400 Remove old networking metric (ava-labs#1619) commit f41e627 Author: Stephen Buttolph <stephen@avalabs.org> Date: Tue Jun 13 16:19:28 2023 -0400 Update CodeQL to v2 (ava-labs#1616) commit a3436f1 Author: Gyuho Lee <6799218+gyuho@users.noreply.github.com> Date: Tue Jun 13 12:53:40 2023 -0700 snow/engine/snowman: instantiate voter after issuer (ava-labs#1610) commit 31db450 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Tue Jun 13 10:09:05 2023 -0700 Add `local-prefixes` setting for `goimports` (ava-labs#1612) commit b157612 Author: Dan Laine <daniel.laine@avalabs.org> Date: Mon Jun 12 16:18:53 2023 -0400 `merkledb` and `sync` -- use time based rand seed (ava-labs#1607) Signed-off-by: Dan Laine <daniel.laine@avalabs.org> Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit 13863e3 Author: Stephen Buttolph <stephen@avalabs.org> Date: Mon Jun 12 14:49:54 2023 -0400 Rename license header file to avoid unintended license indexing (ava-labs#1608) commit d0fe28c Author: Dan Laine <daniel.laine@avalabs.org> Date: Fri Jun 9 14:06:10 2023 -0400 `sync` -- explain algorithm in readme (ava-labs#1600) Co-authored-by: Ron Kuris <ron.kuris@avalabs.org> commit 54d1022 Author: Dan Laine <daniel.laine@avalabs.org> Date: Fri Jun 9 12:58:45 2023 -0400 `merkleDB` -- add inner heap type to syncWorkHeap (ava-labs#1582) commit 6dad1d4 Author: Sam Batschelet <sam.batschelet@avalabs.org> Date: Thu Jun 8 20:08:32 2023 -0400 Log unexpected errors during GetValidatorSet (ava-labs#1592) commit 400dd66 Author: morrisettjohn <60852062+morrisettjohn@users.noreply.github.com> Date: Thu Jun 8 19:32:37 2023 -0400 Fix unreadable message errors (ava-labs#1585) commit cdf86ae Author: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Date: Thu Jun 8 16:36:33 2023 -0400 Merkle db iterator (ava-labs#1533) Signed-off-by: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Co-authored-by: Qian (Tony) Zhang <tonyqzhang@yahoo.com> Co-authored-by: Dan Laine <daniel.laine@avalabs.org> Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> commit 925230d Author: Ceyhun Onur <ceyhun.onur@avalabs.org> Date: Thu Jun 8 22:55:30 2023 +0300 Separate subnet stake connected health and metrics from P-chain (ava-labs#1358) Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit efb7d90 Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Jun 8 15:35:11 2023 -0400 Separate health checks by tags (ava-labs#1579) Co-authored-by: Ceyhun Onur <ceyhun.onur@avalabs.org> commit 26242ce Author: Anusha <63559942+anusha-ctrl@users.noreply.github.com> Date: Thu Jun 8 12:14:51 2023 -0700 Improve metrics error message (ava-labs#1598) commit 7b9912d Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Jun 8 14:47:23 2023 -0400 Cleanup fx interface compliance (ava-labs#1599) commit 0f6b09d Author: Alberto Benegiamo <alberto.benegiamo@gmail.com> Date: Thu Jun 8 18:32:47 2023 +0200 Fix P-Chain GetValidatorSet BLS Keys for Subnets (ava-labs#1584) Co-authored-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Co-authored-by: Sam Batschelet <sam.batschelet@avalabs.org> Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit 1c90eee Author: Sam Batschelet <sam.batschelet@avalabs.org> Date: Thu Jun 8 11:10:27 2023 -0400 Add buf-push github workflow (ava-labs#1556) commit 110bb61 Author: Sam Batschelet <sam.batschelet@avalabs.org> Date: Wed Jun 7 17:21:20 2023 -0400 Clarify break on error during ancestors lookup (ava-labs#1580) commit 9026e30 Author: Dan Laine <daniel.laine@avalabs.org> Date: Wed Jun 7 14:53:07 2023 -0400 `x/sync` -- Add `SyncableDB` proto (ava-labs#1559) commit b456e16 Author: Stephen Buttolph <stephen@avalabs.org> Date: Tue Jun 6 17:46:42 2023 -0400 Revert P-Chain height indexing (ava-labs#1591) commit 00e61d8 Author: Dan Laine <daniel.laine@avalabs.org> Date: Tue Jun 6 13:23:27 2023 -0400 `MerkleDB` -- fix `onEvictCache.Flush` (ava-labs#1589) commit 268f5a9 Author: Dan Laine <daniel.laine@avalabs.org> Date: Tue Jun 6 11:04:25 2023 -0400 `MerkleDB` -- add eviction batch size config (ava-labs#1586) commit aed31ae Author: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Date: Tue Jun 6 09:51:10 2023 -0400 Remove version db from merkle db (ava-labs#1534) Co-authored-by: Dan Laine <daniel.laine@avalabs.org> commit ab20b7d Author: Stephen Buttolph <stephen@avalabs.org> Date: Mon Jun 5 10:39:00 2023 -0400 Remove list from AcceptedFrontier message (ava-labs#1578) commit c2ff5ff Author: Stephen Buttolph <stephen@avalabs.org> Date: Fri Jun 2 12:55:10 2023 -0400 Fix proposervm.GetAncestors test flake (ava-labs#1572) commit 842a6ab Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Jun 1 23:39:58 2023 -0400 Reduce the number of test health checks (ava-labs#1571) commit c32f0d6 Author: Meaghan FitzGerald <meag.fitz@avalabs.org> Date: Thu Jun 1 22:51:42 2023 -0400 Fix typo (ava-labs#1570) commit d63aa56 Author: Chloe <99216251+coffeeavax@users.noreply.github.com> Date: Thu Jun 1 21:46:27 2023 -0500 Add more X-chain tests (ava-labs#1487) Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit bdfa043 Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Jun 1 20:03:50 2023 -0400 Remove lists from Chits messages (ava-labs#1412) commit 2cd81c6 Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Jun 1 18:23:49 2023 -0400 Re-add upgrade tests (ava-labs#1410) commit 6cff7b6 Author: Dan Laine <daniel.laine@avalabs.org> Date: Thu Jun 1 17:01:13 2023 -0400 Reduce number of test iterations in merkledb (ava-labs#1568) commit 277d223 Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Jun 1 15:42:06 2023 -0400 Add GetBalance examples for the P-chain and X-chain wallets (ava-labs#1569) commit 4debc0e Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Thu Jun 1 13:25:59 2023 -0400 Rename `APIAllowedOrigins` to `HTTPAllowedOrigins` (ava-labs#1567) commit 484b735 Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Thu Jun 1 13:17:14 2023 -0400 Only send `PushQuery` messages after building the block (ava-labs#1428) commit b923ef7 Author: Dan Laine <daniel.laine@avalabs.org> Date: Thu Jun 1 12:30:14 2023 -0400 `x/sync` -- Use proto for sending Change Proofs (ava-labs#1541) Co-authored-by: dboehm-avalabs <david.boehm@avalabs.org> Co-authored-by: Ron Kuris <ron.kuris@avalabs.org> commit bfaa7f7 Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Thu Jun 1 12:29:44 2023 -0400 Add allowed http hosts configuration (ava-labs#1566) commit 8fb8afe Author: Dan Laine <daniel.laine@avalabs.org> Date: Wed May 31 21:10:20 2023 -0400 Use `http.Error` instead of separately writing error code and message (ava-labs#1564) commit e8b6a5e Author: marun <maru.newby@avalabs.org> Date: Wed May 31 17:31:51 2023 -0700 Randomize unit test execution order to identify unwanted dependencies (ava-labs#1565) commit d6004f2 Author: Stephen Buttolph <stephen@avalabs.org> Date: Wed May 31 12:52:26 2023 -0400 Add test to ensure that database packing produces sorted values (ava-labs#1560) commit e17a6ca Author: Dan Laine <daniel.laine@avalabs.org> Date: Wed May 31 09:53:44 2023 -0400 `x/sync` -- use for sending Range Proofs (ava-labs#1537) Co-authored-by: dboehm-avalabs <david.boehm@avalabs.org> Co-authored-by: Ron Kuris <ron.kuris@avalabs.org> commit d77e409 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Wed May 31 00:37:55 2023 -0400 Add P-chain `GetBlockByHeight` API method (ava-labs#1448) commit db27133 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Wed May 31 00:16:42 2023 -0400 Add P-chain height indexing (ava-labs#1447) Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit 6ba90f7 Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Tue May 30 14:31:26 2023 -0700 Rename beacon to boostrapper, define bootstrappers in JSON file for cross-language compatiblity (ava-labs#1439) Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit 243e313 Author: Dan Laine <daniel.laine@avalabs.org> Date: Tue May 30 15:18:24 2023 -0400 `x/sync` / `x/merkledb` -- add `SyncableDB` interface (ava-labs#1555) commit b66e25e Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Tue May 30 12:58:19 2023 -0400 Enforce inlining functions with a single error return in `require.NoError` (ava-labs#1500) commit 7403188 Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Wed May 31 00:52:54 2023 +0900 Add workflow to mark stale issues and PRs (ava-labs#1443) commit c374c39 Author: Ceyhun Onur <ceyhun.onur@avalabs.org> Date: Sat May 27 04:57:01 2023 +0300 Add ping uptimes test (ava-labs#1550) commit eaf5256 Author: Alberto Benegiamo <alberto.benegiamo@gmail.com> Date: Thu May 25 22:55:00 2023 +0200 Minor pchain UTs cleanup (ava-labs#1554) commit 4b52f82 Author: Sam Batschelet <sam.batschelet@avalabs.org> Date: Thu May 25 14:56:19 2023 -0400 Bump Protobuf and tooling and add section to proto docs outlining buf publishing (ava-labs#1552) commit a99a809 Author: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Date: Thu May 25 11:41:46 2023 -0400 `x/sync` -- Add proto for P2P messages (ava-labs#1472) Co-authored-by: Ron Kuris <ron.kuris@avalabs.org> Co-authored-by: Dan Laine <daniel.laine@avalabs.org> commit f7307d5 Author: Stephen Buttolph <stephen@avalabs.org> Date: Tue May 23 22:32:22 2023 -0400 Update versions for v1.10.2 (ava-labs#1544) commit 581a673 Author: Dan Laine <daniel.laine@avalabs.org> Date: Tue May 23 14:02:34 2023 -0400 add interface for MerkleDB (ava-labs#1519) commit 14b8c98 Author: Stephen Buttolph <stephen@avalabs.org> Date: Mon May 22 23:37:01 2023 -0400 Log chain shutdown duration (ava-labs#1545) commit a2ae740 Author: Stephen Buttolph <stephen@avalabs.org> Date: Mon May 22 20:55:41 2023 -0400 Add serialization tests for transactions added in Banff (ava-labs#1513) commit ffde992 Author: Stephen Buttolph <stephen@avalabs.org> Date: Mon May 22 20:34:50 2023 -0400 Fetch process resource stats as best-effort (ava-labs#1543) commit 9e6bf96 Author: Chloe <99216251+coffeeavax@users.noreply.github.com> Date: Mon May 22 19:13:27 2023 -0500 Add CPU cycles and number of disk reads/writes metrics by pid (ava-labs#1334) commit b459661 Author: Ceyhun Onur <ceyhun.onur@avalabs.org> Date: Sat May 20 02:00:54 2023 +0300 Deprecate uptimes in pong messages (ava-labs#1362) Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit 37b5735 Author: Stephen Buttolph <stephen@avalabs.org> Date: Fri May 19 14:00:30 2023 -0400 Fix network compression type flag usage (ava-labs#1532) commit 61e7aa6 Author: Ceyhun Onur <ceyhun.onur@avalabs.org> Date: Fri May 19 03:19:09 2023 +0300 Rename `StakingEnabled` to `SybilProtectionEnabled` (ava-labs#1441) commit 85c1d24 Author: Ceyhun Onur <ceyhun.onur@avalabs.org> Date: Thu May 18 21:36:43 2023 +0300 Standardize config keys (ava-labs#1370) commit 49b71b4 Author: Alberto Benegiamo <alberto.benegiamo@gmail.com> Date: Thu May 18 20:32:21 2023 +0200 P-chain validator management refactor (ava-labs#1284) Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit b25c22b Author: Alberto Benegiamo <alberto.benegiamo@gmail.com> Date: Thu May 18 19:02:55 2023 +0200 Fix incorrect test refactor (ava-labs#1526) commit bce0c92 Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Fri May 19 00:58:40 2023 +0800 utils/bag: print type of bag elements (ava-labs#1507) commit 396f975 Author: Stephen Buttolph <stephen@avalabs.org> Date: Wed May 17 22:40:52 2023 -0400 Delete duplicate test var definitions (ava-labs#1518) commit 0c391e8 Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Thu May 18 10:14:47 2023 +0800 Add missing verbo logs checks (ava-labs#1504) Co-authored-by: dhrubabasu <7675102+dhrubabasu@users.noreply.github.com> commit 467b905 Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Thu May 18 09:53:18 2023 +0800 engine/snowman: clean up comments in `bubbleVotes` unit tests (ava-labs#1444) commit c930483 Author: Alberto Benegiamo <alberto.benegiamo@gmail.com> Date: Thu May 18 03:15:01 2023 +0200 Convert P-chain Tx cache to be byte based (ava-labs#1517) Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit ec147ab Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Tue May 16 23:58:26 2023 -0400 Ban usage of `nil` in `require` functions (ava-labs#1498) commit 0eb61fb Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Tue May 16 23:37:03 2023 -0400 Ban usage of `require.Equal` when testing for length (ava-labs#1497) commit d146232 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Tue May 16 23:03:14 2023 -0400 Ban usage of `require.Len` when testing for length `0` (ava-labs#1496) commit 1bcab1f Author: Stephen Buttolph <stephen@avalabs.org> Date: Tue May 16 22:41:49 2023 -0400 Remove comment referencing old IP based tracking (ava-labs#1509) commit 824c3b2 Author: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Date: Tue May 16 16:24:07 2023 -0400 MerkleDB Cleanup (ava-labs#1465) Signed-off-by: Dan Laine <daniel.laine@avalabs.org> Co-authored-by: Dan Laine <daniel.laine@avalabs.org> Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> commit 2e44364 Author: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Date: Tue May 16 14:56:37 2023 -0400 Clean up MerkleDVB Sync Close lock (ava-labs#1469) Co-authored-by: Dan Laine <daniel.laine@avalabs.org> commit 9f6c371 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Mon May 15 10:17:00 2023 -0400 Ban `require.Equal` when testing for `0` (ava-labs#1495) commit 9ac856a Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Sat May 13 02:07:07 2023 +0800 config: disallow `ThrottlerConfig.MaxRecheckDelay` < 1 ms (ava-labs#1435) commit 3d2537b Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Sat May 13 00:45:07 2023 +0800 codec: remove `SetMaxSize` from `Manager` (ava-labs#1481) commit e2b4d9a Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Fri May 12 12:21:34 2023 -0400 Enforce the use of a blank identifier for interface compliance (ava-labs#1493) commit f0a86cc Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Fri May 12 12:02:01 2023 -0400 Fix license header CI checks (ava-labs#1492) commit a16d9fb Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Fri May 12 10:04:28 2023 -0400 Standardize single import formats (ava-labs#1466) commit 7b8bbd6 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Wed May 10 01:21:42 2023 -0400 ban function params for `require.ErrorIs` (ava-labs#1486) commit b870515 Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Tue May 9 03:28:21 2023 +0800 chains: move "msgChan" closer to the first use (readability) (ava-labs#1484) commit 0a0e1bb Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Sat May 6 20:19:45 2023 +0800 Replace deprecated "golang.org/x/crypto/ssh/terminal" with "golang.org/x/term" (ava-labs#1464) commit 755ad40 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Sat May 6 07:48:47 2023 -0400 Use `require` in `api` and `utils/password` packages (ava-labs#1471) commit 1b64bbe Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Sat May 6 06:51:16 2023 -0400 Ban `require.NotEqualValues` (ava-labs#1470) commit 8903335 Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Sat May 6 18:45:22 2023 +0800 vms/platformvm/service: preallocate address slice and improve error msg (ava-labs#1477) commit b3a07d8 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Wed May 3 13:34:55 2023 -0400 Use `require.IsType` for type assertions in tests (ava-labs#1458) commit eb8b52a Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Wed May 3 13:34:32 2023 -0400 Remove zstd Cortina check (ava-labs#1459) commit 850c6fd Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Thu May 4 00:59:51 2023 +0800 chains: do not hold write subnetsLock in health checks (ava-labs#1460) commit c125bc1 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Wed May 3 12:59:03 2023 -0400 Ban `require.EqualValues` (ava-labs#1457) commit a583991 Author: Gyuho Lee <gyuho.lee@avalabs.org> Date: Thu May 4 00:58:03 2023 +0800 utils/logging: add "Enabled" method to remove redundant verbo logs (ava-labs#1461) commit 99f35bd Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Wed May 3 01:26:51 2023 -0400 Fix style nits in vm clients (ava-labs#1449) commit 0d8c59e Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Mon May 1 12:59:48 2023 -0400 Use `require` library functions in more tests (ava-labs#1451)
Why this should be merged
yay docs
How this works
fill out readme
How this was tested
n/a