-
Notifications
You must be signed in to change notification settings - Fork 41
Implement Pagination for Mempool Indexer #657
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
AkarshSahlot
wants to merge
1
commit into
block-core:main
Choose a base branch
from
AkarshSahlot:feature/mempool-pagination
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
|
|
||
| **Title:** Fix missing pagination in FetchAddressHistoryAsync | ||
|
|
||
| **Body:** | ||
| Hey, I noticed a TODO in `MempoolSpaceIndexerApi.cs` regarding the default paging limit for the mempool api. | ||
|
|
||
| Right now `FetchAddressHistoryAsync` only grabs the first page (50 txs) and doesn't loop to get the rest. If an address has more than 50 transactions, we're missing the older history. | ||
|
|
||
| I can fix this by adding a loop to fetch subsequent pages using the `after_txid` param until we get a set with fewer than 50 items. | ||
|
|
||
| Unassigning myself if anyone else wants to grab this, otherwise I'll push a PR shortly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
|
|
||
| **Title:** fix: handle pagination in mempool api history | ||
|
|
||
| **Body:** | ||
| Closes #1 | ||
|
|
||
| Implemented the loop to fetch all pages of transactions for an address. | ||
| Previously it would stop after the first 50 results (header limit). | ||
|
|
||
| Tested locally with an address having >50 txs and verified it pulls the full history now. | ||
|
|
||
| Let me know if the while loop logic looks good. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -259,24 +259,39 @@ public async Task<AddressBalance[]> GetAdressBalancesAsync(List<AddressInfo> dat | |
| return utxoDataList; | ||
| } | ||
|
|
||
| public async Task<List<QueryTransaction>?> FetchAddressHistoryAsync(string address, string? afterTrxId = null) //TODO check the paging (I think it is 50 by default | ||
| public async Task<List<QueryTransaction>?> FetchAddressHistoryAsync(string address, string? afterTrxId = null) | ||
| { | ||
| var txsUrl = $"{MempoolApiRoute}/address/{address}/txs"; | ||
| var allTransactions = new List<QueryTransaction>(); | ||
| string? currentAfterTxId = afterTrxId; | ||
|
|
||
| if (!string.IsNullOrEmpty(afterTrxId)) | ||
| txsUrl += $"?after_txid={afterTrxId}"; | ||
| while (true) | ||
| { | ||
| var txsUrl = $"{MempoolApiRoute}/address/{address}/txs"; | ||
|
|
||
| var response = await GetIndexerClient() | ||
| .GetAsync(txsUrl); | ||
| _networkService.CheckAndHandleError(response); | ||
| if (!string.IsNullOrEmpty(currentAfterTxId)) | ||
| txsUrl += $"?after_txid={currentAfterTxId}"; | ||
|
|
||
| if (!response.IsSuccessStatusCode) | ||
| throw new InvalidOperationException(response.ReasonPhrase); | ||
| var response = await GetIndexerClient().GetAsync(txsUrl); | ||
| _networkService.CheckAndHandleError(response); | ||
|
|
||
| var trx = await response.Content.ReadFromJsonAsync<List<MempoolTransaction>>(new JsonSerializerOptions() | ||
| { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower }); | ||
| if (!response.IsSuccessStatusCode) | ||
| throw new InvalidOperationException(response.ReasonPhrase); | ||
|
|
||
| var trx = await response.Content.ReadFromJsonAsync<List<MempoolTransaction>>(new JsonSerializerOptions() | ||
| { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower }); | ||
|
|
||
| if (trx == null || !trx.Any()) | ||
| break; | ||
|
|
||
| allTransactions.AddRange(trx.Select(t => MapToQueryTransaction(t))); | ||
|
|
||
| if (trx.Count < 50) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No safety guard against infinite loops. |
||
| break; // We've reached the last page | ||
|
|
||
| currentAfterTxId = trx.Last().Txid; | ||
| } | ||
|
|
||
| return trx?.Select(t => MapToQueryTransaction(t)).ToList() ?? new List<QueryTransaction>(); | ||
| return allTransactions; | ||
| } | ||
|
|
||
| public async Task<FeeEstimations?> GetFeeEstimationAsync(int[] confirmations) | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
JsonSerializerOptions re-allocated every iteration. It should be a static readonly