Get/wait transaction by hash with internal indexer #14718

areshand · 2024-09-21T03:59:52Z

Description

use the latest storage ledger info to get transactions from the storage
for get transaction by hash, if the transaction is not committed to the internal indexer, return it as a pending transaction
for wait transaction by hash, if the transaction is not committed to the internal indexer, keep waiting till either committed or timeout
for API response header, use latest ledger info
update the unit test to simulate the delay in internal indexer

Note: this PR is required to fix the flaky test as this adds the functionality to wait for internal indexer to have data before running tests

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

updated unit test

Key Areas to Review

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-09-21T03:59:55Z

⏱️ 19h 37m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose	8h 49m	⬜ ⬜ 🟥 🟥
execution-performance / single-node-performance	3h 16m	🟩 🟩 🟩 🟩 🟩 (+4 more)
forge-compat-test / forge	1h 28m	🟩 🟩 🟥 🟩
forge-e2e-test / forge	1h 1m	🟩 🟩 🟩 🟩
test-target-determinator	48m	🟩 🟩 🟩 🟩 🟩 (+7 more)
execution-performance / test-target-determinator	38m	🟩 🟩 🟩 🟩 🟩 (+4 more)
check	23m	🟩 🟩 🟩 🟩 🟩 (+4 more)
rust-cargo-deny	20m	🟩 🟩 🟩 🟩 🟩 (+7 more)
general-lints	20m	🟩 🟩 🟩 🟩 🟩 (+7 more)
check-dynamic-deps	12m	🟩 🟩 🟩 🟩 🟩 (+7 more)
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose	6h	2m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

testsuite/generate-format/tests/staged/api.yaml

api/types/src/transaction.rs

grao1991 · 2024-09-24T18:21:53Z

ecosystem/indexer-grpc/indexer-grpc-table-info/src/internal_indexer_db_service.rs

@@ -158,6 +158,25 @@ impl InternalIndexerDBService {
            start_version = next_version;
        }
    }
+
+    // For internal testing
+    pub async fn run_with_end_version(


nit: add #cfg[test]?

thought of this. However, api tests are not using this flag to create test context. changing those tests is probably beyond this PR.

api/types/src/transaction.rs

save progress

github-actions · 2024-10-01T23:14:38Z

✅ Forge suite `realistic_env_max_load` success on `f028cae1bd4dcf24b2d5626363dfb6ceabddeda0`

two traffics test: inner traffic : committed: 11760.57 txn/s, latency: 3394.53 ms, (p50: 2700 ms, p70: 2900, p90: 3300 ms, p99: 19900 ms), latency samples: 4471620
two traffics test : committed: 100.01 txn/s, latency: 1672.01 ms, (p50: 1500 ms, p70: 1600, p90: 1800 ms, p99: 7800 ms), latency samples: 1720
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.249, avg: 0.227", "QsPosToProposal: max: 1.200, avg: 1.090", "ConsensusProposalToOrdered: max: 0.351, avg: 0.339", "ConsensusOrderedToCommit: max: 0.445, avg: 0.426", "ConsensusProposalToCommit: max: 0.783, avg: 0.765"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.10s no progress at version 2610711 (avg 0.24s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 7.43s no progress at version 2610709 (avg 7.43s) [limit 15].
Test Ok

github-actions · 2024-10-01T23:16:07Z

✅ Forge suite `framework_upgrade` success on `7ef01a26f8d8a38610e3d364b722df517c970749` ==> `f028cae1bd4dcf24b2d5626363dfb6ceabddeda0`

Compatibility test results for 7ef01a26f8d8a38610e3d364b722df517c970749 ==> f028cae1bd4dcf24b2d5626363dfb6ceabddeda0 (PR)
Upgrade the nodes to version: f028cae1bd4dcf24b2d5626363dfb6ceabddeda0
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 932.97 txn/s, submitted: 934.33 txn/s, failed submission: 1.36 txn/s, expired: 1.36 txn/s, latency: 3289.77 ms, (p50: 2100 ms, p70: 3600, p90: 6900 ms, p99: 9200 ms), latency samples: 82100
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 983.13 txn/s, submitted: 984.37 txn/s, failed submission: 1.24 txn/s, expired: 1.24 txn/s, latency: 3434.09 ms, (p50: 2100 ms, p70: 3900, p90: 7800 ms, p99: 9900 ms), latency samples: 79360
5. check swarm health
Compatibility test for 7ef01a26f8d8a38610e3d364b722df517c970749 ==> f028cae1bd4dcf24b2d5626363dfb6ceabddeda0 passed
Upgrade the remaining nodes to version: f028cae1bd4dcf24b2d5626363dfb6ceabddeda0
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 968.57 txn/s, submitted: 971.25 txn/s, failed submission: 2.68 txn/s, expired: 2.68 txn/s, latency: 3157.06 ms, (p50: 2400 ms, p70: 3300, p90: 6100 ms, p99: 7200 ms), latency samples: 86800
Test Ok

github-actions · 2024-10-01T23:21:09Z

✅ Forge suite `compat` success on `7ef01a26f8d8a38610e3d364b722df517c970749` ==> `f028cae1bd4dcf24b2d5626363dfb6ceabddeda0`

Compatibility test results for 7ef01a26f8d8a38610e3d364b722df517c970749 ==> f028cae1bd4dcf24b2d5626363dfb6ceabddeda0 (PR)
1. Check liveness of validators at old version: 7ef01a26f8d8a38610e3d364b722df517c970749
compatibility::simple-validator-upgrade::liveness-check : committed: 14644.24 txn/s, latency: 2211.36 ms, (p50: 2100 ms, p70: 2100, p90: 2200 ms, p99: 7400 ms), latency samples: 547360
2. Upgrading first Validator to new version: f028cae1bd4dcf24b2d5626363dfb6ceabddeda0
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7019.69 txn/s, latency: 3762.04 ms, (p50: 4100 ms, p70: 4300, p90: 4700 ms, p99: 5000 ms), latency samples: 144900
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7523.20 txn/s, latency: 4218.68 ms, (p50: 4400 ms, p70: 4400, p90: 6500 ms, p99: 6900 ms), latency samples: 251600
3. Upgrading rest of first batch to new version: f028cae1bd4dcf24b2d5626363dfb6ceabddeda0
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7793.46 txn/s, latency: 3652.95 ms, (p50: 4100 ms, p70: 4200, p90: 4300 ms, p99: 4400 ms), latency samples: 145340
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7347.91 txn/s, latency: 4324.39 ms, (p50: 4300 ms, p70: 4300, p90: 6700 ms, p99: 6900 ms), latency samples: 239560
4. upgrading second batch to new version: f028cae1bd4dcf24b2d5626363dfb6ceabddeda0
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 8607.76 txn/s, latency: 2949.52 ms, (p50: 2400 ms, p70: 2800, p90: 5300 ms, p99: 6300 ms), latency samples: 166600
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10043.54 txn/s, latency: 3098.00 ms, (p50: 2700 ms, p70: 3300, p90: 5000 ms, p99: 6800 ms), latency samples: 328400
5. check swarm health
Compatibility test for 7ef01a26f8d8a38610e3d364b722df517c970749 ==> f028cae1bd4dcf24b2d5626363dfb6ceabddeda0 passed
Test Ok

areshand added the CICD:build-images when this label is present github actions will start build+push rust images from the PR. label Sep 21, 2024

areshand force-pushed the use_storage_ledger_version branch 2 times, most recently from addade0 to b9e8cac Compare September 24, 2024 16:17

areshand added the CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR label Sep 24, 2024

areshand force-pushed the use_storage_ledger_version branch 2 times, most recently from 216094e to bc84621 Compare September 24, 2024 16:42

areshand changed the title ~~use storage ledger info when getting transaction by hash~~ Get transaction by hash with internal indexer Sep 24, 2024

areshand force-pushed the use_storage_ledger_version branch from bc84621 to f63503b Compare September 24, 2024 16:46

areshand changed the title ~~Get transaction by hash with internal indexer~~ Get/wait transaction by hash with internal indexer Sep 24, 2024

areshand force-pushed the use_storage_ledger_version branch from f63503b to b353c35 Compare September 24, 2024 17:02

areshand marked this pull request as ready for review September 24, 2024 17:03

areshand requested review from msmouse, lightmark, grao1991, banool, gregnazario and 0xmaayan as code owners September 24, 2024 17:03