Caqti upgrade with fix for rosetta

[dkijania]

Reimplementing caqti upgrade after revertion (#16694).

PR Structure

• (aa12df7) First commit is pure re-implementation without any fix which fails compilation
• (cd41822) Second commit is fix for rosetta
• (d3ece53) Third commit is fix for archive
• (6abe968) updating toolchains
• (2803494) make fail hard all perf tests which were previously failing due to memory leak issue

Previous PR was reverted as it introduced regression in rosetta. Lesson was learnt and we added more E2E tests for rosetta which now will catch it (#16692). What is more, we stabilized tests here (#16695).

Regarding source of the issue:

After caqti update postgres postgres lost information about type of parameter in some cases. Especially in where clause in search command:

  ... WHERE ($1 IS NULL OR (b.height <= $1)) AND (($2 IS NULL OR (u.hash = $2)) ....

Which resulted in postgres error

could not determine data type of parameter IS NULL

Solution

The possible solution is to always add ::{type} suffix to parameter when using it. For example $1::int or $2::text. For this purpose is modified a bit code responsible for formatting parameter and always use ::{type} suffix for checking for null and CAST when comparing with expected value

How it is tested:

Currently All known rosetta tests are automated in CI:

• unit tests: buildkite/scripts/unit-test.sh dev src/app/rosetta
• indexer unit tests: https://github.com/MinaProtocol/mina/blob/compatible/buildkite/scripts/rosetta-indexer-test.sh
• api tests: https://github.com/MinaProtocol/mina/tree/compatible/src/app/rosetta/test-curl
• integration tests (using rosetta-cli): https://github.com/MinaProtocol/mina/blob/compatible/buildkite/scripts/rosetta-integration-tests.sh
• connectivity tests (devnet,mainnet): https://github.com/MinaProtocol/mina/blob/compatible/scripts/tests/rosetta-connectivity.sh
• e2e command sanity tests: https://github.com/MinaProtocol/mina/blob/compatible/scripts/tests/rosetta-sanity.sh

We added load test to measure memory gain:

• load test: https://github.com/MinaProtocol/mina/blob/compatible/scripts/tests/rosetta-load.sh

We added new test coverage as well for archive node:

• bootstrap test: https://github.com/MinaProtocol/mina/blob/compatible/src/test/archive/archive_node_tests/load_genesis_ledger.ml
• precomputed send tests: https://github.com/MinaProtocol/mina/blob/compatible/src/test/archive/archive_node_tests/archive_precomputed_blocks_test.ml

Performance improvements metrics

Archive bootstrap test is used as a load test since it try to insert mainnet ledger to archive db in one go.

Rosetta load test fires steady load and measure postgres memory

Running load tests on orchestrator env using rosetta & archive is now possible, thanks to caqti upgrade. Before archive couldn't bootstrap with such big genesis ledger

Test	Before fix	After fix	Comment
Rosetta Load test	PostgreSQL Usage : 3323.45 MiB Test Log	PostgreSQL Usage: 2074.36 MiB Test Log	Single node tests (rosetta, daemon, archive ) Average TPS: 3.08, 📊 Total Requests: 1850
Bootstrap test	PostgreSQL Usage: above 4GiB (assertion failed) Test Log	PostgreSQL Usage: 2062 MiB Test Log	Single archive node test with bootstrapping from mainnet genesis ledger

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-nightly-me

[dkijania]

Nightly: https://buildkite.com/o-1-labs-2/mina-end-to-end-nightlies/builds/3463

[dkijania]

!ci-toolchain-me

[dkijania]

!ci-build-me

[dkijania]

!ci-toolchain-me

[dannywillems]

Is this PR still relevant?

[dkijania]

!ci-build-me

[dkijania]

!ci-toolchain-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-nightly-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-nigthly-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[glyh]

Although I'm not requested to review this: This is too big, is it possible to break it down to several smaller PRs?

[dkijania]

@glyh I thought about it, but it is a library bump PR so it is hard/artificial to do it partially

[georgeee]

In a follow-up PR let's remove the line:

echo "WARN: Silencing the errors from the archive node test app. One of the tests is expected to fail."

[dkijania]

!ci-build-me

[SanabriaRusso]

lgtm on infra side. Updated toolchains and updated scripts.