DPP-496 HA indexer integration test #11033

rautenrieth-da · 2021-09-27T12:56:27Z

Additions in this PR:

StorageBackend.verifyIntegrity(), which (currently) verifies that all event sequential IDs are correct
EndlessReadService, a ReadService implementation that serves a deterministic infinite stream of artificial state updates
IndexerStabilityTestFixture, a helper that can start N parallel running indexers from the above ReadService
IndexerStabilitySpec, a test which verifies that N concurrently running indexers correctly cooperate
StorageBackendTestsDebug, an extension of the StorageBackend suite that checks that the new verifyIntegrity() method actually catches errors

changelog_begin changelog_end

nmarton-da

Nice work, looks great!
Added several questions, and comments

...tegration-api/src/main/scala/platform/store/backend/common/DebugStorageBackendTemplate.scala

ledger/participant-integration-api/src/main/scala/platform/store/backend/StorageBackend.scala

.../participant-integration-api/src/test/lib/scala/platform/indexer/ha/EndlessReadService.scala

...ant-integration-api/src/test/lib/scala/platform/indexer/ha/IndexerStabilityTestFixture.scala

...articipant-integration-api/src/test/lib/scala/platform/indexer/ha/IndexerStabilitySpec.scala

nmarton-da

solid work

the error assertion needs some more care
the test is currently failing on ci
otherwise looks good to me

nmarton-da · 2021-09-29T06:32:07Z

...articipant-integration-api/src/test/lib/scala/platform/indexer/ha/IndexerStabilitySpec.scala

+      result
+    } else {
+      // If HA mode is not supported, the test must fail, but there are multiple reasons why it can fail.
+      // E.g., duplicate parameter table initialization, or duplicate event sequential ids.


oh yeah, nice we are catching this one as well :)
but here we are after the duplicate seq id-s... other type of errors might shade some real problem 🤔
how about starting the indexers after each other to prevent the occurrence of the duplicate param table initialization, and assert on the duplicate sequential id errors?

That would be possible, but then we don't test that parallel initialization does work on Postgres/Oracle, which is IMO more important to test.

can you think of a way to test both?

Otherwise logback complains about missing classes at startup

Otherwise the user has to wait for 10min to get useful log output about the connection issue

The previous defaults are not valid for Oracle

ledger/participant-integration-api/src/main/scala/platform/indexer/ha/HaCoordinator.scala

@adriaanm-da

This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @adriaanm-da is in charge of this release. Commit log: ``` cfcdc13 Add deprecation output for sandbox classic (#11119) ac5c52d DPP-609 Add StorageBackendTestsDeduplication (#11088) 55af7ad HttpServiceTestFixture provides a sandbox-classic ledger used for performance tests (#11128) 63ab3f3 Fix change ID references in services.rst (#11130) 31db15d Port error code changes (#11113) 9fd8182 move reusable functions from ContractsFetch to new fetch-contracts library (#11057) c2084f6 Try and improve the explanation of compatibility test exclusions (#11114) 4df9b7c Rename command completion dedup time to duration in participant code [KVL-1057] (#10901) 11bc22d LF: builtins to create, signatory, and obersver on interface payload. (#11120) d4cb1f9 KV ledgers: disable participant side command de-duplication [KVL-1083] (#11095) b9a5a83 Minor: fix trace logging in `TimeBasedWriteSetSelector` (#11117) 018e908 [JSON-API] Make key_hash idx non unique (#11102) 429f437 Remove HA indexer integration test on H2 (#11104) 9cffa1f LF: check LF transaction protobuf aginst local snapshots (#11064) d7ee278 Optional table prefix for trigger service (#11047) 268b2d3 Command deduplication - test case that uses CommandService and CommandSubmissionService [KVL-1106] (#11098) fac05f6 bypass value enrichment for the performance test (#11112) 4860271 Port file was not written to... where? (#11108) a8d703b Increase akka server request-timeout for HttpServiceIntegrationTests (#11109) 37d94aa Add unit for daily performance tests (#11107) eb4ac8a Increase timeouts for resetting database (#11103) db75f7d interface methods: Scala Typechecker (#11097) d3e6f16 [JSON-API] Migrating tests to use sandbox next (#11034) ffc8d68 Data dependencies deserialize imports (#11037) 4b31bf0 update compat versions for 1.17.0 (#11075) 9fd6326 interfaces: consuming/non-consuming iface choices (#11009) 229ce47 Data dependencies serialize imports (#11038) 5d9ec65 Add test case for parallel command deduplication using mixed clients [KVL-1090] (#11093) d64d965 kvutils: Print state updates before and after normalization. (#11096) be216aa update NOTICES file (#11089) 517e866 Extract common code for command dedup conformance tests [KVL-1090] (#11092) 7a4963b add `--max-inbound-message-size` flag to `daml ledger export` (#11087) 26d10b2 Drop NodeId type parameter (#10921) 3cbd922 More NonEmpty functions (#10930) bcd4686 interface methods: Speedy (#11076) 85adaab [DPP-417][DPP-595] Error codes switching - follow-up (#11074) d7b280b Remove debug print in daml-lf/interpreter build (#11091) 1ed6428 LF: move archive snapshots in a separate directory (#11081) 5e424f8 Command dedup conformance suites readme (#11051) 8290347 DPP-496 HA indexer integration test (#11033) 735c309 Switch from InputStream to Byte Array at the binary content JDBC transport (#11072) 6ae3afa fix perf reporting (#11073) bf801a6 Fix SimpleDeduplicationBasic when participant deduplication is disabled (#11083) 62234dc [Short] Remove unused code (#11079) c1d1521 interface methods: Scala AST (#11070) 7dd9c2d Remove expectations for internal failures from parallel command dedup tests (#11061) 5112599 Explicit discard in daml-lf/interpreter (#11067) 554b36c rotate release duty after 1.18.0-snapshot.20210928.7948.0.b4d00317 (#11059) d156964 Release SDK 1.17.0 (#11062) 6d8cf70 Handle interface methods in LF conversion (#11054) 87ecf1f daml-ledger: better errors for non-json responses. (#11055) ec2d26f [Divulgence pruning] Fixes divulgence pruning offset update query (#11046) f13c6d6 release 1.18.0-snapshot.20210928.7948.0.b4d00317 (#11058) d1805a3 More dependabot fun (#11063) 07b273a update NOTICES file (#11060) ``` Changelog: ``` - [JSON-API] make key_hash indexes non-unique, this fixes a bug where a duplicate key conflict was raised on the query store when the same contract was being witnessed twice by two separate parties * [Trigger Service] Enable the new `tablePrefix` setting in the `--jdbc` flag to add a prefix to all tables used by the trigger service to avoid collisions with other components using the same db-schema. * [Trigger Service] Enable the new ``--allow-existing-schema`` flag to initialize the trigger service on a database with a pre-existing schema. - [Daml Assistant] Add support for `--max-inbound-message-size` flag to the Ledger Export tool. Add cons, find, delete and deleteBy functions to NonEmpty module. ``` CHANGELOG_BEGIN CHANGELOG_END

@adriaanm-da

This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @adriaanm-da is in charge of this release. Commit log: ``` cfcdc13 Add deprecation output for sandbox classic (#11119) ac5c52d DPP-609 Add StorageBackendTestsDeduplication (#11088) 55af7ad HttpServiceTestFixture provides a sandbox-classic ledger used for performance tests (#11128) 63ab3f3 Fix change ID references in services.rst (#11130) 31db15d Port error code changes (#11113) 9fd8182 move reusable functions from ContractsFetch to new fetch-contracts library (#11057) c2084f6 Try and improve the explanation of compatibility test exclusions (#11114) 4df9b7c Rename command completion dedup time to duration in participant code [KVL-1057] (#10901) 11bc22d LF: builtins to create, signatory, and obersver on interface payload. (#11120) d4cb1f9 KV ledgers: disable participant side command de-duplication [KVL-1083] (#11095) b9a5a83 Minor: fix trace logging in `TimeBasedWriteSetSelector` (#11117) 018e908 [JSON-API] Make key_hash idx non unique (#11102) 429f437 Remove HA indexer integration test on H2 (#11104) 9cffa1f LF: check LF transaction protobuf aginst local snapshots (#11064) d7ee278 Optional table prefix for trigger service (#11047) 268b2d3 Command deduplication - test case that uses CommandService and CommandSubmissionService [KVL-1106] (#11098) fac05f6 bypass value enrichment for the performance test (#11112) 4860271 Port file was not written to... where? (#11108) a8d703b Increase akka server request-timeout for HttpServiceIntegrationTests (#11109) 37d94aa Add unit for daily performance tests (#11107) eb4ac8a Increase timeouts for resetting database (#11103) db75f7d interface methods: Scala Typechecker (#11097) d3e6f16 [JSON-API] Migrating tests to use sandbox next (#11034) ffc8d68 Data dependencies deserialize imports (#11037) 4b31bf0 update compat versions for 1.17.0 (#11075) 9fd6326 interfaces: consuming/non-consuming iface choices (#11009) 229ce47 Data dependencies serialize imports (#11038) 5d9ec65 Add test case for parallel command deduplication using mixed clients [KVL-1090] (#11093) d64d965 kvutils: Print state updates before and after normalization. (#11096) be216aa update NOTICES file (#11089) 517e866 Extract common code for command dedup conformance tests [KVL-1090] (#11092) 7a4963b add `--max-inbound-message-size` flag to `daml ledger export` (#11087) 26d10b2 Drop NodeId type parameter (#10921) 3cbd922 More NonEmpty functions (#10930) bcd4686 interface methods: Speedy (#11076) 85adaab [DPP-417][DPP-595] Error codes switching - follow-up (#11074) d7b280b Remove debug print in daml-lf/interpreter build (#11091) 1ed6428 LF: move archive snapshots in a separate directory (#11081) 5e424f8 Command dedup conformance suites readme (#11051) 8290347 DPP-496 HA indexer integration test (#11033) 735c309 Switch from InputStream to Byte Array at the binary content JDBC transport (#11072) 6ae3afa fix perf reporting (#11073) bf801a6 Fix SimpleDeduplicationBasic when participant deduplication is disabled (#11083) 62234dc [Short] Remove unused code (#11079) c1d1521 interface methods: Scala AST (#11070) 7dd9c2d Remove expectations for internal failures from parallel command dedup tests (#11061) 5112599 Explicit discard in daml-lf/interpreter (#11067) 554b36c rotate release duty after 1.18.0-snapshot.20210928.7948.0.b4d00317 (#11059) d156964 Release SDK 1.17.0 (#11062) 6d8cf70 Handle interface methods in LF conversion (#11054) 87ecf1f daml-ledger: better errors for non-json responses. (#11055) ec2d26f [Divulgence pruning] Fixes divulgence pruning offset update query (#11046) f13c6d6 release 1.18.0-snapshot.20210928.7948.0.b4d00317 (#11058) d1805a3 More dependabot fun (#11063) 07b273a update NOTICES file (#11060) ``` Changelog: ``` - [JSON-API] make key_hash indexes non-unique, this fixes a bug where a duplicate key conflict was raised on the query store when the same contract was being witnessed twice by two separate parties * [Trigger Service] Enable the new `tablePrefix` setting in the `--jdbc` flag to add a prefix to all tables used by the trigger service to avoid collisions with other components using the same db-schema. * [Trigger Service] Enable the new ``--allow-existing-schema`` flag to initialize the trigger service on a database with a pre-existing schema. - [Daml Assistant] Add support for `--max-inbound-message-size` flag to the Ledger Export tool. Add cons, find, delete and deleteBy functions to NonEmpty module. ``` CHANGELOG_BEGIN CHANGELOG_END Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>

rautenrieth-da added 2 commits September 27, 2021 14:30

Add indexer stability test

0b73080

changelog_begin changelog_end

Add test for DebugStorageBackend

47180b3

rautenrieth-da requested review from meiersi-da and a team as code owners September 27, 2021 12:56

rautenrieth-da added 4 commits September 27, 2021 16:17

Reduce the log output for IndexerStabilitySpec

b2d6a44

Add Oracle test

f10385a

Rename file

1c334b1

Increase timeout

26f96d2

rautenrieth-da force-pushed the dpp-496-ha-indexer-integration-test branch from 4b4302c to 26f96d2 Compare September 27, 2021 20:16

nmarton-da reviewed Sep 27, 2021

View reviewed changes

rautenrieth-da added 6 commits September 28, 2021 09:19

Rename DebugStorageBackend

bbd6463

Address review comments

91540f6

Fixup rename

e4c32d3

fixup address review comments

9c86dc1

Refactor IndexerStabilitySpec

668e45c

Reset can only be called on an aborted read service

e25c7ca

rautenrieth-da requested a review from nmarton-da September 28, 2021 19:53

Non-HA test can fail for multiple reasons

7d37bfe

nmarton-da approved these changes Sep 29, 2021

View reviewed changes

rautenrieth-da added 6 commits September 29, 2021 12:05

Fix logback for Oracle tests

142d7c6

Otherwise logback complains about missing classes at startup

Fix Oracle jdbc url

131bece

Improve assert message

ad675ec

Improve log output

f38e8cf

Otherwise the user has to wait for 10min to get useful log output about the connection issue

Change default log ids

d24a286

The previous defaults are not valid for Oracle

fmt

2f4e08b

rautenrieth-da commented Sep 29, 2021

View reviewed changes

ledger/participant-integration-api/src/main/scala/platform/indexer/ha/HaCoordinator.scala Show resolved Hide resolved

rautenrieth-da merged commit 8290347 into main Sep 29, 2021

rautenrieth-da deleted the dpp-496-ha-indexer-integration-test branch September 29, 2021 20:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DPP-496 HA indexer integration test #11033

DPP-496 HA indexer integration test #11033

rautenrieth-da commented Sep 27, 2021 •

edited

Loading

nmarton-da left a comment

nmarton-da left a comment

nmarton-da Sep 29, 2021

rautenrieth-da Sep 29, 2021

nmarton-da Sep 29, 2021

DPP-496 HA indexer integration test #11033

DPP-496 HA indexer integration test #11033

Conversation

rautenrieth-da commented Sep 27, 2021 • edited Loading

nmarton-da left a comment

Choose a reason for hiding this comment

nmarton-da left a comment

Choose a reason for hiding this comment

nmarton-da Sep 29, 2021

Choose a reason for hiding this comment

rautenrieth-da Sep 29, 2021

Choose a reason for hiding this comment

nmarton-da Sep 29, 2021

Choose a reason for hiding this comment

rautenrieth-da commented Sep 27, 2021 •

edited

Loading