Ignore flaky test #13631

ggwpez · 2023-03-17T14:48:26Z

Disabling all of paritytech/polkadot-sdk#48 until they are confirmed to not be flaky anymore.
https://github.com/paritytech/ci_cd/issues/758 should should help to spot and verify the flaky-ness of tests.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

bkchr · 2023-03-17T14:49:33Z

bin/node/cli/tests/running_the_node_and_interrupt.rs

+#[cfg(disable_flaky)]
+#[allow(dead_code)]
+// FIXME: https://github.com/paritytech/substrate/issues/11321


This should also be fixed with my latest pr or have you seen it failing since then?

Have not rechecked it. The only ones that seem to be up-to-date flaky are follow_report_multiple_pruned_block and follow_forks_pruned_block.
I can re-enable it if you want.

Yeah please reenable it.

If you also have not seen the other ones failing recently, we should also not disable them

Okay just re-checked: returns_status_for_pruned_blocks failed two days ago, so also diabling.
The other two ones are re-enabled.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

bkchr · 2023-03-17T14:52:37Z

client/rpc-spec-v2/src/chain_head/tests.rs

@@ -1024,6 +1024,9 @@ async fn follow_prune_best_block() {
 }

 #[tokio::test]


Extracting the error from the CI job(https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2529507/raw):

--- STDERR: sc-rpc-spec-v2 chain_head::tests::follow_forks_pruned_block --- thread 'chain_head::tests::follow_forks_pruned_block' panicked at 'assertion failed: `(left == right)` left: `Finalized(Finalized { finalized_block_hashes: ["0x0af997b079e96b7f333bf2fe71cb914ad3eb49355923fd3db96c0cc91243bd7a"], pruned_block_hashes: [] })`, right: `BestBlockChanged(BestBlockChanged { best_block_hash: "0x0af997b079e96b7f333bf2fe71cb914ad3eb49355923fd3db96c0cc91243bd7a" })

These seem to be flaky for at least follow_forks_pruned_block, I couldn't find in the logs the error for follow_report_multiple_pruned_block but I expect that to also be affected.

Offhand looking at the code it seems that the Finalized event is received before NewBlock event. Which is not generating the BestBlock before the Finalized.

I expect the error is coming from:

substrate/client/rpc-spec-v2/src/chain_head/chain_head_follow.rs

Lines 394 to 398 in 0506da0

self.best_block_cache = Some(*hash);

},

// This is the first best block event that we generate.

None => {

self.best_block_cache = Some(*hash);

Those lines should already be handled by the generate_import_events.

Thanks for pointing this out! Will create a patch for it!

Fix published at: #13632, will make a note to reenable the tests if this PR gets merged first! Thanks!

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

lexnv

LGTM! 👍

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

ggwpez · 2023-03-17T15:43:42Z

bot merge

paritytech-processbot · 2023-03-17T15:43:50Z

Waiting for commit status.

mateo-moon · 2023-03-20T17:05:39Z

Instead of disabling flaky we could use the feature of nextest with retries and marking flaky tests.

* Ignore flaky test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Re-enable running_the_node_works_and_can_be_interrupted Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Re-enable notifications_back_pressure Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Unused import Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

mateo-moon · 2023-05-05T12:22:34Z

Nextest flaky setting has been turned on and we now collect flaky tests data. Means that you don't need to disable tests manually. They are safe to fail with 5 retries. The results of such tests will be reported.

#13654

bkchr · 2023-05-05T13:48:08Z

@oleg-plakida where are these reports collected? How can we see them?

* Ignore flaky test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Re-enable running_the_node_works_and_can_be_interrupted Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Re-enable notifications_back_pressure Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Unused import Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Ignore flaky test

810f2da

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

bkchr reviewed Mar 17, 2023

View reviewed changes

bkchr approved these changes Mar 17, 2023

View reviewed changes

Re-enable running_the_node_works_and_can_be_interrupted

71efc2f

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

bkchr reviewed Mar 17, 2023

View reviewed changes

ggwpez added 2 commits March 17, 2023 15:53

Re-enable notifications_back_pressure

f722f17

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Unused import

81ef3cb

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

lexnv approved these changes Mar 17, 2023

View reviewed changes

Merge master for faster tests

6655b0d

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

lexnv mentioned this pull request Mar 17, 2023

fix/chain_head: Ensure correct events for finalized branch #13632

Merged

1 task

paritytech-processbot bot merged commit 44c5c99 into master Mar 17, 2023

paritytech-processbot bot deleted the oty-ignore-test branch March 17, 2023 16:06

github-actions bot mentioned this pull request Jun 15, 2023

Update substrate/polkadot/cumulus from v0.9.40 to v0.9.43 moonbeam-foundation/moonbeam#2354

Closed

kacperzuk-neti mentioned this pull request Jun 23, 2023

Polkadot v0.9.43 liberland/liberland_substrate#295

Merged

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ignore flaky test #13631

Ignore flaky test #13631

ggwpez commented Mar 17, 2023 •

edited

Loading

bkchr Mar 17, 2023

ggwpez Mar 17, 2023

bkchr Mar 17, 2023

bkchr Mar 17, 2023

ggwpez Mar 17, 2023 •

edited

Loading

bkchr Mar 17, 2023

lexnv Mar 17, 2023

lexnv Mar 17, 2023

lexnv left a comment

ggwpez commented Mar 17, 2023

paritytech-processbot bot commented Mar 17, 2023

mateo-moon commented Mar 20, 2023

mateo-moon commented May 5, 2023 •

edited

Loading

bkchr commented May 5, 2023

		@@ -1024,6 +1024,9 @@ async fn follow_prune_best_block() {
		}

		#[tokio::test]

	self.best_block_cache = Some(*hash);
	},
	// This is the first best block event that we generate.
	None => {
	self.best_block_cache = Some(*hash);

Ignore flaky test #13631

Ignore flaky test #13631

Conversation

ggwpez commented Mar 17, 2023 • edited Loading

bkchr Mar 17, 2023

Choose a reason for hiding this comment

ggwpez Mar 17, 2023

Choose a reason for hiding this comment

bkchr Mar 17, 2023

Choose a reason for hiding this comment

bkchr Mar 17, 2023

Choose a reason for hiding this comment

ggwpez Mar 17, 2023 • edited Loading

Choose a reason for hiding this comment

bkchr Mar 17, 2023

Choose a reason for hiding this comment

lexnv Mar 17, 2023

Choose a reason for hiding this comment

lexnv Mar 17, 2023

Choose a reason for hiding this comment

lexnv left a comment

Choose a reason for hiding this comment

ggwpez commented Mar 17, 2023

paritytech-processbot bot commented Mar 17, 2023

mateo-moon commented Mar 20, 2023

mateo-moon commented May 5, 2023 • edited Loading

bkchr commented May 5, 2023

ggwpez commented Mar 17, 2023 •

edited

Loading

ggwpez Mar 17, 2023 •

edited

Loading

mateo-moon commented May 5, 2023 •

edited

Loading