CI Tests: Investigate randomly failing tests #325
Description
Sometimes, tests are failing, but the error why the test is failing is not (always) reproducible.
after improvements in the DevP2P protocol, some of the tests have been taken out from the regular test pipeline to a specific one,
what slows down further development.
One of the tests that has been identified is should_return_unordered_and_not_populate_the_cache_with_non_enforced_fees
It could be connected to the introduction of deadline timing based logic in the DevP2P Protocol.
For example, if the server detects it is to slow to answer a request, for example PooledTransactions, it returns what it can provide within the given timeframe.
This solved locking problems of the Hbbft Engine, that could not work, because DevP2P was to busy and holding locks.
If the Github Testing machines does a lot of context switches on slow hardware, this could easily lead to the situation that a test fails, just because a context switch is not going to happen for example within 250ms.
At the moment, this timings are hardcoded.
To have it configurable would be beneficial anyway, so we can call the tests with larger deadlines.
This is just a theory that needs to get proven, it could also be connected to the a different source of origin.