C5GeneralizedReplicationServiceTest is flaky

[joshua-g]

C5GeneralizedReplicationServiceTest, single node case, timed out in https://circleci.com/gh/OhmData/c5/57 . Output:

Running c5db.replication.C5GeneralizedReplicationServiceTest
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 18.864 sec <<< FAILURE! - in c5db.replication.C5GeneralizedReplicationServiceTest
logsToASingleQuorumReplicator(c5db.replication.C5GeneralizedReplicationServiceTest) Time elapsed: 16.13 sec <<< ERROR!
java.lang.Exception: test timed out after 9000 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:292)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at c5db.replication.C5GeneralizedReplicationServiceTest$QuorumOfReplicatorsController.waitUntilAReplicatorIsReady(C5GeneralizedReplicationServiceTest.java:309)
at c5db.replication.C5GeneralizedReplicationServiceTest.logsToASingleQuorumReplicator(C5GeneralizedReplicationServiceTest.java:105)

[joshua-g]

I have run this over a thousand times locally with no failure. I'm wondering if exceeding the 9 second timeout could have happened due to some momentary slowness on the circle build machine. The test normally takes ~ 2 seconds on my computer.

Either way, the next PR to OhmData/c5-replicator will edit the circle.yml file to save the test logger output as a build artifact. That way if something goes wrong again, there might be something useful in the log.

[ryanobjc]

i wonder if we could get GC logs for the test run as well?

[joshua-g]

Sure. I'll add that to the PR too.