jumbotest_messenger_messenger_test timesout on jenkins

[eppingere]

jumbotest_messenger_messenger_test timesout on jenkins

Summary

http://jenkins.db.cs.cmu.edu:8080/blue/organizations/jenkins/terrier/detail/master/791/tests
It seems that the test times out. Potentially due to deadlock or other hangup.

Environment

OS: Ubuntu 20.04 (LTS)

Compiler: GCC 9.3 release

[crd477]

I think there is no output only because the test was terminated by timeout since it ran too long.

[eppingere]

Wait did it? It took 50 mins and the timeout is an hour? Is the timeout short?

[crd477]

There appears to be a discrepancy between the Jenkins stage designation and the actual ctest invocation. The enforced timeout is apparently 3000 seconds. See the full pipeline stage results here:
http://jenkins.db.cs.cmu.edu:8080/blue/rest/organizations/jenkins/pipelines/terrier/branches/master/runs/791/nodes/47/log/?start=0

Specifically:

[0/1] cd /jenkins/workspace/terrier_master@3/build && ctest --resource-spec-file /jenkins/workspace/terrier_master@3/build-support/data/ctest_resource_specs.json -j 24 -L jumbotests --no-compress-output -T Test --timeout 3000
[...]
37/40 Test #162: jumbotest_messenger_messenger_test ...............***Timeout 3000.08 sec
[...]

[eppingere]

Hmmmm ok i will update to be about the test timing out

[lmwnshn]

I've been trying to debug this and haven't been able to reproduce it locally. It happened before too.

[lmwnshn]

Today: it appears that sometimes this happens when there's a lingering port and it fails to bind one of the replicas.

[lmwnshn]

In retrospect, I am dumb and there is a race condition especially in BasicListenTest where B tries to send before A is ready. Will be fixed in an upcoming PR that adds a spin_until_init() to each forked process + sets some ZeroMQ flags to only queue on completed connections + switch from indefinite send/recv blocking to a 1s timeout.

[lmwnshn]

In general, the state of testing forked processes seems to be sad.

[lmwnshn]

http://jenkins.db.cs.cmu.edu:8080/blue/organizations/jenkins/terrier/detail/master/847/pipeline/ Testing forked processes sucks.

[linmagit]

Unfortunately, this shows up again:(
http://jenkins.db.cs.cmu.edu:8080/blue/organizations/jenkins/terrier/detail/PR-1509/23/pipeline

[lmwnshn]

This test no longer exists.