Laggy tests when leftover noisepage instances are on the box.

[lmwnshn]

Bug Report

Summary

Laggy tests when leftover noisepage instances are on the box.

Environment

Ran manually on jenkins h2.

Steps to Reproduce

1. There were three old leftover noisepage processes on h2.

3129392 jenkins   20   0 7159664 525448  27668 R 127.2   0.2  32073:22 noisepage
2573020 jenkins   20   0 8874336 144488  27812 R 126.2   0.1  44112:26 noisepage
3806962 jenkins   20   0   15.3g   2.6g  27556 R 118.6   1.0   8762:42 noisepage

2. While those processes were running, the newly added Model Server Manager Infrastructure for Pilot #1288 modeling_test took many many minutes to run. It ran successfully, but extremely slowly (1 minute in ideal case, to over 30 minutes in strange case).

There are two problems here:

1. Why are there leftover processes? From error'd runs? From old TPCC tests? Is this part of our nightly/long-running testing (if so, let's find a new machine for them)?
2. Why did it get so much slower? There was plenty of free memory, so I imagine it is because of some network stuff. The only port that was being listened on per lsof was 15721 and 9022, but the messenger communicates with IPC in /tmp.

I suspect this may have something to do with the messenger and/or network layer. This may also explain why #1319 occurs sometimes.

Also, perhaps the messenger should actually not create IPC files in /tmp and instead create files in the build directory so that they get cleaned up.

[lmwnshn]

I haven't seen this since the CI refresh.