I'm marking this issue with question label but hope to convert it to feature request after some discussions.

Right now LightGBM lacks any tests related to parallel (in located in network folder) code. MPI job which run on some our CIs simply execute ordinary tests with serial single machine tree learner but with dynamic library compiled with MPI support. In other words, we just check that MPI code can be compiled.

Recently added Dask module are getting more and more tests and this is very good but they are

• run only on Linux so far;
• limited to Dask functionality;
• not very clear about errors in terms what is going wrong: either some Dask internal processes or LightGBM underlying code.

I believe it will be good to write some basic tests that will cover low-level LightGBM cpp code. Ideally it should cover both socket and MPI implementations.

Adding such tests will help to improve standalone LightGBM parallel library and in consequence Dask-package built on top of it.

Linking #261.

Refer to #3839 (comment) for test example.