Run many tasks on one worker

[Totktonada]

test-run puts tasks into queues at start, but size of those queues are not infinite (AFAIR, it is restricted by size of a pipe buffer).

How to reproduce:

./test/test-run.py -j 1 $(yes box-tap/key_def.test.lua | head -n 10000)

[Totktonada]

While refactoring of the input/output queues processing seems a bit complicated, I've created a simple workaround.

diff --git a/lib/worker.py b/lib/worker.py
index db54de9..d7cd479 100644
--- a/lib/worker.py
+++ b/lib/worker.py
@@ -95,12 +95,24 @@ def get_task_groups():
         stable_task_ids = [task.id for task in suite.stable_tests()]
         fragile_task_ids = [task.id for task in suite.fragile_tests()]
         if stable_task_ids:
-            res[key] = {
-                'gen_worker': gen_worker,
-                'task_ids': stable_task_ids,
-                'is_parallel': suite.is_parallel(),
-                'show_reproduce_content': suite.show_reproduce_content(),
-            }
+            if len(stable_task_ids) > 100:
+                group_max_size = 100
+                group_num = 1
+                for i in range(0, len(stable_task_ids), group_max_size):
+                    res[key + '_' + str(group_num)] = {
+                        'gen_worker': gen_worker,
+                        'task_ids': stable_task_ids[i:i+group_max_size],
+                        'is_parallel': suite.is_parallel(),
+                        'show_reproduce_content': suite.show_reproduce_content(),
+                    }
+                    group_num += 1
+            else:
+                res[key] = {
+                    'gen_worker': gen_worker,
+                    'task_ids': stable_task_ids,
+                    'is_parallel': suite.is_parallel(),
+                    'show_reproduce_content': suite.show_reproduce_content(),
+                }
         if fragile_task_ids:
             res[key + '_fragile'] = {
                 'gen_worker': gen_worker,

[Totktonada]

I've found that the problem with blocking a write to the underlying pipe is solved in multiprocessing.Queue (even on Python 2):

When a process first puts an item on the queue a feeder thread is started which transfers objects from a buffer into the pipe.

SimpleQueue writes to the underlying pipe directly. Queue accumulates incoming data in an intermediate buffer that is flushed from a separate thread in background.