RuntimeError with run_until_complete()

[gbtami]

PyChess is a chess client app https://github.com/pychess/pychess You can play human-human human-engine engine-engine games or play on freecehess.org also. I like to migrate it from threading to asyncio because of hard to track deadlock issues we are facing time to time. We run chess engines (like https://stockfishchess.org/) in background processes using threads to enable two way communication. Current SubProcess class is here: https://github.com/pychess/pychess/blob/master/lib/pychess/System/SubProcess.py

First I changed human-human game play to use asyncio in pychess asyncio branch. So far so good. Now I'm experimenting with create asyncio based subprocess to communicate with chess engines and get:

  File "Subproc_Asyncio.py", line 86, in on_subprocess
    proc = loop.run_until_complete(start("/usr/games/stockfish"))
  File "/usr/local/lib/python3.5/dist-packages/gbulb/glib_events.py", line 129, in run_until_complete
    self.run_forever(**kw)
  File "/usr/local/lib/python3.5/dist-packages/gbulb/glib_events.py", line 368, in run_forever
    super().run_forever()
  File "/usr/local/lib/python3.5/dist-packages/gbulb/glib_events.py", line 142, in run_forever
    "Recursively calling run_forever is forbidden. "
RuntimeError: Recursively calling run_forever is forbidden. To recursively run the event loop, call run().

import gi
gi.require_version('Gtk', '3.0')
from gi.repository import GLib, Gio, Gtk

import gbulb
gbulb.install(gtk=True)

MENU_XML="""
<?xml version="1.0" encoding="UTF-8"?>
<interface>
  <menu id="app-menu">
    <section>
      <item>
        <attribute name="action">app.subprocess</attribute>
        <attribute name="label">Subprocess</attribute>
      </item>
      <item>
        <attribute name="action">app.quit</attribute>
        <attribute name="label">Quit</attribute>
    </item>
    </section>
  </menu>
</interface>
"""


@asyncio.coroutine
def start(cmd):
    create = asyncio.create_subprocess_exec(cmd,
                                            stdin=asyncio.subprocess.PIPE,
                                            stdout=asyncio.subprocess.PIPE)
    proc = yield from create
    return proc


@asyncio.coroutine
def write(writer, line):
    try:
        print(line)
        writer.write(line)
        yield from writer.drain()
        # writer.close()
    except BrokenPipeError:
        print('stdin: broken pipe error')
    except ConnectionResetError:
        print('stdin: connection reset error')


@asyncio.coroutine
def read_stdout(reader):
    while True:
        line = yield from reader.readline()
        if line:
            print(line)

class Application(Gtk.Application):
    def __init__(self):
        Gtk.Application.__init__(self, application_id="org.subprocess")
        self.window = None

    def do_startup(self):
        Gtk.Application.do_startup(self)

        action = Gio.SimpleAction.new("subprocess", None)
        action.connect("activate", self.on_subprocess)
        self.add_action(action)

        action = Gio.SimpleAction.new("quit", None)
        action.connect("activate", self.on_quit)
        self.add_action(action)

        builder = Gtk.Builder.new_from_string(MENU_XML, -1)
        self.set_app_menu(builder.get_object("app-menu"))

    def do_activate(self):
        if not self.window:
            self.window = Gtk.ApplicationWindow(application=self)

        self.window.present()

    def on_subprocess(self, action, param):
        print("subprocess")
        loop = asyncio.get_event_loop()
        proc = loop.run_until_complete(start("/usr/games/stockfish"))
        print(proc.pid)
        asyncio.ensure_future(read_stdout(proc.stdout))
        asyncio.ensure_future(write(proc.stdin, b"uci\n"))

    def on_quit(self, action, param):
        self.quit()


if __name__ == "__main__":
    app = Application()

    loop = asyncio.get_event_loop()
    loop.run_forever(application=app)

[nhoad]

Yep, this is correct - you can't use both run_until_complete and run_forever. You'll need to do something like this...

--- orig.py	2017-03-28 22:13:44.002777334 +1000
+++ new.py	2017-03-28 22:19:30.261224670 +1000
@@ -1,3 +1,5 @@
+import asyncio
+
 import gi
 gi.require_version('Gtk', '3.0')
 from gi.repository import GLib, Gio, Gtk
@@ -25,11 +27,12 @@
 
 
 @asyncio.coroutine
-def start(cmd):
+def start(cmd, loop):
     create = asyncio.create_subprocess_exec(cmd,
                                             stdin=asyncio.subprocess.PIPE,
                                             stdout=asyncio.subprocess.PIPE)
     proc = yield from create
+    loop.stop()
     return proc
 
 
@@ -52,6 +55,9 @@
         line = yield from reader.readline()
         if line:
             print(line)
+        # (tiny unrelated bugfix, handle the stream closing)
+        else:
+            break
 
 class Application(Gtk.Application):
     def __init__(self):
@@ -81,7 +87,9 @@
     def on_subprocess(self, action, param):
         print("subprocess")
         loop = asyncio.get_event_loop()
-        proc = loop.run_until_complete(start("/usr/games/stockfish"))
+        f = asyncio.ensure_future(start("/usr/games/stockfish", loop))
+        loop.run()
+        proc = f.result()
         print(proc.pid)
         asyncio.ensure_future(read_stdout(proc.stdout))
         asyncio.ensure_future(write(proc.stdin, b"uci\n"))

Which is a frustrating, roundabout way of doing what you originally intended - if you could confirm this works for you that would be good, but I'd like to leave this issue open to motivate me into making run_until_complete work in the way you tried to use it. It looks more intuitive to me.

[gbtami]

Yes this workaround worked. Confirmed.

[gbtami]

Just notice that after windows support merge this workaround doesn't work. There is no output on stdout at all.

[nhoad]

Hmm, that's puzzling. I'll take a look at this over the weekend and try to figure out why. Thanks for testing that!

[nhoad]

A small update - I've reproduced this and tracked the bug down to pipe transports being blocking, and our use of GLib.IOChannels being buffered. I've got a fix for it, I'm just coming up with a reliable test before committing to anything.

[nhoad]

Alright, I've got my test for this sorted, so I think it should be working with current master - could you give that another go? Then I can start working on a version of run_until_complete that works in an already running loop.

[gbtami]

Seems OK on Linux, but fails on Windows.
In glib_events.py "import fcntl" and os.set_blocking() are Unix only :(
Filed it as issue #23

[gbtami]

Simply changing run_forever() call to run() call inside def run_until_complete() in glib_events.py seems to work in my case. Do you think can it has any drawback?

[nhoad]

Apologies for radio silence! I've been away on holiday. I'll take a look as soon as I can :)

[freakboy3742]

This appears to be related to Windows support; the README calls out that Gbulb doesn't support Windows.

I'm very much open to a PR that addresses this (although #32 would also need to be addressed); but until the project actually claims support for Windows, reporting bugs in Windows support isn't reasonable.