patch_stdout

[iwasz]

Hi, I have problems with an ./asyncio-prompt.py example. Running it under Ubuntu 18.10, Python 3.6.8 source taken from git master branch (today) I encountered :

./asyncio-prompt.py
Traceback (most recent call last):
File "./asyncio-prompt.py", line 64, in
main()
File "./asyncio-prompt.py", line 57, in main
loop.run_until_complete(shell_task)
File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "./asyncio-prompt.py", line 41, in interactive_shell
session = PromptSession('Say something: ')
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/shortcuts/prompt.py", line 433, in init
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/shortcuts/prompt.py", line 671, in _create_application
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/application.py", line 260, in init
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/current.py", line 61, in output
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/output/defaults.py", line 29, in create_output
AttributeError: 'StdoutProxy' object has no attribute 'isatty'

After adding isatty method (my method returns always True), the error has changed to:

./asyncio-prompt.py
Counter: 0
Traceback (most recent call last):
File "./asyncio-prompt.py", line 64, in
main()
File "./asyncio-prompt.py", line 57, in main
loop.run_until_complete(shell_task)
File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "./asyncio-prompt.py", line 46, in interactive_shell
result = await session.prompt_async ()
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/shortcuts/prompt.py", line 1030, in prompt_async
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/application.py", line 738, in run_async
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/application.py", line 723, in _run_async2
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/application.py", line 664, in _run_async
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/application.py", line 878, in _request_absolute_cursor_position
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/renderer.py", line 443, in request_absolute_cursor_position
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/renderer.py", line 435, in do_cpr
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/output/vt100.py", line 663, in ask_for_cpr
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/output/vt100.py", line 634, in flush
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/patch_stdout.py", line 141, in write
File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/patch_stdout.py", line 121, in _write
TypeError: a bytes-like object is required, not 'str'

The only thing I came up to with this then, was to remove "with patch_stdout():" from the example whatsoever.

Best regards.

[lillian27]

iwasz,

Not a fix but more information about the issue.

patch_stdout.py is missing isatty as you observed, and should probably just pass through that property from the original stdout:

def isatty(self) -> bool:
        return self.original_stdout.isatty()

I believe the bytes-alike vs str issue is due to how the StdoutProxy class now works. The TypeErrors are the result of PT trying to get a cursor position report from the terminal by writing '\x1b[6n' to what it thinks is stdout .

output/vt100.py

    def ask_for_cpr(self) -> None:
        """
        Asks for a cursor position report (CPR).
        """
        self.write_raw('\x1b[6n')
        self.flush()

So Vt100Output does a raw write to its internal buffer - fine so far - but when it calls flush() this happens:

output/vt100.py

            if self.write_binary:
                if hasattr(self.stdout, 'buffer'):
                    out = self.stdout.buffer  # Py3.
                else:
                    out = self.stdout
                out.write(data.encode(self.stdout.encoding or 'utf-8', 'replace'))
            else:
                self.stdout.write(data)

The first 'if' is True, the second False, so the buffer gets written to self.stdout which is a StdoutProxy object. The proxy then tries to split out newlines (to avoid excessive write() calls, see stdout_proxy.py:113), and expecting a 'str' gets a bytes-like instead, raising the TypeError. If the data variable at line 121 is examined it does hold the control code for cursor position.

StdoutProxy understands about writing binary vs not but only when invoked:

    def __init__(self, raw: bool = False,
                 original_stdout: Optional[TextIO] = None) -> None:
        ...
        self._raw = raw

However the _raw attribute is not actually used anywhere so it looks like the proxy needs to be extended to enable raw/cooked mode switching as needed. This is as far down the rabbit hole as I went. I don't know how Jonathan might want to fix this since 3.0 is in-progress so I haven't tried to fix it myself. I hope my analysis isn't too badly incorrect ;-)

[iwasz]

Wow, very through analysis. I added my stub isatty only to check if it's the only problem that was, or not. By any means I wanted to propose this as a fix :D Thanks.

…

On Wed, Jul 3, 2019 at 11:14 PM lillian27 ***@***.***> wrote: iwasz, Not a fix but more information about the issue. patch_stdout.py is missing isatty as you observed, and should probably just pass through that property from the original stdout: def isatty(self) -> bool: return self.original_stdout.isatty() I believe the bytes-alike vs str issue is due to how the StdoutProxy class now works. The TypeErrors are the result of PT trying to get a cursor position report from the terminal by writing '\x1b[6n' to what it thinks is stdout . output/vt100.py def ask_for_cpr(self) -> None: """ Asks for a cursor position report (CPR). """ self.write_raw('\x1b[6n') self.flush() So Vt100Output does a raw write to its internal buffer - fine so far - but when it calls flush() this happens: output/vt100.py if self.write_binary: if hasattr(self.stdout, 'buffer'): out = self.stdout.buffer # Py3. else: out = self.stdout out.write(data.encode(self.stdout.encoding or 'utf-8', 'replace')) else: self.stdout.write(data) The first 'if' is True, the second False, so the buffer gets written to self.stdout which is a StdoutProxy object. The proxy then tries to split out newlines (to avoid excessive write() calls, see stdout_proxy.py:113), and expecting a 'str' gets a bytes-like instead, raising the TypeError. If the data variable at line 121 is examined it does hold the control code for cursor position. StdoutProxy understands about writing binary vs not but only when invoked: def __init__(self, raw: bool = False, original_stdout: Optional[TextIO] = None) -> None: ... self._raw = raw However the _raw attribute is not actually used anywhere so it looks like the proxy needs to be extended to enable raw/cooked mode switching as needed. This is as far down the rabbit hole as I went. I don't know how Jonathan might want to fix this since 3.0 is in-progress so I haven't tried to fix it myself. I hope my analysis isn't too badly incorrect ;-) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#934?email_source=notifications&email_token=AAB4N334OCHOD3MK7P2WEMDP5UJEBA5CNFSM4H5EUDQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZFWIQQ#issuecomment-508257346>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB4N36N3ELY3NVGXKD3CVTP5UJEBANCNFSM4H5EUDQA> .

[sthibaul]

This is still an issue for me with 3.0.7.