Skip to content
This repository was archived by the owner on Oct 21, 2025. It is now read-only.

Decode stdout using preferred encoding not ascii, and combine multibyte as necessary #13

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Decode stdout using preferred encoding not ascii, and combine multibyte as necessary #13

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fee5a51
Decode stdout using preferred encoding not ascii, and combine multiby…
yarikoptic Apr 18, 2022
1524249
Assert that e.start should never be negative
yarikoptic May 7, 2024
dc97bd9
Add handling of remaining_text
yarikoptic May 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions epdb/epdb_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@

import errno
import fcntl
import locale
import os
import select
import signal
Expand Down Expand Up @@ -102,10 +103,12 @@ def write(self, buffer):
else:
telnetlib.Telnet.write(self, buffer)

def interact(self):
def interact(self, encoding=None):
encoding = encoding or locale.getpreferredencoding()
self.set_raw_mode()
try:
self.updateTerminalSize()
remaining_text = b""
while not self.eof:
readyWriters = []
readyReaders = []
Expand Down Expand Up @@ -143,13 +146,25 @@ def interact(self):
print('*** Connection closed by remote host ***')
break
if text:
sys.stdout.write(text.decode('ascii'))
all_text = remaining_text + text
try:
unicode_text = all_text.decode(encoding)
remaining_text = b""
except UnicodeDecodeError as e:
assert e.start < 0, f"Got an unexpected negative start for unicode: {e.start}"
unicode_text = all_text[:e.start].decode(encoding)
remaining_text = all_text[e.start:]
Copy link
Contributor

@mibanescu mibanescu Jun 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am sorry, i now see i had added my comment on the wrong line. Here, remaining_text gets reinitialized with a slice from all_text. If e.start is 0 and the first char is not a valid Unicode start sequence, then remaining_text will be appended to, but will always fail to decode, which, if I am right, leads to the infinite loop.

Just re-raising the exception if e.start <= 0 should be safe i think

Copy link
Author

@yarikoptic yarikoptic May 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just had to use this patch again to realize that we never finished this PR.

For e.start < 0 - I accepted my own assertion above since I think it should simply never happen.

For e.start == 0 -- I think it is unlikely but legit case whenever we read only few bytes which are beginning of an incomplete unicode. Then we just store it all and append to next block -- I do not see how we could get an infinite loop here since we are not adding any looping and should exit at eof (would still do) ... the only thing would be left is at the end to assert that we have no remaining_text left -- as if incomplete unicode at the end was provided and we failed to decode it. I will add a stab for that.

sys.stdout.write(unicode_text)
sys.stdout.flush()
if sys.stdin in readyReaders and self in readyWriters:
line = sys.stdin.read(4096)
if not line:
break
self.write(line.encode('ascii'))
self.write(line.encode(encoding))
if remaining_text:
# trying to decode remaining text which should fail and raise exception
# but that would be just what it is -- there was a non-decodable remainder
sys.stdout.write(remaining_text.decode(encoding))
finally:
self.restore_terminal()

Expand Down