Cant decode token (invalid start byte)

Hi,
i try to discover the vacuum robot on macos.
mirobo discover does find it, but cannot decode or show the token:

(dustcloud-PBjoGnIX) bash-3.2$ mirobo discover --handshake true
INFO:miio.device:Sending discovery to with timeout of 5s..
WARNING:miio.device:error while reading discover results: 'utf-8' codec can't decode byte 0xb2 in position 3: invalid start byte
(dustcloud-PBjoGnIX) bash-3.2$ 5~

Is there some kind of workaround?
Or is it possible to write the token undecoded to a file?

[rytilahti]

Could you please try mirobo discover --debug --handshake true to get more output why it fails?

Great idea!
Thank you very mutch.

(dustcloud-PBjoGnIX) bash-3.2$ mirobo --debug discover --handshake true
INFO:miio.vacuum_cli:Debug mode active
INFO:miio.device:Sending discovery to <broadcast> with timeout of 5s..
DEBUG:miio.protocol:Unable to decrypt, returning raw bytes: b''
DEBUG:miio.device:Got a response: Container: 
    data = Container: 
        data =  (total 0)
        value =  (total 0)
        offset1 = 32
        offset2 = 32
        length = 0
    header = Container: 
        data = !1\x00 \x00\x00\x00\x00\x04o\x04\xb2\x00\x016\x9c (total 16)
        value = Container: 
            length = 32
            unknown = 0
            device_id = \x04o\x04\xb2 (total 4)
            ts = 1970-01-01 22:05:16
        offset1 = 0
        offset2 = 16
        length = 16
    checksum = KXY2qGP2N6jCu9NN (total 16)
WARNING:miio.device:error while reading discover results: 'utf-8' codec can't decode byte 0xb2 in position 3: invalid start byte

[rytilahti]

Looks like it doesn't work anymore in newer versions of construct, the last known version to work was construct==2.9.28 . You can downgrade the package for testing, and then update it back to 2.9.30 (in case some parts of the protocol are broken in 2.9.28, I'm not sure anymore what breaks and when).

[arekbulski]

It looks like a problem with one of String classes (there are 4 in construct) but I cant make what is the problem from just looking at this log. String classes recently added support for UTF16/32 but that clearly aint the problem, they also droped support for encoding=None but that aint either. Can you link me the exact source line that parses that string?

If its String or CString class, then encoding must be on this list: (utf-8 is not, utf8 is)
https://construct.readthedocs.io/en/latest/api/strings.html#construct.possiblestringencodings
If that happens to be the problem, send me a PR to include it.

[yawor]

All Construct code used for miio protocol is in miio.protocol module. There're no Strings in the Message struct.

The problem is with Message.header.value.device_id. It is defined as "device_id" / Hex(Bytes(4)). Up to Construct 2.9.28 this produced bytes in the decoded message, since 2.9.29 it produces construct.lib.hex.HexDisplayedBytes. When trying to decode() that object (miio/device.py at line 187) it raises the above exception.

[arekbulski]

Right, but you got it half wrong. Hex/HexDump still produce bytes or rather bytes-derived objects, its just that the content was changed. It used to produce hexlified bytes, now it produces the original bytes but with modified __repr__. See example:
https://construct.readthedocs.io/en/latest/misc.html#hex-and-hexdump

It seems I should expand the example:

>>> d = Hex(GreedyBytes)
>>> obj = d.parse(b"\x00\x00\x01\x02")
>>> obj
b'\x00\x00\x01\x02'
>>> print(obj)
unhexlify('00000102')

[arekbulski]

The code you pointed at:
https://github.com/rytilahti/python-miio/blob/master/miio/device.py#L187

Simple fix: (you probably need to adjust few other lines as well)

binascii.hexlify(m.header.value.device_id).decode(),

I will update Construct docs to mention this.

EDIT: For explanation, the decode method exists on returned bytes as well, but the contents is not hexlified and therefore contain values that arent "utf8" decodable.

[syssi]

Who is affected of this issue?

[arekbulski]

I updated the API page to be more accurate what Hex results in.
https://construct.readthedocs.io/en/latest/misc.html#hex-and-hexdump

If I may explain myself, the Hex class was never meant (by me) to transform bytes, it was meant to provide hex view of it. But I was sloppy and never got around to properly implementing it, until now. Thats why API change. Old implemenataion was just stale code that wasnt written by me.

[smartjx]

I'm facing the same issue when I try to connect a mi ir remote

root@hassbian:~# mirobo --debug discover --handshake true
INFO:miio.vacuum_cli:Debug mode active
INFO:miio.device:Sending discovery to with timeout of 5s..
DEBUG:miio.protocol:Unable to decrypt, returning raw bytes: b''
DEBUG:miio.device:Got a response: Container:
data = Container:
length = 0
value = (total 0)
offset1 = 32
data = (total 0)
offset2 = 32
header = Container:
length = 16
value = Container:
length = 32
unknown = 0
device_id = \x05;\xdc[ (total 4)
ts = 1970-01-01 22:07:47
offset1 = 0
data = !1\x00 \x00\x00\x00\x00\x05;\xdc[\x00\x0173 (total 16)
offset2 = 16
checksum = \xa5@yi\xcd\x0f\xb2yA\x1c\xc3\xe2b\xf5\xde\xbc (total 16)
WARNING:miio.device:error while reading discover results: 'utf-8' codec can't decode byte 0xdc in position 2: invalid continuation byte

[syssi]

@smartjx The issue will be fixed with the merge of https://github.com/rytilahti/python-miio/pull/226/files

[smartjx]

I've done pip3 install construct==2.9.31 but seems this issue has not been resolved