Usage of PlutusData for Datum deserialization is unclear

[while0x1]

Describe the bug

Pycardano 0.10.0 cannot create an inline datum from CBOR hex - PlutusData().from_cbor(<CBOR_HEX>)

pycardano.exception.DeserializeException: Unexpected constructor ID for <class 'pycardano.plutus.PlutusData'>. Expect None, got 121 instead.

To Reproduce
Find an Inline datum from a blockfrost request - create an empty PlutusData() object , empty = PlutusData()
inline_datum = empty.from_cbor(<cbor_hex>)

empty = PlutusData()
d = empty.from_cbor('d8799f581c7876ebac44945a88855442692b86400776e0a2987c5f54a19b457d86d8799f4040ff1b000000012a05f200d8799f1a001e84801a002191c01a0016e360d8799f19271000193a981a000249f0194e201a0006ddd01961a8d8799fd8799f581c0c8b9cc1657e5139be7a331036c5499f0c2dc09fd8680e9773e4a01affd8799fd8799fd8799f581c6e0defd3cf3a4307652e956b3ca65789ca5b7836ae5494ebc546ad8affffffffff1a001e84801a02faf0801a02faf0801a002dc6c01a000ab34ed8799f1a0006ddd01975301a000124f81a002dc6c0ffffd8799f581c8fef2d34078659493ce161a6c7fba4b56afefa8535296a5743f695874441414441ff1b000000012a05f2001a00011af11b00000052aeaea7801b0000006673cc7c5b1b0000018de01f5008582046ffb569dc0d84d965733a5ff92bb7b2244a6f72d4726558e49d1d41ad4fbc8fd8799f581c13dfcd07acf9c62ae28f7578e637210dddd7f77b393d0983b89c270758202f426424960c554bf256c1e7f2ee74013271613fd6cffdbe1b2f337600ed774cffd8799f581c13dfcd07acf9c62ae28f7578e637210dddd7f77b393d0983b89c270758202f426424960c554bf256c1e7f2ee74013271613fd6cffdbe1b2f337600ed774cffd87a80ff')

Expected behavior
Pycardano 0.8.0 serialises the CBOR_HEX to create the object and deals with the CBOR tags appropriately , Pycardano 0.10.0 fails with error

Environment and software version (please complete the following information):

• PyCardano Version [e.g. 0.8.0]
• PyCardano Version [e.g. 0.10.0]

[cffls]

Debug log pasted from discord:

d = empty.from_cbor('d8799f581c7876ebac44945a88855442692b86400776e0a2987c5f54a19b457d86d8799f4040ff1b000000012a05f200d8799f1a001e84801a002191c01a0016e360d8799f19271000193a981a000249f0194e201a0006ddd01961a8d8799fd8799f581c0c8b9cc1657e5139be7a331036c5499f0c2dc09fd8680e9773e4a01affd8799fd8799fd8799f581c6e0defd3cf3a4307652e956b3ca65789ca5b7836ae5494ebc546ad8affffffffff1a001e84801a02faf0801a02faf0801a002dc6c01a000ab34ed8799f1a0006ddd01975301a000124f81a002dc6c0ffffd8799f581c8fef2d34078659493ce161a6c7fba4b56afefa8535296a5743f695874441414441ff1b000000012a05f2001a00011af11b00000052aeaea7801b0000006673cc7c5b1b0000018de01f5008582046ffb569dc0d84d965733a5ff92bb7b2244a6f72d4726558e49d1d41ad4fbc8fd8799f581c13dfcd07acf9c62ae28f7578e637210dddd7f77b393d0983b89c270758202f426424960c554bf256c1e7f2ee74013271613fd6cffdbe1b2f337600ed774cffd8799f581c13dfcd07acf9c62ae28f7578e637210dddd7f77b393d0983b89c270758202f426424960c554bf256c1e7f2ee74013271613fd6cffdbe1b2f337600ed774cffd87a80ff')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/cardano/.local/lib/python3.10/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
  File "/home/cardano/.local/lib/python3.10/site-packages/pycardano/serialization.py", line 485, in from_cbor
    return cls.from_primitive(value)
  File "/home/cardano/.local/lib/python3.10/site-packages/pycardano/serialization.py", line 162, in wrapper
    return func(cls, value)
  File "/home/cardano/.local/lib/python3.10/site-packages/pycardano/plutus.py", line 573, in from_primitive
    raise DeserializeException(
pycardano.exception.DeserializeException: Unexpected constructor ID for <class 'pycardano.plutus.PlutusData'>. Expect None, got 121 instead.

[cffls]

Looks like the latest version (in main branch) can decode this cbor with RawPlutusData.

myData = RawPlutusData.from_cbor('d8799f581c7876ebac44945a88855442692b86400776e0a2987c5f54a19b457d86d8799f4040ff1b000000012a05f200d8799f1a001e84801a002191c01a0016e360d8799f19271000193a981a000249f0194e201a0006ddd01961a8d8799fd8799f581c0c8b9cc1657e5139be7a331036c5499f0c2dc09fd8680e9773e4a01affd8799fd8799fd8799f581c6e0defd3cf3a4307652e956b3ca65789ca5b7836ae5494ebc546ad8affffffffff1a001e84801a02faf0801a02faf0801a002dc6c01a000ab34ed8799f1a0006ddd01975301a000124f81a002dc6c0ffffd8799f581c8fef2d34078659493ce161a6c7fba4b56afefa8535296a5743f695874441414441ff1b000000012a05f2001a00011af11b00000052aeaea7801b0000006673cc7c5b1b0000018de01f5008582046ffb569dc0d84d965733a5ff92bb7b2244a6f72d4726558e49d1d41ad4fbc8fd8799f581c13dfcd07acf9c62ae28f7578e637210dddd7f77b393d0983b89c270758202f426424960c554bf256c1e7f2ee74013271613fd6cffdbe1b2f337600ed774cffd8799f581c13dfcd07acf9c62ae28f7578e637210dddd7f77b393d0983b89c270758202f426424960c554bf256c1e7f2ee74013271613fd6cffdbe1b2f337600ed774cffd87a80ff')
myData
RawPlutusData(data=CBORTag(121, [b'xv\xeb\xacD\x94Z\x88\x85TBi+\x86@\x07v\xe0\xa2\x98|_T\xa1\x9bE}\x86', CBORTag(121, [b'', b'']), 5000000000, CBORTag(121, [2000000, 2200000, 1500000, CBORTag(121, [10000, 0, 15000, 150000, 20000, 450000, 25000, CBORTag(121, [CBORTag(121, [b'\x0c\x8b\x9c\xc1e~Q9\xbez3\x106\xc5I\x9f\x0c-\xc0\x9f\xd8h\x0e\x97s\xe4\xa0\x1a']), CBORTag(121, [CBORTag(121, [CBORTag(121, [b'n\r\xef\xd3\xcf:C\x07e.\x95k<\xa6W\x89\xca[x6\xaeT\x94\xeb\xc5F\xad\x8a'])])])])]), 2000000, 50000000, 50000000, 3000000, 701262, CBORTag(121, [450000, 30000, 75000, 3000000])]), CBORTag(121, [b'\x8f\xef-4\x07\x86YI<\xe1a\xa6\xc7\xfb\xa4\xb5j\xfe\xfa\x855)jWC\xf6\x95\x87', b'AADA']), 5000000000, 72433, 355118000000, 440029445211, 1708862165000, b'F\xff\xb5i\xdc\r\x84\xd9es:_\xf9+\xb7\xb2$Jor\xd4reX\xe4\x9d\x1dA\xadO\xbc\x8f', CBORTag(121, [b"\x13\xdf\xcd\x07\xac\xf9\xc6*\xe2\x8fux\xe67!\r\xdd\xd7\xf7{9=\t\x83\xb8\x9c'\x07", b'/Bd$\x96\x0cUK\xf2V\xc1\xe7\xf2\xeet\x012qa?\xd6\xcf\xfd\xbe\x1b/3v\x00\xedwL']), CBORTag(121, [b"\x13\xdf\xcd\x07\xac\xf9\xc6*\xe2\x8fux\xe67!\r\xdd\xd7\xf7{9=\t\x83\xb8\x9c'\x07", b'/Bd$\x96\x0cUK\xf2V\xc1\xe7\xf2\xeet\x012qa?\xd6\xcf\xfd\xbe\x1b/3v\x00\xedwL']), CBORTag(122, [])]))

[nielstron]

Find an Inline datum from a blockfrost request - create an empty PlutusData() object , empty = PlutusData(), empty.from_cbor(...)

I highly doubt that this every worked intentionally. This may have worked for data that was coincidentally CONSTR_ID 0 and only primitive fields, but should not have worked for anything else (PlutusData always expected to be subclasses with the type structure in place). RawPlutusData is the way to go.

[nielstron]

Output from pycardano 0.8.0

# test.py
from pycardano import PlutusData

empty = PlutusData()
d = empty.from_cbor('d8799f581c7876ebac44945a88855442692b86400776e0a2987c5f54a19b457d86d8799f4040ff1b000000012a05f200d8799f1a001e84801a002191c01a0016e360d8799f19271000193a981a000249f0194e201a0006ddd01961a8d8799fd8799f581c0c8b9cc1657e5139be7a331036c5499f0c2dc09fd8680e9773e4a01affd8799fd8799fd8799f581c6e0defd3cf3a4307652e956b3ca65789ca5b7836ae5494ebc546ad8affffffffff1a001e84801a02faf0801a02faf0801a002dc6c01a000ab34ed8799f1a0006ddd01975301a000124f81a002dc6c0ffffd8799f581c8fef2d34078659493ce161a6c7fba4b56afefa8535296a5743f695874441414441ff1b000000012a05f2001a00011af11b00000052aeaea7801b0000006673cc7c5b1b0000018de01f5008582046ffb569dc0d84d965733a5ff92bb7b2244a6f72d4726558e49d1d41ad4fbc8fd8799f581c13dfcd07acf9c62ae28f7578e637210dddd7f77b393d0983b89c270758202f426424960c554bf256c1e7f2ee74013271613fd6cffdbe1b2f337600ed774cffd8799f581c13dfcd07acf9c62ae28f7578e637210dddd7f77b393d0983b89c270758202f426424960c554bf256c1e7f2ee74013271613fd6cffdbe1b2f337600ed774cffd87a80ff')
print(d)

$ python3 test.py
{
  'unknown_field0': b'xv\xeb\xacD\x94Z\x88\x85TBi+\x86@\x07v\xe0\xa2\x98|_T\xa1\x9bE}\x86',
  'unknown_field1': CBORTag(121, [b'', b'']),
  'unknown_field10': b'F\xff\xb5i\xdc\r\x84\xd9es:_\xf9+\xb7\xb2$Jor\xd4reX\xe4\x9d\x1dA\xadO\xbc\x8f',
  'unknown_field11': CBORTag(121, [b"\x13\xdf\xcd\x07\xac\xf9\xc6*\xe2\x8fux\xe67!\r\xdd\xd7\xf7{9=\t\x83\xb8\x9c'\x07", b'/Bd$\x96\x0cUK\xf2V\xc1\xe7\xf2\xeet\x012qa?\xd6\xcf\xfd\xbe\x1b/3v\x00\xedwL']),
  'unknown_field12': CBORTag(121, [b"\x13\xdf\xcd\x07\xac\xf9\xc6*\xe2\x8fux\xe67!\r\xdd\xd7\xf7{9=\t\x83\xb8\x9c'\x07", b'/Bd$\x96\x0cUK\xf2V\xc1\xe7\xf2\xeet\x012qa?\xd6\xcf\xfd\xbe\x1b/3v\x00\xedwL']),
  'unknown_field13': CBORTag(122, []),
  'unknown_field2': 5000000000,
  'unknown_field3': CBORTag(121, [2000000, 2200000, 1500000, CBORTag(121, [10000, 0, 15000, 150000, 20000, 450000, 25000, CBORTag(121, [CBORTag(121, [b'\x0c\x8b\x9c\xc1e~Q9\xbez3\x106\xc5I\x9f\x0c-\xc0\x9f\xd8h\x0e\x97s\xe4\xa0\x1a']), CBORTag(121, [CBORTag(121, [CBORTag(121, [b'n\r\xef\xd3\xcf:C\x07e.\x95k<\xa6W\x89\xca[x6\xaeT\x94\xeb\xc5F\xad\x8a'])])])])]), 2000000, 50000000, 50000000, 3000000, 701262, CBORTag(121, [450000, 30000, 75000, 3000000])]),
  'unknown_field4': CBORTag(121, [b'\x8f\xef-4\x07\x86YI<\xe1a\xa6\xc7\xfb\xa4\xb5j\xfe\xfa\x855)jWC\xf6\x95\x87', b'AADA']),
  'unknown_field5': 5000000000,
  'unknown_field6': 72433,
  'unknown_field7': 355118000000,
  'unknown_field8': 440029445211,
  'unknown_field9': 1708862165000,
}

This works because the provided datum just happens to have the same constructor ID as the PlutusData default in 0.8.0 (0). You can also see that PlutusData does not even attempt to deserialize any further than the first layer.
When the constructor id is 1, this parsing also fails.

# test2.py
from pycardano import PlutusData
from dataclasses import dataclass

@dataclass
class MyData(PlutusData):
    CONSTR_ID = 1

cbor_hex = MyData().to_cbor()

empty = PlutusData()
d = empty.from_cbor(cbor_hex)
print(d)

$ python3 test2.py
Traceback (most recent call last):
  File "/home/niels/git/test/test.py", line 11, in <module>
    d = empty.from_cbor(cbor_hex)
  File "/home/niels/git/test/lib/python3.10/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
  File "/home/niels/git/test/lib/python3.10/site-packages/pycardano/serialization.py", line 398, in from_cbor
    return cls.from_primitive(value)
  File "/home/niels/git/test/lib/python3.10/site-packages/pycardano/serialization.py", line 126, in wrapper
    return func(cls, value)
  File "/home/niels/git/test/lib/python3.10/site-packages/pycardano/plutus.py", line 503, in from_primitive
    raise DeserializeException(
pycardano.exception.DeserializeException: Unexpected constructor ID for <class 'pycardano.plutus.PlutusData'>. Expect 121, got 122 instead.

[mmahut]

I highly doubt that this every worked intentionally.

Recently we have seen a pretty bad outage for a customer that used it this way, so the CBOR parsing did fail for them. Is there a way to document / warning this a little bit better?

[nielstron]

Surely there is a way. Maybe you can describe exactly what it is that would have prevented the outage (due to what exactly)? Or ideally present a Pull Request if you know already what the documentation/warning should say.

Update: found both suspected customer and detailed error report

[nielstron]

Based on the error report I am fairly certain that the outage on the client had nothing to do with this issue.

[mmahut]

Based on the error report I am fairly certain that the outage on the client had nothing to do with this issue.

As I maybe misunderstood the comment #320 (comment), the error pycardano.exception.DeserializeException: Unexpected constructor ID for <class 'pycardano.plutus.PlutusData'>. Expect 121, got 122 instead. is not coming from a malformated CBOR, but from using from_cbor(node_utxo.output.datum.cbor) with a different than default constructor ID when not using a class with type structure. I tried this and confirmed the parsing is working fine with RawPlutusData.

Edit: I did review the report linked but it is missing some important details, so just to make it clear: I did not mean to base my comment on what is in the report. My aim was to see if there is a way to document this better, as it appears it is not an isolated usage pattern.

[nielstron]

@mmahut I created a PR that adds some info for deserializing PlutusData. Anything you would like you have added?