Repository navigation
Request: normalization and depythonize #80
Description
Activity
Thanks for the report & sorry for the slow reply.
I suspect that #69 was the cause here; I see now that it was a behavioural change where positive values previously in
i64range will now be deserialized asu64values.I think it will be hard to guarantee round-tripping of
i64andu64values, because as soon as the data goes from Rust to Python, for positive integers we don't know whether they came from a signed or unsigned value.What we might be able to do is add a config option as to whether you prefer signed or unsigned types, and make
depythonizeselect according to your preference. Does that potentially help?Reacted by Caleb HattinghHi @davidhewitt thank you for the detailed response, #69 was indeed the
pythonizechange, sharing that really helped us.What we might be able to do is add a config option as to whether you prefer signed or unsigned types, and make
depythonizeselect according to your preference.What the
tantivy-pyteam is doing is recursively forcing I64/F64 after deserialization, effectively eliminating the U64 vs I64 ambiguity.I think an option to prefer or force signed/unsigned is useful, it would enable
tantivy-pyto remove the new forcing logic. Up to you though, also feel free to close this out as resolved.Thanks again for the help here btw, much appreciated.
When upgrading
pythonizefrom version0.21.0to version0.23.0intantivy-py's quickwit-oss/tantivy-py#401, we switched fromextracttoextract_bound(per PyO3/pyo3#3916).Now, we are hitting an issue where
I64values are becomingU64afterdepythonize.I came up with a solution in jamesbraza/tantivy-py#1 that involves re-casting U64 back to I64 if it fits, but this is pretty hacky.
Is there some way to preserve integer types across serialization/deserialization?
I64values remainingI64U64values remainingU64I am wondering if there is some way
pythonizecan accommodate this, maybe by adding an opt-in flag somewhere?Also, I am not 100% sure if this is a
pythonizevspyo3issue, but it seems to be a deserialization issue, so I opened this here.