RowBinary: de- and encode dynamic JSON#296
Conversation
|
👋 @iStefo Thank you for the contribution! I'll be able to take a closer look over the weekend. But it looks good! Please feel free to ignore the CI failures / typos / etc. :) I can handle it after the PR is merged. |
|
I noticed that encoding JSON as a Dynamic type (e.g. during inserts) wasn't supported, so I added that, too. |
|
👋 @iStefo Thank you again! I'd like to attempt rewriting the decoding logic without the addition of |
|
Hi @ruslandoga, of course, feel free to modify this in whatever way you see fit. I would expect pattern matching on the whole JSON type header to fall short because e.g. here we need to read a variable number of varints. But who knows :) |
I have implemented basic support for decoding dynamic JSON in RowBinary.
The implementation essentially skips over all the type information encoded in the type header and then passes the remaining payload to the regular JSON decoding.
The current limitations of this approach are:
typed_pathsencode the type headers of contained types. To strip the correct number of bytes, a more sophisticated approach would be necessary. I currently haveskip_type/1throw if it encounters a multi-byte type header.output_format_binary_write_json_as_string = 1. If JSON is not stringified by ClickHouse, we pass incomprehensible data toJason.decode. Unfortunately, I don't see a way of asserting correct use in code (matching on{as opening brace doesn't work since all types of JSON values can be returned)type_paths, which means that encoded Dates etc. aren't converted to the more ergonomic Elixir types. However, this is probably not a feature people expect from JSON anyways.Please let me know what you think.