Removed Features: Binary Literals

[nitz]

One of my favorite part about SDLang is their easy drop-in of base64 encoded binary data. Obviously that's not always super useful from a "configuration" usage point of view, but from a "serialization format" viewpoint, there will absolutely come a time when I'd want the ability to store binary data.

The majority of the things removed from SDLang make perfect sense, there are other ways to express the same things and/or reduce unneeded complexity. The same point could be made for binary data (could use multiple hex literals or the sort), or that could be "left as an exercise for the user" to implement the roundtripping of base64 encoded data.

I may be alone in this, but the built-in use of binary literals is something that I found as one of the attractive bits in SDLang to start with, and think it would benefit being included especially if kdl is intended to be used as a serialization format.

Keep up the great work!

[zkat]

What does the binary format get you that you can't just get with a string with base64 in it?

[nitz]

A few things that seem trivial but have been gotchas for me in the past:

Explicit typing (or type hinting, is maybe the better phrase): Having base64 between quotes and angle brackets definitely looks more or less the same to a parser, and mostly to a human. But if you think about a zero-length field, suddenly the differentiation makes a big difference —

my-b64-a ""
my-b64-b []

The brackets would say to someone hand-modifying (or otherwise parsing) the kdl would immediately know that my-b64-b is intended to be binary data currently of no length, disambiguating it from myb64-a, which is what I'd (at least, coming from a C/C++ background) assume would be a zero-length, null terminated string.

"OOTB": I'm sure many of the SDLang features removed are undoubtedly to reduce code size as well, but knowing that the language specifies (and therefore the parsers would be expected to implement) the conversion to and from base64 out of the box is one less thing that I've got to bring my own solution for. Sure going to-and-from base64 is pretty trivial, but being able to get/put binary data directly from the parser simplifies the end users' workflow, in my opinion.

Definite portability: This one is incredibly minor compared to the other points I bring up for sure, but portability between applications using binary data would be far more guaranteed. Given a hypothetical situation of your application sending kml data to a server that you aren't controlling. If they were using a different variant of base64, you might find a situation where sometimes you'd produce output the server could handle, and maybe it couldn't. Some edge case like they leverage the MIME base64 functionality built into their server suite, but your application doesn't break lines at 76 characters.

---

Self admittedly: most of these issues are something that could probably considered trivial, and I'm sure I'm not seeing the benefits and ease in parsing that comes with dropping binary literals, so perhaps that outweighs!

[lilyball]

My opinion here is pretty weak, but I’m in favor of supporting binary literals using angle brackets. I kind of like the idea of being able to embed hex binary, for the sake of humans trying to read it, but base64 is probably better from a space perspective.

There’s also Ascii85 which sounds interesting, and I think it would be neat to support e.g. <01020304 05060708> as hex and <~9jqo^BlbD~> as Ascii85, but it’s esoteric enough that this may not be a good idea (it’s apparently used in PostScript and PDF, but probably not well-known). As for Base64, I agree with @nitz that having the spec define it would help avoid issues with different variants of base64, or with padding vs no-padding (I vote no padding on encoding as it doesn’t appear necessary if you’re not concatenating base64 blobs together).

The question really comes down to how much of a burden do we think it will be on parsers to have to deal with binary blobs?

Also FWIW if we add this we’ll need to define < as part of the “cannot be used to start an identifier” set. We should probably reserve it immediately anyway, to allow us the ability to add binary literals down the road if not done now without invalidating existing documents. Of course, reserving it means I can no longer write silly things like

<div>
  <p style="color: red;" >="This would parse with an HTML parser"
  </p>
</div>

But that’s probably for the best anyhow.

[zkat]

The thing about specifying base64 allowing normalization across languages makes me think that, in practice, we would either put the community in a position where different languages interpret KDL differently based on particular quirks of their (existing) popular base64 implementations, or we would end up in a position where implementations HAVE to write their own base64 parsing because of (unintentional?) quirks in our own specification. I'd much rather leave this uncertainty in the hands of individuals, which is why I prefer letting people use strings for this.

Note that this is not me saying that this conversation is over or anything. I'm mostly just expressing my own (current) thoughts on this.

I wonder: What other document/configuration/etc formats have binary literals available? How do those work? SDLang is probably the first time I'd ever run into that in the wild.

[lilyball]

Apple Property Lists support base64-encoded data. They also support dates, and I’m mildly sad that KDL doesn’t support them directly.

As for language support, I wager that languages that have popular base64 libraries are probably implementing RFC 4684 §4, and this is the most reasonable variant for us to choose. If we add this, we’d want to say that implementations SHOULD omit padding on encoding and MUST support padding on decoding.

[lilyball]

Plists use base64 encoding in the XML property list format. The obsolete “Old-Style ASCII Property Lists” also supported binary data but using hex literals (e.g. <0fbd777 1c2735ae>).

[nitz]

A few things come to mind:

• Not specifically a literal, but XML often uses CDATA to pack base64 encoded binary. (Technically iirc, I think CDATA can hold anything except it's ending delimiter, but is often used for base64 encoded data.)Sure, you could put it in any attribute/element value, but XML suffers the same problems I was touching on earlier that a binary literal could help to mitigate.
• Not technically a configuration/serialization format, but often used as a backup/intermediate stage, MySQL uses hex literals, which has a number of times been helpful for me to be able to adjust data coming out of or going into a database/file.
• Python, PHP, whatever other scripting/runtime processed language very often uses its own language features in config files. It's definitely more rare to see full binary blobs in them, but I've used them on occasion. Obviously support and styling depends on the language, but it's at least plausible. I personally have used hex literals and PHP's pack function to round trip data. (... don't ask.)

SDL is 100% the first intended as config/serialization language I've seen that treated binary data as a first class citizen, though, which is something that I personally flagged as a "wow I really like this" about it. Far too many times I've wanted to tack a bit of binary in a JSON object. It almost always ends up with trying to just use hex literals, but going back to base64 (or base32, or ascii85, or whatever other roundtrip method I happen to have on hand on both my and the remote side) anyways. I've also been on the receiving end of JSON messages that included binary data that had been poorly escaped (hex sequences with no escape characters, so I had to manually parse it out,) or outright broke the JSON spec to do it (seriously had one instance where the binary data was base64 all fine... but the value wasn't quoted. What parser in the world wouldn't blow up on you there?)

Having the binary literal be part of the spec is an explicit statement that says "this is a way to roundtrip binary data that is guaranteed to work no matter where the data is handled." Which is why SDL's use of it caught my eye in the first place: no matter what language/tech stack I was working in on any side of a project, I knew that the library I had to read/write SDL would be able to read/write arbitrary binary data without me needing to sweat the implementation. Of course, being familiar with SDL itself meant that I'd be able to persist the SDL and modify the binary by hand if I needed to later as well.

I'd agree with @lilyball completely in that specifying one of the RFCs for base64 should be "enough" to assure implementations are at least expected to adhere to it. RFC 4648 §4 is also what I'd pick, as it's not only one of the least restrictive but also one of the most commonly used already.

I will again yield to you that this is certainly a feature that doesn't seem to add very much for the average use case, but it's something that I've personally wanted in a language like this for a long time.

[nitz]

Also, @lilyball, I was similarly minded about dates at first, but the more I thought about it the more I couldn't come up with a way to handle dates that was as simple as it was flexible. I feel like it's reasons like that I almost always store dates/times as unix timestamps or windows filetimes (depending on my use case and desired resolution), which both keep just nicely as plain uint64. Sure they're more of a pain to tune by hand, but if I needed a distinctly human-editable date-time format, I'd almost guaranteed reach for a string with a custom serialize/deserialize to fit my use case.

[lilyball]

What’s wrong with ISO8601 dates (specifically the profile in RFC3339)? Preferably supporting both date-only and date-time (with optional microseconds).

[nitz]

While writing that last bit of jibberjabber, I did come up with another use case example for something that I've done in the past, where a binary literal would have been something I would have employed.

I had a configuration file in JSON that would round-trip to the application. App might make changes to it, and would write it's config out on exit, but the user could modify the config while it wasn't running. I had a case I wanted to support where I would load in a small image (think website favicon style). The config had always allowed specification of a file path in the setting, but we came upon a situation where we had small chunks of binary PNGs that we wanted to use "one time". We decided on using a temporary directory and used paths to that, but it would have been a "neat" feature to be able to have the value for that key be optionally typed to a string or binary. The string would act as a file path as usual, but binary would be able to skip the extra file read and go right into the slot it was intended for.

There's many ways to cook an egg, and I just think this is a novel one, I suppose!

[zkat]

I'm still not entirely into this feature, but there's #35 now so we can always add it later after we've had some more time to discuss it. Thanks for all your feedback! I know I'm not quite on board with this yet, but I'm definitely thinking about it and it might be harmless enough to add.

[nitz]

Looks like #35 is a great measure to be able to safely table the idea! Looking at your stated goals for kdl, this feature definitely would fall under "Flexibility", but certainly tug in the wrong direction for "Ease of implementation".

I was giving it some more thought this afternoon too, and spent a little bit more time comparing it [kdl] to MySQL data types. Now obviously, as a database solution rather than serialization/config, MySQL cares a lot more about the intricacies of data types due to how they can (and can't) be leveraged in queries.

My mental link in similarity to kdl (or sdl) between the BLOB and TEXT types are sorta the same as I was pointing to before. They're effectively the same at rest. The difference comes in the functionality that retrieves and stores them, as well as the types being nods to what the data actually is.

In conclusion, I think I have to agree with you, especially if minimalist simplicity is a larger goal as kdl gets rolling. I think I personally like the idea of separate data types because of the mental organization and aesthetics of [BINDATA] or <BINDATA> vs "BINDATA", but that may very well come down to personal preference, too 🙂!

This has been a fun discussion, and I've enjoyed thinking about it!

[zkat]

I've enjoyed this too! I think [BINDATA] is definitely still on the table for me and I've really appreciated all your thoughtful feedback! Maybe we'll add it later, after I've at least specified the current base types!

[zkat]

Please see my response over here: #38 (comment).

I think this is what I'm gonna end up going with for 1.0.