Custom sections in the text format

[copy]

The design document on the text format says:

WebAssembly will define a standardized text format that encodes a WebAssembly module with all its contained definitions in a way that is equivalent to the binary format.

However the specification doesn't specify how to encode custom sections: https://webassembly.github.io/spec/text/modules.html#text-module and wasm2wat ignores custom sections.

[binji]

Good point, there should be a way to specify these sections in the text format. It seems like this was probably discussed in the past, but I can't remember where that may have been.

@rossberg, any thoughts on this?

[rossberg]

Well, a couple of issues with expressing custom sections directly:

• The text format does not reflect the section order of the binary format. Hence, it is not clear how it could express where custom sections are supposed to go.

• The same goes for the abstract syntax in the spec, which does not currently represent custom sections, because they are immaterial. So we cannot easily specify how they would interconvert.

• Since the structure of custom sections is custom, they could only be given as raw bytes in the text format. That makes them only mildly useful for anything but low-level tests.

In general, the custom section format is more like a detail of the binary format. The assumption was that relevant custom sections are not written verbatim but rather synthesised from the text format, like the name section or the binding section.

What we should think about, then, is a generic syntax for annotations that can be put anywhere in the syntax tree. That would already be needed by the binding proposal. My suggestion would be to use nodes of the form (@id ...) that would be allowed anywhere in an S-expr and are uninterpreted by the core spec. The id would roughly correspond to a certain custom section, so that it is generic and extensible in a similar manner. A given tool may choose to interpret certain annotations and turn them into custom sections according to some separate spec. With global annotations in the module body such a spec could even enable spelling custom sections almost verbatim (up to their position in the binary).

WDYT?

[lukewagner]

Since the core spec does have a defined notion of a custom section, I think it makes sense to give a fully-specified representation in the text format. While it's true that we'd have a hard time expressing the precise placement of the custom section, I expect it's fine to just say that (custom ...) sections are just appended to the end of the module in the order they are encountered.

(Honestly, I wonder about the utility of allowing custom sections anywhere but at the end; I bet we could remove that "feature" and nothing would break.)

[binji]

The text format does not reflect the section order of the binary format. Hence, it is not clear how it could express where custom sections are supposed to go.

All known sections have to be ordered, so you could just use a number to specify which known section it comes after. Something like (custom 0 ...) would come before the type section (1). (custom 3 ...) would go after the function section (3) and before the table section (4).

Since the structure of custom sections is custom, they could only be given as raw bytes in the text format. That makes them only mildly useful for anything but low-level tests.

I agree if we assume the purpose of the text format is just to generate tests for the spec. But we're already using the text format as a way to express the contents of the binary, and AFAICT it doesn't lose much information currently. The only thing I can think of right now is the length of varint values and custom section data. Are there others?

Also, we could make it slightly nicer than raw bytes by having a structured data format. The name section and the reloc section follow the same basic structure of other sections, using varints, strings and vectors. If we provided those primitives we could make it pretty easy to generate. They wouldn't roundtrip very nicely of course. Something like this, maybe:

(custom 12 "foo"
  (string "hello")
  (vector
    (group (varuint32 1) (f32 3.4))
    (group (varuint32 2) (bytes "12345"))
  )
)

RE: annotations

Agreed, annotations would be useful. I believe @yurydelendik was suggesting something like this before, maybe he has some thoughts about it. And you're right, I think we could handle custom sections in a structured way doing this. But I'd like to see a way to handle a custom section that has unstructured data, or one that is unknown to the parser too.

[rossberg]

@binji

All known sections have to be ordered, so you could just use a number to specify which known section it comes after. Something like (custom 0 ...) would come before the type section (1). (custom 3 ...) would go after the function section (3) and before the table section (4).

That would be rather brittle and expose low-level details of the binary encoding. In particular, we have assumed that we may insert new sections anywhere in future extensions of the binary format, so a numeric scheme is not future-proof. If we could adopt @lukewagner's suggestion of eliminating free placement of custom sections then I'd feel more comfortable, but I'm not sure how realistic that is.

The only thing I can think of right now is the length of varint values and custom section data. Are there others?

No, none that I'm aware of.

Also, we could make it slightly nicer than raw bytes by having a structured data format. The name section and the reloc section follow the same basic structure of other sections, using varints, strings and vectors. If we provided those primitives we could make it pretty easy to generate. They wouldn't roundtrip very nicely of course. Something like this, maybe: (custom 12 "foo" (string "hello") (vector (group (varuint32 1) (f32 3.4)) (group (varuint32 2) (bytes "12345")) ) )

That would be cute. But I immediately worry about this becoming an open-ended DSL without ever eliminating bias towards known custom sections.

Agreed, annotations would be useful. I believe @yurydelendik <https://github.com/yurydelendik> was suggesting something like this before, maybe he has some thoughts about it. And you're right, I think we could handle custom sections in a structured way doing this. But I'd like to see a way to handle a custom section that has unstructured data, or one that is unknown to the parser too.

Sure thing, we can simply support `(@Custom "name" "contents")` etc as a generic fallback. AFAICS, that could subsume the suggestion above.

[binji]

That would be rather brittle and expose low-level details of the binary
encoding. In particular, we have assumed that we may insert new sections
anywhere in future extensions of the binary format, so a numeric scheme is
not future-proof.

Right, I forgot that new known sections may not be ordered. I think it will still work, though. If we assume that all known sections can occur only 0 or 1 times, as is currently true, then it doesn't seem like this is a problem. The number can just mean which section the custom section is before in the given module. If the section doesn't occur in the module, we could say that the text for that section is invalid. If we decide later that a known section can occur more than once, we can extend the text format at the same time to indicate which section we mean. And if using a number is gross/ugly, we can always use the names given in the spec:

(@custom "foo" (after import) "...")
(@custom "bar" (before data) "...")

If we did this, we'd probably also want to require that you can't specify sections out of order. Not so sure about the before/after thing either, but it's easy to understand and allows all placements.

If we could adopt @lukewagner's suggestion of eliminating free placement of
custom sections then I'd feel more comfortable, but I'm not sure how
realistic that is.

It probably isn't used much, but I would prefer not to break compatibility over it.

Sure thing, we can simply support (@custom "name" "contents") etc as a
generic fallback. AFAICS, that could subsume the suggestion above.

Right, this covers everything, it just is inconvenient.

[lukewagner]

It probably isn't used much, but I would prefer not to break compatibility over it.

In addition to text-format motivations, there's also the fact that if it's an infrequently used feature, it will be undertested and likely to have problems in practice. I know we've had specific bugs about custom sections in weird places.

Maybe worth putting discussion/poll on CG agenda?

[eholk]

At the most recent CG meeting, we had some opposition to the idea of requiring custom sections to be at the end. The reason is that some uses cases for custom section involve informing later stages of the compilation pipeline. For example, tools might want to provide extra hints (which functions should be compiled first, which locals should get registers, etc.) that VMs could optionally consume. In this case, we'd want to read the hints before we start streaming compilation of the code.

[binji]

First pass proposal overview for custom sections in text format: https://gist.github.com/binji/d1cfff7faaebb2aa4f8b1c995234e5a0

[binji]

I've updated the gist after some feedback. Sorry I didn't notice this earlier, it seems that gist comments don't show up in my notifications (or I missed them).

[Pauan]

@binji GitHub doesn't send notifications for Gist comments, it's very annoying.