Suggestion: language-independent conformance suite

[simonw]

Since this project defines a new serialization protocol, I recommend building a language-independent conformance suite for it. Basically a set of data-driven test files that can be used to test implementations that are written in other languages.

https://github.com/html5lib/html5lib-tests is a great example of this. A few people (myself included) have been having a lot of fun over the past few days using that 9,200+ test suite to build new HTML5 parsing libraries in various languages - Python and JavaScript and OCaml and Swift so far.

[simonw]

For fun I had a go at prototyping this in my branch over here: https://github.com/simonw/lite3/tree/conformance-suite

I used Codex CLI and GPT-5.2-codex medium to build the Python library and conformance suite: https://gistpreview.github.io/?995c1d2abfcedbbc25d04c16b0530810

And then a second session to get the C code to run against the new conformance tests: https://gistpreview.github.io/?9c4e0a4cee94c4357dd37f4f2913ae26

Tests look like this: https://github.com/simonw/lite3/blob/conformance-suite/lite3-python/format_suite/002-string-bool.yaml

name: "object string and bool"
json: |
  {"a":"hi","b":true}

binary: |
  # root gen/type
  # little-endian 0x00000206
  # low byte (0x06) = LITE3_TYPE_OBJECT
  # upper 24 bits (0x00000200) = generation = 2
  \x06\x02\x00\x00
  # root hash[0] = 0x0002B606
  \x06\xB6\x02\x00
  # root hash[1] = 0x0002B607
  \x07\xB6\x02\x00
  ...

There are a bunch more in here: https://github.com/simonw/lite3/tree/conformance-suite/lite3-python/format_suite

You can run the C test like this (here's the test suite runner):

git clone https://github.com/simonw/lite3
cd lite3
git checkout conformance-suite
make tests

Which outputs some compilation warnings and then:

========= Running Tests =========
-n [RUN] collisions ... 
PASS
-n [RUN] john_doe ... 
PASS
-n [RUN] format_suite_runner ... 
[PASS] object single int
[PASS] object string and bool
[PASS] array ints
[PASS] nested object and array
[PASS] object scalars
[PASS] mixed array
[PASS] empty containers
[PASS] nested arrays

All format suite tests passed
PASS
============ Results ============
   [+] 3 passed   [-] 0 failed
=================================

I had it build me a Python library too, which you can run the tests for with uv like this:

cd lite3-python
uv run pytest -v

Output:

========================================== test session starts ===========================================
platform darwin -- Python 3.14.0, pytest-9.0.2, pluggy-1.6.0 -- /private/tmp/lite3/lite3-python/.venv/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/lite3/lite3-python
configfile: pyproject.toml
testpaths: tests
collected 1 item                                                                                         

tests/test_format_suite.py::test_format_suite SUBPASSED(case='object single int')                  [100%]
tests/test_format_suite.py::test_format_suite SUBPASSED(case='object string and bool')             [100%]
tests/test_format_suite.py::test_format_suite SUBPASSED(case='array ints')                         [100%]
tests/test_format_suite.py::test_format_suite SUBPASSED(case='nested object and array')            [100%]
tests/test_format_suite.py::test_format_suite SUBPASSED(case='object scalars')                     [100%]
tests/test_format_suite.py::test_format_suite SUBPASSED(case='mixed array')                        [100%]
tests/test_format_suite.py::test_format_suite SUBPASSED(case='empty containers')                   [100%]
tests/test_format_suite.py::test_format_suite SUBPASSED(case='nested arrays')                      [100%]
tests/test_format_suite.py::test_format_suite PASSED                                               [100%]

================================== 1 passed, 8 subtests passed in 0.03s ==================================

Here's a demo of that Python library:

uv run --with-editable . python

Then:

>>> import lite3
>>> lite3.encode({"a": 123, "b": "hello"})
b'\x06\x02\x00\x00\x06\xb6\x02\x00\x07\xb6\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x82\x00\x00\x00`\x00\x00\x00l\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08a\x00\x02{\x00\x00\x00\x00\x00\x00\x00\x08b\x00\x05\x06\x00\x00\x00hello\x00'
>>> lite3.decode(_)
{'a': 123, 'b': 'hello'}

[simonw]

I pretty much vibe-coded that entire thing, so I have no expectation that any of it is code that you would use - but I think it's a neat proof-of-concept and you're welcome to take any or all of it that you find useful - it's MIT licensed.

[simonw]

To further test the conformance suite prototype I started Claude Code (Opus 4.5) in the lite3 directory and prompted this:

Build a pure JavaScript library (no dependencies) for encoding and decoding this binary format. Start by looking at how the lite3-python library works - the JavaScript one should have the same API and probably the same code design too. Build the JS one in lite3-javascript - it should be a single JavaScript module which works in both Node.js and in the browser. There should be a test script that runs with Node.js which runs against the files in the lite3-python/format_suite folder. Write the test script first, run it and watch it fail, then build the JavaScript library and keep running the tests until they pass.

Then later:

Add a README.md with instructions on using the library and running the tests.

That was all it took to get this working JavaScript library: https://github.com/simonw/lite3/tree/conformance-suite/lite3-javascript

Here's the full Claude Code transcript: https://gistpreview.github.io/?44e9b4f097d86eb05969c86d21bc76a5

[simonw]

You can try the JavaScript library in a browser like this:

<script type="module">
import { encode, decode } from 'https://simonw.github.io/lite3/lite3-javascript/lite3.js';
console.log(encode({a: 123,  b: "hello"}));
console.log(decode(encode({a: 123,  b: "hello"})));
</script>

[eliasdejong]

Hi,

These (seemingly functioning) Python and JS ports made using LLMs are impressive. For Python however, the proper way will be to use CPython bindings.
For JavaScript the current library could be compiled to WebAssembly, however the calling boundary between JS <---> webasm can sometimes be (surprisingly) slower than a pure JS port, especially when calling small functions many times. So we'll have to decide what will be the best way to bring the format to JavaScript.

Since this project defines a new serialization protocol, I recommend building a language-independent conformance suite for it. Basically a set of data-driven test files that can be used to test implementations that are written in other languages.

This is a great idea. In general a lot more testing is needed. Having a conformance suite can definitely help with this. I'll be looking at your html5lib for inspiration.

Right now there is work being done to write down an official specification for the format. This standardization is necessary and should generally make language ports much easier to make by having a fixed target.

[simonw]

I'm always disappointed when protocols like this ONLY have a C binding approach in Python, because it can so easily rule out other important Python variants - Pyodide and MicroPython today, Jython and IronPython back when those were more popular and actively maintained.

Obviously a C binding is a great thing to have, but for a protocol to reach as wide an audience as possible I think it's worth having dependency-free pure-X bindings as well.

If you're still working on the official specification how about building out a conformance suite at the same time? I feel like it's one of those things that's a lot less expensive to build in small steps along the way as opposed to taking it on as a giant project later down the line.

[simonw]

(Sorry about that, I posted some comments here that were intended for a thread in an entirely different repository - now deleted.)

[eliasdejong]

I agree Pyodide and MicroPython would also be good to support.

A new special repository has been created for the official TRON format specification: https://github.com/fastserial/tron-spec
This would also a be a good place for a conformance suite.

I will leave this issue open and update it when relevant.

[terales]

I found that JSON suggested by @simonw has a few limitations that an official conformance suite might want to consider when choosing a format:

• validation of appends is tricky as we need to communicate in data something like "start { "a": null } and change to { "a": 1 } -- this must leave a gap as i64 is larger than null type.
• validation of in-place updates lack communication of "Generation ID" convept: { "a": null } and change to { "a": 1 } -- the root object must have gen ID incremented.
• validation of removals of data that leaves gaps

[simonw]

I'm certain the JSON shape I proposed is lacking and an official conformance suite can do a whole lot better.

[lukechampine]

+1 for a conformance suite. I've been working on a Go port and it surfaced up a few interesting edge cases. (Will file issues for these.)