serialization.protocol-buffer README

STARTED Introduction

The serialization.protocol-buffer system is a pure Common Lisp implementation of a Protocol Buffer toolchain without any shortcuts. In particular, it does not depend on Google’s protocol buffer compiler, protoc, or other external tools [fn:graphviz: GraphViz can be used for visualization.]. It consists of:

• A parser for the “debug text” format
• A parser for the “proto” format
• A collection of functions for reading and writing the various “wire types” used in the protocol buffer format
• (work in progress) A code generator that emits reasonably efficient (de)serializer functions for protocol buffer messages

STARTED Tutorial

This section briefly explains each of the modules mentioned in the introduction.

STARTED Parsing Notes

All parsing functionality provided by this system is based on the esrap system.

Parse results are constructed using the architecture.builder-protocol system. For simplicity’s sake, the examples in this tutorial use the src_lisp[:exports code]{list} builder provided by that system.

STARTED Parsing the “Debug Text” Format

The debug text format is mostly used for printing and reading protocol buffer message literals. There does not seem to be an official grammar or other formal specification but the format seems simple enough.

Messages are written as sets of key-value pairs (with sometimes optional colons) and lists (for repeated fields) using the widespread square-bracket syntax:

{ foo: 1 bar: [ 2,3,4 ] baz: { whoop: "fez" } }

The debug text format can be parsed using the rules in the src_lisp[:exports code]{:serialization.protocol-buffer.parser.text-format} package. The src_lisp[:exports code]{serialization.protocol-buffer.parser.text-format:message} rule corresponds to a message literal:

(architecture.builder-protocol:with-builder ('list)
  (esrap:parse 'serialization.protocol-buffer.parser.text-format:message
               (string-trim '(#\Newline) input)))

That’s basically it for the debug text format.

STARTED Parsing the “Proto” Format

The “proto” format is used to define Protocol Buffer messages and services. These definitions can be used to generate data structures and corresponding (de)serialization functions for different programming languages. Currently, there are two versions of the “proto” format (and Protocol Buffers as a whole): version 2 and version 3.

A very simple message definition using version 2 syntax looks like this:

syntax = "proto2";

message Foo {
    repeated string bar = 1;
}

The proto format can be parsed using the rules in the src_lisp[:exports code]{:serialization.protocol-buffer.parser.protocol} package. The src_lisp[:exports code]{serialization.protocol-buffer.parser.proto:protocol} rule corresponds to a “.proto” file:

(architecture.builder-protocol:with-builder ('list)
  (let ((serialization.protocol-buffer.parser::*comment-rule*
         'serialization.protocol-buffer.parser.proto:comment))
    (esrap:parse 'serialization.protocol-buffer.parser.proto:proto
                 input)))

STARTED The syntax Directive

Since, the src_protobuf[:exports code]{syntax} directive has to appear at the beginning of the file or not at all, the parser takes it into account when parsing the remainder of the file.

To choose the behavior for files that do not contain a src_protobuf[:exports code]{syntax} directive, the caller can bind the src_lisp[:exports code]{serialization.protocol-buffer.parser.proto:*version*} variable to the desired version:

(let ((serialization.protocol-buffer.parser.proto:*version* 3))
  …)

TODO Generating Code

TODO Dictionary

settings