Home

lexi

lexi is a regex-based lexer, or tokenizer, for .NET, published to NuGet as MSL.Lexi. You declare a vocabulary of named regex patterns, each mapped to a uint token id, and the lexer scans source text left to right, emitting the next matching token. It is general-purpose; SQL is only one demonstration. It is built to feed simple recursive-descent parsers, and a math expression parser and a SQL-like predicate parser ship as samples.

Unlike most of the projects in this collection, lexi is published and actively maintained: it builds clean, carries passing tests and CI, and a span-first rework of the core is in flight on a spike branch.

Architecture

Architecture walks the tokenization pipeline type by type — how input text becomes a Source, how the Lexer selects the winning pattern, and how the Symbol it emits serves as the token. Start there before reading the core.

Branches

• Branch-main — the default branch: the MSL.Lexi lexer with its VocabularyBuilder API and sample parsers, recently modernized to the analyzer-strict library pattern.
• Branch-spike-span-first-lexer — the span-first rework of the scanning core, slated to land on main as the next breaking change.

A note on state

Both branches build clean under dotnet build (net10.0 with C# 14, no warnings, AOT-compatible and trimmable) with passing test suites, verified 2026-07-16. The published package is MSL.Lexi v2.2.2, which predates main's breaking analyzer-strict modernization. The span-first spike layers a second breaking rework on top, and the commit history slates the combination for the next major release. The only loose end in source is a // todo noting the character-literal pattern does not yet handle escape sequences.