spec: add grammar (ABNF)

[jkowalleck]

This PR defines the PackageURL grammar using ABNF.

• fixes publish the grammar #535

ABNF is a standardized grammar notation defined in RFC 5234 and widely used across internet protocol specifications (HTTP, SIP, email, URI syntax, etc.). Unlike BNF, which exists in many incompatible variants, ABNF has a single, formal, and stable specification.

Because PackageURL is conceptually similar to URLs, using ABNF aligns the grammar with established internet standards. The IETF URL/URI specification (RFC 3986) also uses ABNF to define its syntax, so adopting the same notation keeps PackageURL consistent with the broader ecosystem of identifier formats.

ABNF is a notation for describing grammars, not a parser‑generator language like ANTLR or yacc. It is tool‑agnostic: various libraries can parse ABNF and generate validators or parsers from it, but the standard itself does not prescribe any specific generator. This makes ABNF a well‑recognized, interoperable choice for defining a grammar intended to be implemented across multiple languages and platforms.

---

Note

the state of the ABNF is not final - thats why it is in "draft" mode.

Especially the non-canonical form might be much too strict.
see #578 (comment) ff

[jkowalleck]

The ABNF is done according to latest spec, ready for review.

possible followup: have a pipeline that checks the test-suite against that ABNF.

[ppkarwasz]

@jkowalleck, thanks for submitting this!

I'll check it in detail later, right now these two problems show up:

[ppkarwasz]

Thanks again for the PR! 🎉

I only spotted a couple of issues, the rest looks solid.

Random thought while reading: what if we defined a common set of productions for the basic character sequences, and then split things into two separate ABNF grammars: one for canonical PURLs and one for non-canonical PURLs?

The spec is very lenient about repeated slashes /: they’re allowed not just at the start, but also smack in the middle of a namespace or subpath. Basically, you could take a nap with your finger on the / key and still be fully PURL-compliant. 😴///😴///😴

If you fall asleep, you'll need a bigger screen to display the PURL, though. 😉

[jkowalleck]

The spec is very lenient about repeated slashes /: they’re allowed not just at the start, but also smack in the middle of a namespace or subpath. Basically, you could take a nap with your finger on the / key and still be fully PURL-compliant. 😴///😴///😴

tha's not what the current spec says.
unlimited slashes are allowed on the start/end:

purl-spec/PURL-SPECIFICATION.rst

Line 149 in c53ba0e

- All leading and trailing slashes '/' are not significant and SHOULD be

but not on the middle:

purl-spec/PURL-SPECIFICATION.rst

Lines 147 to 148 in c53ba0e

	- If present, the ``namespace`` MAY contain one or more segments, separated
	by a single unencoded slash '/' character.

[ppkarwasz]

Hi @jkowalleck,

that's not what the current spec says. unlimited slashes are allowed on the start/end:

purl-spec/PURL-SPECIFICATION.rst

Line 149 in c53ba0e

- All leading and trailing slashes '/' are not significant and SHOULD be

but not on the middle:

purl-spec/PURL-SPECIFICATION.rst

Lines 147 to 148 in c53ba0e

	- If present, the ``namespace`` MAY contain one or more segments, separated
	by a single unencoded slash '/' character.

The spec doesn’t mention slashes in the middle explicitly, but the spec instructs parsers to ignore empty path segments, which are precisely what repeated slashes (///) represent.

purl-spec/PURL-SPECIFICATION.rst

Lines 390 to 398 in c53ba0e

	- The left side is the ``remainder``
	- Strip the right side from leading and trailing '/'
	- Split this on '/'
	- Discard any empty string segment from that split
	- Percent-decode each segment
	- Discard any '.' or '..' segment from that split
	- UTF-8-decode each segment if needed in your programming language
	- Join segments back with a '/'
	- This is the ``subpath``

So in practice, multiple slashes are allowed anywhere (start, middle, or end), since they collapse during parsing.

[jkowalleck]

So in practice, multiple slashes are allowed anywhere (start, middle, or end), since they collapse during parsing.

I understand, but this is not the right scope for your concerns. I've built the grammar based on the spec, and not on the parser-rules.
If you feel there is something off in the spec, then better open a dedicated ticket for this.

[jkowalleck]

@Scanteianu would you give this a review, since you opened package-url/community#24

[Scanteianu]

@Scanteianu would you give this a review, since you opened package-url/community#24

I guess my first question is how one tests this?

My proposal is to use Antlr, as then one can write a small test program in one of their 10 languages of choice (perhaps python?) to actually teat a large sample of strings against the grammar and validate output.

[jkowalleck]

@Scanteianu would you give this a review, since you opened package-url/community#24

I guess my first question is how one tests this?

My proposal is to use Antlr, as then one can write a small test program in one of their 10 languages of choice (perhaps python?) to actually teat a large sample of strings against the grammar and validate output.

Sure, you can translate the ABNF into anything and go from there.
Reading a bit of https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_form should be enough

I was thinking of using something like https://pypi.org/project/abnf/ or https://sabnf.com/, or any tool that already understands ABNF. Mainly because I don’t want to introduce mistakes while translating ABNF into something else.
Writing proper tests is still an open issue. I mean, there are existing test cases that everyone agrees on in the test suite here: https://github.com/package-url/purl-spec/tree/main/tests
And the tests aren’t meant to fix or canonicalize a URL - they’re meant to take existing valid or invalid (canonical) PURLs and check whether they correctly pass the grammar; and if they don;t, we need to triage the case :-)

[Scanteianu]

@Scanteianu would you give this a review, since you opened package-url/community#24

I guess my first question is how one tests this?
My proposal is to use Antlr, as then one can write a small test program in one of their 10 languages of choice (perhaps python?) to actually teat a large sample of strings against the grammar and validate output.

Sure, you can translate the ABNF into anything and go from there. Reading a bit of https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_form should be enough

I was thinking of using something like https://pypi.org/project/abnf/ or https://sabnf.com/, or any tool that already understands ABNF. Mainly because I don’t want to introduce mistakes while translating ABNF into something else. Writing proper tests is still an open issue. I mean, there are existing test cases that everyone agrees on in the test suite here: https://github.com/package-url/purl-spec/tree/main/tests And the tests aren’t meant to fix or canonicalize a URL - they’re meant to take existing valid or invalid (canonical) PURLs and check whether they correctly pass the grammar; and if they don;t, we need to triage the case :-)

ah, that pypi package might be the ticket. instead of working on porting it to antlr maybe I can use that package to generate a parser directly, and throw test cases at it that way. I can maybe fork this repo and make a branch in my fork where I do that on top of this branch.

Antlr may be more flexible in terms of languages it supports, but I now understand your reasoning for wanting to use this format as the source of truth. Antlr can come later if/when needed.

[Scanteianu]

@jkowalleck I have opened jkowalleck#1 on your fork with the minimal implementation of a validator based on the parser. This seems to have found some issues somewhere in the last 2 rules in the grammar (ie: if i don't delete them it doesn't compile). I'd be super grateful if you fixed the grammar so it passes compilation (unfortunately I'm not an expert on abnf syntax or url encoding so i don't know exactly what's wrong). Once that's fixed I'm happy to run through all the other test cases and ensure the validator also passes.

[jkowalleck]

Could we move the discussion about the specific testing in jkowalleck#1 over to that PR’s own comment thread?
That PR is a very niche proof‑of‑concept and currently removes parts of the ABNF grammar just to make it runnable, so it’s not something I expect to merge.

To keep this PR (#578) focused and easier for everyone to follow, it would help if conversations about the POC, its test setup, or its implementation details continued in its own PR discussion instead of here.

Thanks for helping keep the threads organized.

[jkowalleck]

I've wriiten a plan how testing this grammar against existing test suites may work: jkowalleck#2
The tests are not intended to be merged - their pure purpose if to see whether the ABNF in compatible with our existing test suites.

I had AI create test suites in various ecosystems, using different libraries, as I expect some being non-feature complete.
work still in progress ... but this is what is currently going on

• Python
  • abnf — Add ABNF grammar validation tests (Python / abnf library) jkowalleck/fork_purl-spec#3
  • abnfparser (ended up using abnf in practice) — Add ABNF grammar validation tests driven by JSON suites (Python/abnf) jkowalleck/fork_purl-spec#4
• Java
  • softlib-abnf (branch named abnf4j) — Add ABNF grammar validation tests driven by JSON test suites (Java / softlib-abnf) jkowalleck/fork_purl-spec#5
  • apg-java — tests(grammar): ABNF grammar validation tests — Java / apg-java jkowalleck/fork_purl-spec#6
  • ratcoder/abnf-parser (self-contained ABNF→regex approach) — Add ABNF grammar validation tests (Java, self-contained ABNF-to-regex compiler) jkowalleck/fork_purl-spec#7
• JavaScript / Node.js
  • apg-js — Add ABNF grammar validation tests [JavaScript/Node.js, apg-js] jkowalleck/fork_purl-spec#8
• Go
  • go-abnf (github.com/pandatix/go-abnf) — Add ABNF grammar validation tests (Go / github.com/pandatix/go-abnf) jkowalleck/fork_purl-spec#9
• Rust
  • static_regular_grammar — Add Rust ABNF grammar validation tests using static-regular-grammar jkowalleck/fork_purl-spec#10
• C# / .NET
  • abnf.net (Abnf.Net) — ABNF grammar validation tests – C#/.NET / Abnf.Net jkowalleck/fork_purl-spec#12
• C++
  • abnf-cpp — Add ABNF grammar validation tests — C++ / abnf-cpp jkowalleck/fork_purl-spec#13
• to be continued.

[Scanteianu]

I think it makes sense for one of the tests (arguably a python one since the existing scripts in this repo are python) to be merged along with the grammar, especially as it makes it easier for someone new coming along to see that the grammar is tested, and would help people in the future validate any changes to the grammar or validate that a pURL conforms to the grammar

[jkowalleck]

I think it makes sense for one of the tests (arguably a python one since the existing scripts in this repo are python) to be merged along with the grammar, especially as it makes it easier for someone new coming along to see that the grammar is tested, and would help people in the future validate any changes to the grammar or validate that a pURL conforms to the grammar

agree. eventually, we will have test scripts that automatically check the grammar against the test suites and vice-versa.

But the things it've currently in preparation is noting planned for merging. it is done to see if the grammar is correct and adheres existing test suite.

[jkowalleck]

@prabhu crafted a purl parser that works directly from the ABNF .
he had it running against the test suite and came with these reports:

• https://github.com/cdxgen/cdx-purl/blob/main/specification/tests/STRICT_ABNF_MISMATCHES.md
• https://github.com/cdxgen/cdx-purl/blob/main/specification/tests/STRICT_CHECKSUM_MISMATCHES.md

need to review and triage them .
will do so soon, and put my ideas and findings below

---

MY IDEAS AND FINDINGS WILL BE AMENDED HERE 

---

below are soem remarks and notes for my future review and triage i need to prepare.

main points seams to be about qualifier values and subpaths

re: qualifier values

• spec:

  purl-spec/docs/standard/specification.md

  Lines 224 to 225 in c756cac

  	- A **value** may contain any Unicode character and all characters shall
  	be encoded as described in the _Character encoding_ clause.

• does this comply with previously mentioned list of allowed charracters?
  see

  purl-spec/docs/standard/specification.md

  Lines 148 to 158 in c756cac

  	## Rules for each PURL component
  	A PURL string is an ASCII URL string composed of seven components.
  	Except as expressly stated otherwise in this Clause, each component:
  	- may be composed of any of the characters defined in the _Permitted
  	characters_ clause
  	- shall be encoded as defined in the _Character encoding_ clause
  	The "lowercase" rules are defined in the Case folding clause.

  
  re: subpaths
• spec:

  purl-spec/docs/standard/specification.md

  Line 235 in c756cac

  - shall not contain any slash '/' characters

• spec:

  purl-spec/docs/standard/specification.md

  Line 238 in c756cac

  - may contain any Unicode character other than '/' unless the package's **type** definition provides otherwise.

• does this comply with previously mentioned list of allowed charracters?
  see

  purl-spec/docs/standard/specification.md

  Lines 148 to 158 in c756cac

  	## Rules for each PURL component
  	A PURL string is an ASCII URL string composed of seven components.
  	Except as expressly stated otherwise in this Clause, each component:
  	- may be composed of any of the characters defined in the _Permitted
  	characters_ clause
  	- shall be encoded as defined in the _Character encoding_ clause
  	The "lowercase" rules are defined in the Case folding clause.

[jkowalleck]

downstream implementations might use ABNF-based parser generators ...

ooooor convert the ABNF to RegEx (once on build-time) and then use it for validation and parsing on run-time.
Others utilize extended RegEx to do such things - like this SPDX validator here: https://github.com/composer/spdx-licenses/blob/5ecd0cb4177696f9fd48f1605dda81db3dee7889/src/SpdxLicenses.php#L305-L343

ABNF to regex converters:

• https://pypi.org/project/abnf-to-regexp
• https://www.msweet.org/abnf/
• ...

[jkowalleck]

with the latest comments in tickets and pull-requests here and there,
i got the feeling that this grammar needs some modification:

• the purl-canonical shall be renamed to purl-strict -- this is strictly adhering to the ECMA spec
• the current purl shall be renamed to purl-permissive-- this is the thing that is backwards-compatible, and allowed all situations described in the "how-to-parse.md" document

@mjherzog what do you think?

PS: purl-permissive grammar might not belong to the spec, so i guess this shall be moved to an own document.