Parser customization should be rare

[KathleenDollard]

With limitations of the middleware and validation, a variety of things happened in custom parsers.

I was really struck by a comment made in the public chat on our last API review. I cannot it right, but the impact on me was a better appreciation of our responsibility to the ecosystem to nudge the world to common CLI syntax. We can't force this. A huge number of CLIs are in use in scripts and many have quirks that need to be supported. It is not helpful for people to have to write their own parser with more quirks just because we cannot accommodate their needs. So, I believe we should remain flexible and custom parsing should be an option.

I have thought a lot about how we balance the responsibility of encourage a consistent stable ecosystem with few weirdnesses in CLIs and the need for CLI authors to be able to get their job done no matter what that job entails (well almost at least). As a high level goal have nothing but custom parsing in custom parsers. So first some questions:

What do you do in a custom parser other than validation and custom parsing?

If you do custom parsing, what do you do it for? Is it a legacy quirk you need to support? List support? If you are doing non-Posix, non-Windows syntax, why are you doing it?

Do you have any thoughts on the balance we should strive for between encouraging a consistent table ecosystem and helping people get done what they need to get done whyever they need to get it done?

[KalleOlaviNiemitalo]

Perhaps not exactly what you asked….

I have an application that takes an XPath expression as the argument of an option. That may reference XML namespaces via prefixes that are defined with options like --xmlns="uri" --xmlns:prefix="uri", as this syntax should look familiar to XPath users.

The implementation using System.CommandLine declares an option named "--xmlns" and parses prefix="uri" as an argument of the option. IIRC, it's an Option<Dictionary<string, string>>, and the custom parser adds an error to the parse result if there is a duplicate key.

Alas, the custom parser cannot distinguish --xmlns:prefix="uri" from --xmlns="prefix=uri" and has to use a heuristic: if the argument of the option starts with an XML NCName and an equals sign, then the NCName is a namespace prefix, else the whole string is the namespace URI. This heuristic will fail if the user actually wants to map the unprefixed names to a namespace URI that starts with those things.

[KathleenDollard]

Thank you.

I really appreciate understanding both what you are doing - it is strictly custom parsing for a special case - and where it comes up short.

I didn't find an issue on the problem distinguishing --xmlns:prefix="uri" from --xmlns="prefix=uri". Is that as simple as providing more information to the custom parser? I am assuming that --xmlns is mapped as an option and you are custom parsing the rest. I do not know the guts of the parser (except to have great respect for it) so do not know how easy/hard it would be to include get the separator token to the custom parser.

I would suspect that would also be interesting if someone had a goal to restrict the separator that was used.

[jonsequitur]

We currently don't support custom separators (or any customization of the tokenizer beyond the symbol names, aliases, and arity) because they represent changes to the core grammar that could lead to more nonstandard CLI UX. We support whitespace, :, and = as separators.

We treat these as equivalent though we need to fix at least one case where we shouldn't (#2072).

[KalleOlaviNiemitalo]

If I run the application with --xmlns="http://www.w3.org/2001/XMLSchema" --xmlns:s="http://www.w3.org/2003/05/soap-envelope", then System.CommandLine calls the custom parser just once and gives it an ArgumentResult whose Tokens property lists two tokens "http://www.w3.org/2001/XMLSchema" and "s=http://www.w3.org/2003/05/soap-envelope". So the ambiguity cannot be fixed by adding a public char OptionSeparator { get; } property to ArgumentResult, as different tokens in the same ArgumentResult can have different separators. If the property were added to Token instead, then that would let me fix the ambiguity.

Other custom parsers that I am using:

• Parsing for CliOption<System.Net.Http.HttpMethod>. This could be replaced with CliOption<string> and a validator.
• Parsing for CliArgument<Uri>, specifically with UriKind.Relative. This could likewise be replaced with CliArgument<string>, a validator, and a separate Uri constructor call afterwards; but if I need to call Uri.TryCreate in order to validate, then I can as well do that in the parser so that the resulting Uri is saved.
• Parsing for CliArgument<FileInfo[]> expanding wildcards.
• Parsing for CliOption<expression-tree> with a recursive-descent parser. This is an application-specific expression-tree type, not related to System.Linq.Expressions.