Whitespace is underspecified in punctuation

[eernstg]

Cf. dart-lang/sdk#50776.

The Dart grammar specifies symbol literals in terms of smaller parts, i.e., symbols are not recognized at the lexical level, but rather at the context free level. This implies that it is possible, according to the grammar, to have whitespace in symbol literals, for example # foo . bar.

The specification says that the value of the symbol is based on the source code of the symbol literal without any whitespace (which is again a confirmation that the whitespace is allowed).

A similar situation exists with a number of punctuation symbols which are used in different ways in different contexts (in particular > which can be the end of an actual type argument list, and a part of an operator like >> or >>>=).

However, the tools (the analyzer and the common front end) report an error for many occurrences of whitespace, including #[ ], #[] =, x >>> = 1, etc.

A Dart parser needs to maintain the detailed structure during lexical analysis, because the same sequence of characters can have a different structure, depending on the context: x >>>= 1 makes >>>= a single compound-assignment operator, but List<List<List<int>>>==List<List<int>> makes each of the > an end-of-type-arguments marker, and == an equality operator. So we can't simply let a lexer swallow >>>= as one token.

This implies that the rules that we might have to make # [ ] an error would be specified outside the grammar.

(Aside: As far as I know, the Dart parser used by the analyzer and the common front end does actually step out of the level of regular languages during the construction of tokens, e.g., by changing > > to >> if there are no matching <s in the tokens produced so far, but we can't assume that all Dart parsers will be able to match up parenthesis-like tokens during lexical analysis.)

It would make sense to specify such rules about errors for whitespace, at least in the cases where the tools currently report an error for this kind of whitespace.