Incorrect parsing behaviour with assignment expressions

[JarryShaw]

Recently, we encountered several issues with parso when implementing a backport compiler for assignment expressions.

1. Parsing f-strings with := formatter produces incorrect SyntaxError
2. Parsing invalid use cases of assignment expressions do not raise SyntaxError

---

NB: all codes are running under Python 3.8.0; actual behaviours are running through parso.parse(code, error_recovery=False, version='3.8').

Case 1

Source code:

f'{x:=5}'

Expected behaviour: valid, passes =5 to formatter

Actual behaviour:

Traceback (most recent call last):
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/parser.py", line 181, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(:=)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/__init__.py", line 58, in parse
    return grammar.parse(code, **kwargs)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/grammar.py", line 78, in parse
    return self._parse(code=code, **kwargs)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/grammar.py", line 147, in _parse
    root_node = p.parse(tokens=tokens)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/python/parser.py", line 82, in parse
    return super(Parser, self).parse(tokens)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/parser.py", line 128, in parse
    self._add_token(token)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/parser.py", line 187, in _add_token
    self.error_recovery(token)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/python/parser.py", line 151, in error_recovery
    return super(Parser, self).error_recovery(token)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/parser.py", line 151, in error_recovery
    raise ParserSyntaxError('SyntaxError: invalid syntax', error_leaf)
parso.parser.ParserSyntaxError: ('SyntaxError: invalid syntax', <ErrorLeaf: TokenType(OP):':=', (1, 4)>)

Case 2

Source code:

(lambda: x := 1)

Expected behaviour:

SyntaxError: cannot use named assignment with lambda

Actual behaviour: parsed as valid code

Case 3

Source code:

(a[i] := x)

Expected behaviour:

SyntaxError: cannot use named assignment with subscript

Actual behaviour: parsed as valid code

Case 4

Source code:

(a.b := c)

Expected behaviour:

SyntaxError: cannot use named assignment with attribute

Actual behaviour: parsed as valid code

Case 5

Source code:

[i := 0 for i, j in range(5)]
[[(i := i) for j in range(5)] for i in range(5)]
[i for i, j in range(5) if True or (i := 1)]
[False and (i := 0) for i, j in range(5)]

Expected behaviour:

SyntaxError: assignment expression cannot rebind comprehension iteration variable 'i'

Actual behaviour: parsed as valid code

Case 6

Source code:

[i+1 for i in (i := range(5))]
[i+1 for i in (j := range(5))]
[i+1 for i in (lambda: (j := range(5)))()]

Expected behaviour:

SyntaxError: assignment expression cannot be used in a comprehension iterable expression

Actual behaviour: parsed as valid code

Case 7

Source code:

class Example:
    [(j := i) for i in range(5)]

Expected behaviour:

SyntaxError: assignment expression within a comprehension cannot be used in a class body

Actual behaviour: parsed as valid code

[davidhalter]

Thanks for listing all the cases!

While this list will definitely help for another cause, most of them are not bugs in the way you think they are. The parser does not list all syntax errors. If you want to list all syntax errors, please use:

grammar = load_grammar()
for error in grammar.iter_errors(module):
    print(error)

The first case is definitely a bug that might be harder to fix. The other ones, might be quite easy. If you want to try yourself, you're very welcome :)

If you want to fix this stuff, it's in parso/python/errors.py. You basically need to add additional SyntaxRule classes like this:

@ErrorFinder.register_rule(type='namedexpr_test')
class _NamedExprRule(SyntaxRule):            
    def is_issue(self, named_expr):

It helps to look at the Python grammar to do these.

[JarryShaw]

Oki 😊 I've made the pull request at #90

[davidhalter]

Should be fixed by #90.