Expressions with format specifiers in f-strings have incorrect offsets

[alexcres]

def test(something):
    # "somethi" and "ng:b" are in different color.
    # it should be that "something" and ":b" are different color.
    return f'{something:b}'

[numirias]

Ha! Thanks for catching this. I'll look into it. (Unless someone else wants to supply a patch...)

[numirias]

I'm wondering if that's actually a bug in Python. Compare these two syntax trees:

>>> ast.parse("f'{a}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=3, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=None,
                    ),
                ],
            ),
        ),
    ],
)

>>> ast.parse("f'{a:b}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(                        col_offset=0,
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=1, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=JoinedStr(
                            lineno=1,
                            col_offset=0,
                            values=[Str(lineno=1, col_offset=0, s='b')],
                        ),
                    ),
                ],
            ),
        ),
    ],
)

The relevant part here is that the name a in the first f-string is positioned at

Name(lineno=1, col_offset=3, id='a', ctx=Load())

but in the second one at

Name(lineno=1, col_offset=1, id='a', ctx=Load())

although we're just adding a format specifier, not changing the position of the name in the source. To me it seems like the the column offset should remain at 3 chars. (Because f'{ are three characters.)

Can someone have a look at this as well? I'm tempted to file an issue in the Python tracker, but I may be overlooking something.

[numirias]

I filed a bug: https://bugs.python.org/issue35212

[alexcres]

Sorry I just went though python tutorial, can't help at all.

[sluongng]

hmm I am getting this as well :D... seems like the bug is still not fixed from Python side :(...

[numirias]

I'm currently experimenting with a parser based on tree-sitter instead of relying on Python's AST module. Once implemented, that would also eliminate bugs like this one.

[sluongng]

I'm currently experimenting with a parser based on tree-sitter instead of relying on Python's AST module. Once implemented, that would also eliminate bugs like this one.

I suggest putting this behind a feature flag and have a default value to be which ever one is more stable. Sure tree-sitter may work but I think Python AST may stick closer with the language to help with newer syntax(s) in the future