Vague behavior when single quoted f-string format_spec contains line break

[yf-yang]

Bug report

Bug description:

>>> value = 123
>>> f"{value:.{'2'}f}" # '123.00'
>>> f"{value:.
{'2f'}}" # '123.00'
# but, shouldn't this one below also be valid?
>>> f"{value:.
... {'2'}f}" # invalid
>>> f"{value:
... {'.2f'}}" # '123.00'

I am checking the code of CPython's lexer and am confused why the lexer resets to normal mode when it encounters line break in format spec

cpython/Parser/lexer/lexer.c

Lines 1338 to 1348 in d7672e5

	// If we are in a format spec and we found a newline,
	// it means that the format spec ends here and we should
	// return to the regular mode.
	if (in_format_spec && c == '\n') {
	tok_backup(tok, c);
	TOK_GET_MODE(tok)->kind = TOK_REGULAR_MODE;
	current_tok->in_format_spec = 0;
	p_start = tok->start;
	p_end = tok->cur;
	return MAKE_TOKEN(FSTRING_MIDDLE);
	}

Go back to the original PR: #110271
Seems this case is not covered by the test cases

The case above could be recognized by tokenizer, so maybe pegen should be modified.
However, since it goes back to normal mode, some corner cases cannot be handled by tokenizer:

>>> value = 123
>>> f"{value:#{'x'}}" # '0x7b'
>>> f"{value:
... #{'x'}}" # breaks the tokenizer

CPython versions tested on:

3.13

Operating systems tested on:

macOS

Linked PRs

• gh-129958: Properly disallow newlines in format specs in single-quoted f-strings #130063
• [3.13] gh-129958: Properly disallow newlines in format specs in single-quoted f-strings (GH-130063) #132692

[ZeroIntensity]

cc @pablogsal

[pablogsal]

Check #110259 for context on why new lines are not allowed in format specifiers

[pablogsal]

and am confused why the lexer resets to normal mode when it encounters line break in format spec

Because otherwise the newline will cause the string to terminate because that's not allowed as part of the string part. You would get something like:

>>> f"{value:
  File "<python-input-8>", line 1
    f"{value:
    ^
SyntaxError: unterminated f-string literal (detected at line 1)

[gvanrossum]

Seems to me that newlines should perhaps not be allowed in single-quoted f-strings at all? Allowing this violates my original design for strings.

[pablogsal]

Seems to me that newlines should perhaps not be allowed in single-quoted f-strings at all? Allowing this violates my original design for strings.

We can try to disallow newlines in everything other than expressions for single quoted f-strings. This will include the format spec and the string chunks themselves.

Note that newlines need to be allowed in expressions inside f-strings and that cannot be changed because it would be breaking existing code and is compliant with the PEP. With this I mean something like:

>>> f"These are the letters: {", ".join([
...     'A',
...     'B',   
...     'C'       
... ])}"

This is also documented and in the 3.12 what's new.

[gvanrossum]

That’s too bad, I wish I had paid attention at the time. :-)

We always say PEPs can be amended. Is there a chance for deprecating this behavior?

[pablogsal]

We always say PEPs can be amended. Is there a chance for deprecating this behavior?

I suppose we can try to propose it but given that that will involve breakage on something that was a very visible feature in the grammar I am not sure how that would be received. And in any case we would be looking at a horizon of 2 or 5 years (depending on the new deprecation policies).

It will potentially complicate quite a lot the lexer since we would need to add some different mode that's normal-mode-but-not-quite and this is non trivial complexity given how complex the lexer already is.

[gvanrossum]

Ok, fine, no worries.

[yf-yang]

We can try to disallow newlines in everything other than expressions for single quoted f-strings. This will include the format spec and the string chunks themselves.

Sounds good!

>>> f"These are the letters: {", ".join([
...     'A',
...     'B',   
...     'C'       
... ])}"

I don't exactly understand what aspect of guido's original design is violated but this one looks intuitive. Instead

>>> f"These are the letters: {", ".join([ \
...     'A', \
...     'B', \  
...     'C' \     
... ])}"

is more counter intuitive. These are just some of my personal thoughts.