Raise non-silent warning for invalid escape sequences

[Vgr255]

BPO	32912
Nosy	@rhettinger, @terryjreedy, @gpshead, @mdickinson, @ncoghlan, @bitdancer, @njsmith, @ambv, @serhiy-storchaka, @zooba, @Vgr255, @CuriousLearner, @miss-islington, @asmeurer, @tirkarthi
PRs	bpo-32912: Upgrade warning for invalid escape sequences from silent to non-silent #5849 bpo-32912: Replace a DeprecationWarning with a SyntaxWarning #9652 [3.8] bpo-32912: Revert warnings for invalid escape sequences. #15142 bpo-32912: Revert SyntaxWarning on invalid escape sequences. #15195

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-02-22.18:34:57.749>
labels = ['interpreter-core', 'type-bug', '3.9']
title = 'Raise non-silent warning for invalid escape sequences'
updated_at = <Date 2022-01-13.16:38:26.080>
user = 'https://github.com/Vgr255'

bugs.python.org fields:

activity = <Date 2022-01-13.16:38:26.080>
actor = 'dansebcar'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2018-02-22.18:34:57.749>
creator = 'abarry'
dependencies = []
files = []
hgrepos = []
issue_num = 32912
keywords = ['patch']
message_count = 34.0
messages = ['312575', '312610', '312760', '326766', '328044', '344312', '344764', '345557', '345558', '345639', '345675', '345703', '346871', '348205', '348269', '348270', '348281', '348347', '348348', '348349', '348354', '348357', '348375', '349027', '349103', '349106', '349318', '349320', '349321', '349322', '349323', '349328', '349335', '410496']
nosy_count = 17.0
nosy_names = ['rhettinger', 'terry.reedy', 'gregory.p.smith', 'mark.dickinson', 'ncoghlan', 'r.david.murray', 'njs', 'lukasz.langa', 'serhiy.storchaka', 'steve.dower', 'abarry', 'CuriousLearner', 'miss-islington', 'asmeurer', 'xtreak', 'Eric Wieser', 'dansebcar']
pr_nums = ['5849', '9652', '15142', '15195']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32912'
versions = ['Python 3.9']

[Vgr255]

This is a follow-up to bpo-27364.

Back in Python 3.6, a silent warning was added for all invalid escape sequences in str and bytes. It was suggested that it would remain a silent warning (which still impacts tests, while not visually annoying the average user) for two releases (3.6 and 3.7), then would be upgraded to a non-silent warning for two subsequent releases (3.8 and 3.9) before becoming a full-on syntax error. With the 3.7 feature freeze on and going, I think it's time to evaluate the approach we take for 3.8 :)

I suggest upgrading the DeprecationWarning to a SyntaxWarning, which is visible by default, for 3.8 and 3.9. I have cross-linked bpo-27364 to this issue as well.

-Em

[ncoghlan]

+1 for the DeprecationWarning->SyntaxWarning->SyntaxError approach from me (especially as 3.7 will make the existing deprecation warning visible in interactive shells and __main__ modules by default).

[serhiy-storchaka]

PR 5849 changes not only the Python parser, but codecs. It shouldn't. The Python parser and codecs will go different ways. The warning in the Python parser will finally be upgraded to SyntaxError (it is already replaced with SyntaxError if the warning is raised as error). The warning in codecs will become UnicodeDecodeError or ValueError. This is why the code for emitting the warning is not shared between the parser and codecs at first place.

[serhiy-storchaka]

Since the author of PR 5849 had not the time to work on it, I have created PR 9652 which properly replaces a DeprecationWarning with a SyntaxWarning. It also updates the documentation.

[serhiy-storchaka]

New changeset 6543912 by Serhiy Storchaka in branch 'master':
bpo-32912: Replace a DeprecationWarning with a SyntaxWarning (GH-9652)
6543912

[rhettinger]

I'm seeing these warnings pop up from time to time with various third party packages (such as pyparsing which is invoked by ensurepip).

Am wondering whether this should be deferred for another version or so until be have good confidence that the major third party packages have all had a chance to adapt. Otherwise, this will mostly be noise for end-users. Also, it gets in the way of the end-user strategy of "backslash anything that looks special" as preferred over "remember exactly which things are special". This may be a case of "the cure is worse than the disease".

[asmeurer]

I agree with Raymond that third party libraries are not ready for this.

My biggest issue is that the way Python warns about this makes it very difficult for library authors to fix this. Most won't even notice. The problem is the warnings are only shown once, when the file is compiled. So if you run the code in any way that causes Python to compile it, a further run of 'python -Wd' will show nothing. I don't know if it's reasonable, but it would be nice if Python recompiled when given -Wd, or somehow saved the warning so it could be shown if that flag is given later.

As an anecdote, for SymPy's CI, we went through five (if I am counting correctly) iterations of trying to test this. Each of the first four were subtly incorrect, until we finally managed to find the correct one (for reference, 'python -We:invalid -m compileall -f -q module/'). So most library authors who will attempt to add tests against this will get it wrong. Simply adding -Wd as you would expect is wrong. If the code is already compiled (which it probably is, e.g., if you ran setup.py), it won't show the warnings. At the very least the "correct" way to test this should be documented.

Things would probably be improved if the warnings were always shown, as at least then devs will see the error once (although most will probably be confused when the warning doesn't repeat).

Another problem is the information in the warnings. It seems the line number of the string is now shown, which is an improvement (https://bugs.python.org/issue28128). It would be nice if it showed the actual line and column number in the file of the invalid escape. This is especially annoying when an escape appears in a docstring. It just shows """ as the offending line.

We have a lot of LaTeX in our docstrings in SymPy, so we had quite a few of these to fix. SymPy doesn't have invalid escapes anymore because I was proactive about it, but from what I've seen, most library authors haven't been.

By the way, this looks like a bug (python 3.8b1):

$ cat test.py

"""
a
\p
"""
$ python -We:invalid test.py
  File "test.py", line 2
    
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ^
SyntaxError: invalid escape sequence \p
$

It might not be possible to see in what I pasted, but it filled my terminal with spaces.

[serhiy-storchaka]

In the particular case of pyparsing there was a bug in the docstring.

     def sub(self, repl): 
         """ 
         Return Regex with an attached parse action to transform the parsed 
         result as if called using `re.sub(expr, repl, string) <https://docs.python.org/3/library/re.html#re.sub>`_. 
  
         Example:: 
  
             make_html = Regex(r"(\w+):(.*?):").sub(r"<\1>\2</\1>") 
             print(make_html.transformString("h1:main title:")) 
             # prints "<h1>main title</h1>" 
        """

\1 and \2 were interpreted as control characters U+0001 and U+0002, so the user could see

        make_html = Regex(r"(\w+):(.*?):").sub(r"<^A>^B</^A>")

or

        make_html = Regex(r"(\w+):(.*?):").sub(r"<☺>☻</☺>")

or

        make_html = Regex(r"(\w+):(.*?):").sub(r"<></>")

depending on the output device.

So this examples proves the usefulness of the new warning.

[serhiy-storchaka]

Aaron, you have reported several different issues in your message:

• compiler warnings are emitted only once, therefore it is easy to miss them and hard to reproduce.

• warnings for multiline expressions (and multiline string literals in particularly) could be better.

• strange output of spaces in some cases.

And maybe more.

I think all of them deserve opening separate issues.

[asmeurer]

I agree. Please someone else do that. I don't know what already has issues and I unfortunately don't have time right now to help out with any of this.

I simply mentioned all these things as arguments why Python should not (yet) make these warnings errors, which is the point of this issue.

Also, for the pyparsing example, you would have gotten lucky because it also contained \w, which is not a valid escape. If it didn't, you wouldn't be warned. So clearly this will help things, but it will also be good to have linting tools that, for example, warn about any escape sequences inside docstrings.