3.12+: tokenize of mixed tabs and spaces now produces an error

[asottile]

I read the parts in here and it does mention that some ERRORTOKENs will be converted to exceptions but it doesn't seem to cover this case.

this breaks the E101 check in pycodestyle (or I guess more specifically turns it into a syntax error)

this is probably fine? but should be called out in the documentation ?

Bug report

if True:
    print(1)  # indented with spaces
	print(2)  # indented with tab

$ python3.11 -m tokenize t.py 
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,2:            NAME           'if'           
1,3-1,7:            NAME           'True'         
1,7-1,8:            OP             ':'            
1,8-1,9:            NEWLINE        '\n'           
2,0-2,4:            INDENT         '    '         
2,4-2,9:            NAME           'print'        
2,9-2,10:           OP             '('            
2,10-2,11:          NUMBER         '1'            
2,11-2,12:          OP             ')'            
2,12-2,13:          NEWLINE        '\n'           
3,0-3,1:            INDENT         '\t'           
3,1-3,6:            NAME           'print'        
3,6-3,7:            OP             '('            
3,7-3,8:            NUMBER         '2'            
3,8-3,9:            OP             ')'            
3,9-3,10:           NEWLINE        '\n'           
4,0-4,0:            DEDENT         ''             
4,0-4,0:            DEDENT         ''             
4,0-4,0:            ENDMARKER      ''             
$ python3.12 -m tokenize t.py 
t.py:3:10: error: inconsistent use of tabs and spaces in indentation

Your environment

• CPython versions tested on: d310fc7
• Operating system and architecture: ubuntu 22.04 x86_64

Linked PRs

• gh-105713: Document that tokenize raises when mixing tabs/spaces #105723
• [3.12] gh-105713: Document that tokenize raises when mixing tabs/spaces (GH-105723) #105725

[lysnikolaou]

This is probably alright, yes. I'll open a PR to mention it in the docs.