Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

3.12+: segfault tokenizing multiline fstring placeholder #105718

Closed
asottile opened this issue Jun 13, 2023 · 2 comments
Closed

3.12+: segfault tokenizing multiline fstring placeholder #105718

asottile opened this issue Jun 13, 2023 · 2 comments
Assignees
@pablogsal @lysnikolaou
Labels
3.11 only security fixes 3.12 bugs and security fixes type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@asottile
Copy link
Contributor

asottile commented Jun 13, 2023 •
edited by bedevere-bot
Loading

Bug report

f'{
    hello
}:{world}'

output:

$ python3.12 -m tokenize t.py 
free(): invalid pointer
Aborted (core dumped)

here's a backtrace:

(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352824640)
    at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737352824640)
    at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737352824640, signo=signo@entry=6)
    at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c42476 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c896f6 in __libc_message (action=action@entry=do_abort, 
    fmt=fmt@entry=0x7ffff7ddbb8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007ffff7ca0d7c in malloc_printerr (
    str=str@entry=0x7ffff7dd9764 "free(): invalid pointer")
    at ./malloc/malloc.c:5664
#7  0x00007ffff7ca2ac4 in _int_free (av=<optimized out>, p=<optimized out>, 
    have_lock=0) at ./malloc/malloc.c:4439
#8  0x00007ffff7ca54d3 in __GI___libc_free (mem=<optimized out>)
    at ./malloc/malloc.c:3391
#9  0x000055555572430d in PyMem_RawFree (ptr=<optimized out>)
    at Objects/obmalloc.c:685
#10 _PyObject_Free (ctx=<optimized out>, p=<optimized out>)
    at Objects/obmalloc.c:1853
#11 _PyObject_Free (ctx=<optimized out>, p=<optimized out>)
    at Objects/obmalloc.c:1843
#12 0x000055555569540c in _PyTokenizer_Free (tok=0x555555d113a0)
    at Parser/tokenizer.c:1007
#13 0x00005555557cddeb in tokenizeriter_dealloc (it=0x7ffff7427f10)
    at Python/Python-tokenize.c:280
#14 0x00005555558065aa in Py_DECREF (op=<optimized out>)
    at ./Include/object.h:692
#15 Py_XDECREF (op=<optimized out>) at ./Include/object.h:788
#16 _PyFrame_ClearExceptCode (frame=0x7ffff74b38d8) at Python/frame.c:140
#17 0x00005555557de92f in clear_gen_frame (frame=0x7ffff74b38d8, 
    tstate=0x555555c17c68 <_PyRuntime+459432>) at Python/ceval.c:1492
#18 _PyEvalFrameClearAndPop (frame=0x7ffff74b38d8, 
    tstate=0x555555c17c68 <_PyRuntime+459432>) at Python/ceval.c:1504
#19 _PyEvalFrameClearAndPop (tstate=0x555555c17c68 <_PyRuntime+459432>, 
    frame=0x7ffff74b38d8) at Python/ceval.c:1498
#20 0x0000555555655e4f in _PyEval_EvalFrameDefault (tstate=<optimized out>, 
--Type <RET> for more, q to quit, c to continue without paging--c
    frame=0x7ffff74c2c88, throwflag=<optimized out>) at Python/bytecodes.c:724
#21 0x00005555556dabfd in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff74c2c88, tstate=<optimized out>) at ./Include/internal/pycore_ceval.h:87
#22 gen_send_ex2 (closing=0, exc=0, presult=<synthetic pointer>, arg=0x0, gen=0x7ffff74c2c40) at Objects/genobject.c:230
#23 gen_iternext (gen=0x7ffff74c2c40) at Objects/genobject.c:608
#24 0x00005555556f08e9 in list_extend (self=self@entry=0x7ffff7313400, iterable=iterable@entry=0x7ffff74c2c40) at Objects/listobject.c:944
#25 0x00005555556f0f68 in list___init___impl (iterable=0x7ffff74c2c40, self=0x7ffff7313400) at Objects/listobject.c:2792
#26 list___init___impl (iterable=0x7ffff74c2c40, self=0x7ffff7313400) at Objects/listobject.c:2778
#27 list_vectorcall (type=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/listobject.c:2817
#28 0x00005555556bcce3 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7ffff74c7b40, callable=0x555555abe780 <PyList_Type>, tstate=0x555555c17c68 <_PyRuntime+459432>) at ./Include/internal/pycore_call.h:92
#29 PyObject_Vectorcall (callable=callable@entry=0x555555abe780 <PyList_Type>, args=args@entry=0x7ffff7fb0318, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:325
#30 0x0000555555657503 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7fb0248, throwflag=<optimized out>) at Python/bytecodes.c:2744
#31 0x00005555557def2f in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb01b8, tstate=0x555555c17c68 <_PyRuntime+459432>) at ./Include/internal/pycore_ceval.h:87
#32 _PyEval_Vector (args=0x0, argcount=0, kwnames=0x0, locals=0x7ffff75fd4c0, func=0x7ffff75a04a0, tstate=0x555555c17c68 <_PyRuntime+459432>) at Python/ceval.c:1610
#33 PyEval_EvalCode (co=co@entry=0x555555ca11f0, globals=globals@entry=0x7ffff75fd4c0, locals=locals@entry=0x7ffff75fd4c0) at Python/ceval.c:567
#34 0x00005555557dc5b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ffff75fd4c0, globals=0x7ffff75fd4c0, source=0x555555ca11f0) at Python/bltinmodule.c:1079
#35 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:583
#36 0x000055555571a943 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7ffff7595d00, args=0x7ffff7fb0180, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:438
#37 0x00005555556bcce3 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7ffff75fd4c0, callable=0x7ffff7595d00, tstate=0x555555c17c68 <_PyRuntime+459432>) at ./Include/internal/pycore_call.h:92
#38 PyObject_Vectorcall (callable=callable@entry=0x7ffff7595d00, args=args@entry=0x7ffff7fb0180, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:325
#39 0x0000555555657503 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7fb00d8, throwflag=<optimized out>) at Python/bytecodes.c:2744
#40 0x0000555555866e39 in pymain_run_module (modname=<optimized out>, set_argv0=set_argv0@entry=1) at Modules/main.c:300
#41 0x00005555558674a7 in pymain_run_python (exitcode=0x7fffffffde40) at Modules/main.c:604
#42 0x0000555555867f70 in Py_RunMain () at Modules/main.c:688
#43 pymain_main (args=0x7fffffffde00) at Modules/main.c:718
#44 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:742
#45 0x00007ffff7c29d90 in __libc_start_call_main (main=main@entry=0x5555556519e0 <main>, argc=argc@entry=4, argv=argv@entry=0x7fffffffdf88) at ../sysdeps/nptl/libc_start_call_main.h:58
#46 0x00007ffff7c29e40 in __libc_start_main_impl (main=0x5555556519e0 <main>, argc=4, argv=0x7fffffffdf88, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdf78) at ../csu/libc-start.c:392
#47 0x000055555565e585 in _start ()

Your environment

  • CPython versions tested on: 8da9d1b
  • Operating system and architecture: ubuntu 22.04 x86_64

Linked PRs
@asottile asottile added the type-bug An unexpected behavior, bug, or error label Jun 13, 2023
@AlexWaygood AlexWaygood added 3.11 only security fixes type-crash A hard crash of the interpreter, possibly with a core dump 3.12 bugs and security fixes and removed type-bug An unexpected behavior, bug, or error labels Jun 13, 2023
@lysnikolaou
Copy link
Contributor

Looking into this.

lysnikolaou added a commit to lysnikolaou/cpython that referenced this issue Jun 13, 2023
@lysnikolaou
pythongh-105718: Fix buffer allocation in tokenizer with readline
9edeeaf
This was referenced Jun 13, 2023
Merged
Merged
pablogsal pushed a commit that referenced this issue Jun 13, 2023
@lysnikolaou
gh-105718: Fix buffer allocation in tokenizer with readline (#105728)
abfbab6
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 13, 2023
@lysnikolaou @miss-islington
pythongh-105718: Fix buffer allocation in tokenizer with readline (py…
139513a 
…thonGH-105728)

(cherry picked from commit abfbab6)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
pablogsal pushed a commit that referenced this issue Jun 13, 2023
@miss-islington @lysnikolaou
[3.12] gh-105718: Fix buffer allocation in tokenizer with readline (G…
9c51ea5 
…H-105728) (#105729)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
@lysnikolaou
Copy link
Contributor

Fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pablogsal pablogsal

@lysnikolaou lysnikolaou

Labels
3.11 only security fixes 3.12 bugs and security fixes type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@asottile @pablogsal @lysnikolaou @AlexWaygood