Rust progams hang on windows

[graydon]

There's a threading bug in the runtime that seems quite easy to trip for windows windows users (and sometimes the buildbots). Described in #1602 but this bug exists to track it. Symptom is just the compiler hanging. Not good.

[brson]

@ssylvan - We've made some simplications and fixes to the scheduler. I cannot reproduce the deadlock on the bots today after a few hours of trying. Can you still reproduce the hang?

[ssylvan]

Is there a windows binary somewhere? I don't have a windows dev environment set up.

[brson]

@ssylvan - http://dl.rust-lang.org.s3.amazonaws.com/dist/rust-0.1.1-install.exe

[ssylvan]

Sorry, still hangs. Doesn't appear to be any user-mode code running, it's completely deadlocked.

[brson]

ok. thanks for all your help, ssylvan!

[graydon]

@ssylvan can I ask for some diagnostic information from your machine? For example: OS version, core count, mingw version, etc? Something like the short text report from hwinfo or the advanced system info panel in windows?

Additionally, could you try attaching to it with gdb --pid <pid> and getting backtraces on the locked threads?

[ssylvan]

Sure. I run Windows 7 Professional SP1. Core i7 2600K (4 cores, 8 threads).

I did just remembered that I have msys installed, so I opened up the msys batch file (which I guess approximates a bash terminal?), and ran rustc from there and it worked fine! EDIT: so to clarify, it hangs when you just run rustc.exe normally, but if I do it through msys.bat it seems to work ENDEDIT.

I haven't used gdb before, but here's the log of me attempting to get the back traces from all running threads from the hung rustc.exe (just run through the normal windows command prompt):

C:\Users\Sebastian\Documents\rust>gdb --pid 14584
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "mingw32".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Attaching to process 14584
[New Thread 14584.0x4a98]
[New Thread 14584.0x2920]
[New Thread 14584.0x2880]
[New Thread 14584.0x3570]
[New Thread 14584.0x1a84]
[New Thread 14584.0x3e94]
[New Thread 14584.0xd00]
[New Thread 14584.0x3974]
[New Thread 14584.0x3c64]
[New Thread 14584.0x3618]
Reading symbols from c:\program files (x86)\rust\bin\rustc.exe...(no debugging s
ymbols found)...done.
Switching to Thread 14584.0x3618 info threads
Id Target Id Frame

• 10 Thread 14584.0x3618 0x775f000d in ntdll!LdrFindResource_U ()
  from C:\Windows\SysWOW64\ntdll.dll
  9 Thread 14584.0x3c64 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  8 Thread 14584.0x3974 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  7 Thread 14584.0xd00 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  6 Thread 14584.0x3e94 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  5 Thread 14584.0x1a84 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  4 Thread 14584.0x3570 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  3 Thread 14584.0x2880 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  2 Thread 14584.0x2920 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  1 Thread 14584.0x4a98 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  (gdb) bt full
  #0 0x775f000d in ntdll!LdrFindResource_U ()
  from C:\Windows\SysWOW64\ntdll.dll
  No symbol table info available.
  Thread a session or semantic context through IL #1 0x7767f896 in ntdll!RtlQueryTimeZoneInformation ()
  from C:\Windows\SysWOW64\ntdll.dll
  No symbol table info available.
  bind's glue function should tail-call its target  #2 0x6eac6ae1 in ?? ()
  No symbol table info available.
  abstract common patterns in AST formatters #3 0x00000000 in ?? ()
  No symbol table info available.
  (gdb) thread 9
  [Switching to thread 9 (Thread 14584.0x3c64)]
  #0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  (gdb) bt full
  #0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  No symbol table info available.
  Thread a session or semantic context through IL #1 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
  from C:\Windows\SysWOW64\ntdll.dll
  No symbol table info available.
  bind's glue function should tail-call its target  #2 0x75460a91 in WaitForSingleObjectEx ()
  from C:\Windows\syswow64\KernelBase.dll
  No symbol table info available.
  warning: (Internal error: pc 0xdf in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xdf in read in psymtab, but not in symtab.)

#3 0x000000e0 in ?? ()
warning: (Internal error: pc 0xdf in read in psymtab, but not in symtab.)

No symbol table info available.
warning: (Internal error: pc 0xdf in read in psymtab, but not in symtab.)

#4 0x00000000 in ?? ()
No symbol table info available.
(gdb) thread 8
[Switching to thread 8 (Thread 14584.0x3974)]
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt full
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#1 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#2 0x75460a91 in WaitForSingleObjectEx ()
from C:\Windows\syswow64\KernelBase.dll
No symbol table info available.
warning: (Internal error: pc 0xd7 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xd7 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xd7 in read in psymtab, but not in symtab.)

#3 0x000000d8 in ?? ()
warning: (Internal error: pc 0xd7 in read in psymtab, but not in symtab.)

No symbol table info available.
warning: (Internal error: pc 0xd7 in read in psymtab, but not in symtab.)

#4 0x00000000 in ?? ()
No symbol table info available.
(gdb) thread 7
[Switching to thread 7 (Thread 14584.0xd00)]
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt full
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#1 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#2 0x75460a91 in WaitForSingleObjectEx ()
from C:\Windows\syswow64\KernelBase.dll
No symbol table info available.
warning: (Internal error: pc 0xcf in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xcf in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xcf in read in psymtab, but not in symtab.)

#3 0x000000d0 in ?? ()
warning: (Internal error: pc 0xcf in read in psymtab, but not in symtab.)

No symbol table info available.
warning: (Internal error: pc 0xcf in read in psymtab, but not in symtab.)

#4 0x00000000 in ?? ()
No symbol table info available.
(gdb) thread 6
[Switching to thread 6 (Thread 14584.0x3e94)]
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt full
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#1 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#2 0x75460a91 in WaitForSingleObjectEx ()
from C:\Windows\syswow64\KernelBase.dll
No symbol table info available.
warning: (Internal error: pc 0xc7 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xc7 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xc7 in read in psymtab, but not in symtab.)

#3 0x000000c8 in ?? ()
warning: (Internal error: pc 0xc7 in read in psymtab, but not in symtab.)

No symbol table info available.
warning: (Internal error: pc 0xc7 in read in psymtab, but not in symtab.)

#4 0x00000000 in ?? ()
No symbol table info available.
(gdb) thread 5 bt full
A syntax error in expression, near `bt full'.
(gdb) thread 5
[Switching to thread 5 (Thread 14584.0x1a84)]
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt full
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#1 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#2 0x75460a91 in WaitForSingleObjectEx ()
from C:\Windows\syswow64\KernelBase.dll
No symbol table info available.
warning: (Internal error: pc 0xbf in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xbf in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xbf in read in psymtab, but not in symtab.)

#3 0x000000c0 in ?? ()
warning: (Internal error: pc 0xbf in read in psymtab, but not in symtab.)

No symbol table info available.
warning: (Internal error: pc 0xbf in read in psymtab, but not in symtab.)

#4 0x00000000 in ?? ()
No symbol table info available.
(gdb) thread 4
[Switching to thread 4 (Thread 14584.0x3570)]
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt full
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#1 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#2 0x75460a91 in WaitForSingleObjectEx ()
from C:\Windows\syswow64\KernelBase.dll
No symbol table info available.
#3 0xffffffff in ?? ()
No symbol table info available.
#4 0x00000000 in ?? ()
No symbol table info available.
(gdb) thread 3
[Switching to thread 3 (Thread 14584.0x2880)]
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt full
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#1 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#2 0x75460a91 in WaitForSingleObjectEx ()
from C:\Windows\syswow64\KernelBase.dll
No symbol table info available.
warning: (Internal error: pc 0xaf in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xaf in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xaf in read in psymtab, but not in symtab.)

#3 0x000000b0 in ?? ()
warning: (Internal error: pc 0xaf in read in psymtab, but not in symtab.)

No symbol table info available.
warning: (Internal error: pc 0xaf in read in psymtab, but not in symtab.)

#4 0x00000000 in ?? ()
No symbol table info available.
(gdb) thread 2
[Switching to thread 2 (Thread 14584.0x2920)]
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt full
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#1 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#2 0x75460a91 in WaitForSingleObjectEx ()
from C:\Windows\syswow64\KernelBase.dll
No symbol table info available.
warning: (Internal error: pc 0xa3 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xa3 in read in psymtab, but not in symtab.)

#3 0x000000a4 in ?? ()
warning: (Internal error: pc 0xa3 in read in psymtab, but not in symtab.)

No symbol table info available.
warning: (Internal error: pc 0xa3 in read in psymtab, but not in symtab.)

#4 0x00000000 in ?? ()
No symbol table info available.
(gdb) thread 1
[Switching to thread 1 (Thread 14584.0x4a98)]
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt full
#0 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#1 0x775ff8b1 in ntdll!RtlUpdateClonedSRWLock ()
from C:\Windows\SysWOW64\ntdll.dll
No symbol table info available.
#2 0x75460a91 in WaitForSingleObjectEx ()
from C:\Windows\syswow64\KernelBase.dll
No symbol table info available.
warning: (Internal error: pc 0xeb in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xeb in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xeb in read in psymtab, but not in symtab.)

#3 0x000000ec in ?? ()
warning: (Internal error: pc 0xeb in read in psymtab, but not in symtab.)

No symbol table info available.
warning: (Internal error: pc 0xeb in read in psymtab, but not in symtab.)

#4 0x00000000 in ?? ()
No symbol table info available.
(gdb)

[ssylvan]

More information (I know what's wrong, and have a workaround)! I logged all the system events for the running application, and it turns out that rustc.exe is attempting to run a bunch of stuff that isn't actually installed by the rust installer. It happens to find e.g. gcc.exe on my system, but that's because I randomly had mingw installed. That's not guaranteed to be the case.

More importantly in my case, it tries to find rm.exe and fails, just before the process stops completely. rm.exe is not on the default mingw bin path (but is in a different path, which is included by msys.bat). Adding my c:\MinGW\msys\1.0\bin path to the system path makes it find rm.exe and the problem goes away.

Repeat: everything now works, I just had to manually make sure rm.exe was in my path.

It would be better if these things used system APIs to handle files instead of trying to call unix utillities in windows...

I've saved a log of the system events so you can see what's going on (with stack traces for each event). It's almost 8MB, so be careful what application you open it in... it can take a while to parse :-)

https://skydrive.live.com/redir.aspx?cid=00108ea783f4fa32&resid=108EA783F4FA32!351&parid=108EA783F4FA32!130

[brson]

I wonder why this manifested as hanging instead of failure.

[graydon]

Ok. Great. I concur we should get away from calling unix utilities as subprocesses. We've been developing on mingw and in the longer term need to switch over to msvc. But this is quite different from a "threads all hang" situation. I'm comfortable closing this and opening a new one for "don't rely on mingw" or such.

[graydon]

Filed as #1768 , closing this one.

[ssylvan]

Shouldn't it at least not hang when it can't find the tool? Should the installer check for all the tools it needs (or install them)? I can imagine lots of people hitting this on windows.