Python program not terminating properly

[kubatjak]

Hello,

I have been running a very simple python script using the SimpleHandler. The program runs correctly till the end but then it fails to terminate properly. The program cannot be killed by keyboard interupt. It happens in Windows command line and PowerShell and GitBash consoles. It happens only after using apply_file, just having an instance of Handler does not cause any issues.

Here's the script:

import osmium
import atexit


class Handler(osmium.SimpleHandler):
    def __init__(self) -> None:
        super().__init__()
        self.num_nodes = 0

    def node(self, n):
        self.num_nodes += 1


def cleanup():
    print('Cleanup called')
    print('Trying to terminate the program ...')


def main():
    osm_file = r'map_14.4086_50.0726_14.4195_50.0761.osm'
    handler = Handler()
    handler.apply_file(osm_file)
    print('# nodes:', handler.num_nodes)


if __name__ == '__main__':
    atexit.register(cleanup)
    try:
        main()
    except Exception as e:
        print(e)

Would someone know what could be the cause of the problem?

Thank you!

[pradeepgudipati]

Bumping it up, I have the same problem and memory is also not released.

I have had to specifically get the pid of the process and kill it.

python_pids = []
def end_program_pids():
    """
    Function to terminate all the python processes that were started by the main process.
    """
    if python_pids and len(python_pids) > 0:
        for pid in python_pids:
            try:
                print(f"Terminating process with pid {pid}")
                process = psutil.Process(pid)
                process.terminate()
            except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):
                pass


def get_python_process_pids():
    pids = psutil.pids()
    global python_pids
    python_pids = []
    parent_pid = os.getppid()  # Get the PPID of the current process

    for pid in pids:
        try:
            process = psutil.Process(pid)
            if 'python' in process.name() and process.ppid() == parent_pid:
                python_pids.append(pid)
        except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):
            pass
    return python_pids

[ImreSamu]

• Please share your environment
• If you are using Windows, please retest in WSL2.
• Check the integrity of your OSM input file.
• Verify if the similar example is working: https://github.com/osmcode/pyosmium/blob/master/examples/osm_file_stats.py

[ImreSamu]

Tested with https://download.geofabrik.de/europe/andorra-latest.osm.pbf
( osm_file = r'andorra-latest.osm.pbf' )
and it is working for me . ( ubuntu linux ; x86-64 ; Python 3.10.5 ; osmium 3.7.0 )

This is the expected results ?

$ python3 test.py && echo "terminated"
# nodes: 385362
Cleanup called
Trying to terminate the program ...
terminated

my theory:

• probably the r'map_14.4086_50.0726_14.4195_50.0761.osm' file is broken ( ~ not a valid osm.xml file )

Please re-test with the similar example code : https://github.com/osmcode/pyosmium/blob/master/examples/osm_file_stats.py

[Ebe66]

I'm having the exact same issue.
On windows 11, python 3.9.13, osmium 3.7.0
I have done multiple tests both om wsl2/Ubuntu and windows powershell. Each time the wsl version terminates after completion, the windows version does not. It finishes the program (last line is printed) but it does not terminate.

The program from the link above https://github.com/osmcode/pyosmium/blob/master/examples/osm_file_stats.py works fine and terminates on wsl. If I run the exact same program using the exact same input file in the same folder from a powershell window, it prints the stats but does not terminate.

Any help would be appreciated.

[lonvia]

None of the maintainers has access to a windows machine, so you will have to do some debugging yourself here, I'm afraid. At a minimum you need to find out, where exactly the applications hangs.

[Ebe66]

I would love to debug this, but how?
The programs run to their very last line of code. The program I "need" is listed below. On wsl it runs, prints the "...done" and returns to the command prompt On windows it does the same but after the "...done" the cursor blinks on a new line but it does not return to the command prompt.
If I comment out all (py)osmium calls it does terminate, if I then uncomment the "rel_parser = RelationsHandler(all_ways)" line it still terminates, but as soon as I uncomment "rel_parser.apply_file(file_name)" it does not terminate anymore.
I think the issue is not in this code as it does create the desired output both on wsl and windows.

So I think the next step would be to debug the pyosmium code but I don't have a clue on how to do so.

`import osmium
import os
import sys

#REL_TYPES = ["historic", "mtb", "bicycle", "foot", "hiking"]
REL_TYPES = ["bicycle", "mtb"]

all_ways = {}

class RelationsHandler(osmium.SimpleHandler):
def init(self, all_ways):
osmium.SimpleHandler.init(self)
self.all_ways = all_ways

def relation(self, r):
    if r.tags.get("type") == "route" and r.tags.get("route") in REL_TYPES:
        relation = {
            "route": "",
            "type": "",
        }

        relation["route"] = r.tags.get("route")
        relation["type"] = r.tags.get("type")

        for m in r.members:
            if m.type == "w":
                if m.ref not in self.all_ways:
                    self.all_ways[m.ref] = []
                self.all_ways[m.ref].append(relation)

class WayHandler(osmium.SimpleHandler):
def init(self, all_ways, way_writer):
osmium.SimpleHandler.init(self)
self.all_ways = all_ways
self.way_writer = way_writer
self.way_id = -800000000000

def way(self, w):
    if w.id in self.all_ways:
        rel_count = len(self.all_ways[w.id])

        for i in range(0, rel_count):
            relation = self.all_ways[w.id][i]

            if (
                (rel_count == 2 and i == 1)
                or
                (rel_count > 2 and (relation["route"] == "bicycle" or relation["route"] == "mtb"))
                ):
                way_nodes = []

                for node in w.nodes:
                    way_nodes.append(node)
                
                new_nodes = []

                for i in range(len(way_nodes)-1, -1, -1):
                    new_nodes.append(way_nodes[i])
                way = w.replace(id=self.way_id, nodes=new_nodes, tags=relation)
            else:
                way = w.replace(id=self.way_id, tags=relation)

            self.way_writer.add_way(way)
            self.way_id = self.way_id + 1

file_name = sys.argv[1]
output_tmp = sys.argv[1] + ".tmp.xml"
output_name = sys.argv[2]

#print("Parsing relations...")

rel_parser = RelationsHandler(all_ways)
rel_parser.apply_file(file_name)

#print("...done, found ways: %d" % len(all_ways))

#print("Extracting relation ways...")

if os.path.exists(output_tmp):
os.remove(output_tmp)

if os.path.exists(output_name):
os.remove(output_name)

way_writer = osmium.SimpleWriter(output_tmp)
way_parser = WayHandler(all_ways, way_writer)
way_parser.apply_file(file_name)
way_writer.close()

with open(output_name, "w") as output:
with open(output_tmp, "r") as tmp:
i = 0
for l in tmp:
output.write(l)
i = i + 1

os.remove(output_tmp)
#os.remove(bbox_name)

print("...done")

`

[lonvia]

Run the program under gdb (or another debugger) and get the stack trace for when it hangs. That might tell something.

[luwhite]

I'm experiencing the same issue when running https://github.com/osmcode/pyosmium/blob/master/examples/osm_file_stats.py on Windows Server 2022 Standard, while it works on Windows 10 Enterprise and Windows Server 2016 Datacenter. I tried to debug the hanging process with gdb and this is the result:

$ gdb python3.12.exe 23488
GNU gdb (GDB) 15.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3.12.exe...
Attaching to program: C:\msys64\mingw64\bin\python3.12.exe, process 23488
[New Thread 23488.0x3dc8]
(gdb) bt
#0  0x00007ffdbb263d81 in ntdll!DbgBreakPoint () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ffdbb29055e in ntdll!DbgUiRemoteBreakin () from C:\Windows\SYSTEM32\ntdll.dll
#2  0x00007ffdba894cb0 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#3  0x00007ffdbb23eceb in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#4  0x0000000000000000 in ?? ()

[lonvia]

Thanks for trying that. Unfortunately it doesn't tell us much. Can you check with info threads if there are more threads running and if yes, see what they have as back trace?

[Ebe66]

Unfortunately this kind of debugging is out of my league. What I did try however is running my ,very, little program with the python debugger and lo and behold, when doing that it works. I doubt it tells you much, but maybe every piece of info helps.

[jiawlu]

@lonvia I am experiencing the same issue. python does not terminate after all lines of code are executed. I'm using python 3.12.4 on windows 11 Education 23H3. osmium version 4.0.0.

[cmaloney111]

Hey everyone. I have also been experiencing this problem, so I spent a while looking into it. There appears to be an issue with the way the Reader class is implemented in libosmium (and not here). The code hangs during the initialization or reading process on Windows, but it works fine on Linux and macOS.

For example, if you run the following code on Windows, it hangs and does not clear memory. If you run it on Linux or MacOS, the program terminates. Note that you can place anything in test.pbf, whether it be blank, one character, or an actual valid pbf file:

from osmium.io import Reader

Reader('test.pbf')

Environment

• Operating System: Windows 11
• Python Version: 3.10.2
• libosmium Version: 2.20.0

Steps to Reproduce (On Windows)

1. Create a PBF file containing any data and give it a name (e.g., test.pbf).

2. Run the following Python code:

   from osmium.io import Reader
   
   Reader('test.pbf')

3. Observe that the program hangs without any output.

If there are any logs or debugging steps you recommend I take to further diagnose this issue, please let me know. I am happy to provide additional information or test potential fixes.

Thank you for your help!

[cmaloney111]

Hi @lonvia.

I tried the code you mentioned (adding the .close()), and unfortunately the result did not change. And yes, even after building pyosmium from the source on my system, I still get the same error. I have also tried to build everything from the source (pyosmium, libosmium, and libosmium's dependencies like zlib) but I can't seem to figure out how to get the debugger to not skip C++ frames Like you said, it definitely seems like the reader not being automatically closed (or "implicitly destroyed") is the problem here.

A few more details about my situation and what I found:

• Running the reader as a context manager generally does not help, but sometimes I have found that the program will terminate successfully. I am not sure why this happens and it is seemingly random.
• I am running the Python code using VScode and was only using MSVC to build C++ code to test libosmium directly. I run my files from the command line, so I doubt any IDEs are messing with the results
• The best solution I have found so far is to just kill the process (I like using task manager on Windows and just searching up "Python" then killing it) because any other attempt at getting the program to terminate does not seem to work.

[lonvia]

Thanks, that excludes some possible causes already. Just for good measure, can you try:

import osmium

r = Reader('test.osm')
del r

[joto]

Also can yout try printing something before and after the r.close() and the del r in lonvias suggestions. That will tell us more exactly where it hangs.

If it prints the "before" message but not the "after" message the next step would then be to look at the osmium::io::Reader::close() function in include/osmium/io/reader.hpp and print something out after each step in that code to tell us in which one is the culprit. Thanks!

[cmaloney111]

Hello @lonvia and @joto,

I have attempted deleting the reader, calling reader.close(), and even exiting from the program. Nothing seems to work. For example, in the following code, every print is executed, and the program still hangs at the end.

import gc
import osmium.io
import os

print("before reader")
r = osmium.io.Reader("data.pbf")
print("after reader")
r.close()
print("after closing")
del r
gc.collect()
print("after deleting and GC")
os._exit(1)

In case it helps, I also found that there appears to be two threads open while the program is hanging. One was mentioned earlier by @luwhite, but it appears the other thread is the one causing the problems. Here are the backtraces for both:

Thread 2 (Thread 33828.0xafe0):

1. ntdll!DbgBreakPoint()

   • Location: C:\WINDOWS\SYSTEM32\ntdll.dll
   • Address: 0x00007ffc42893e21

2. ntdll!DbgUiRemoteBreakin()

   • Location: C:\WINDOWS\SYSTEM32\ntdll.dll
   • Address: 0x00007ffc428c72ae

3. KERNEL32!BaseThreadInitThunk()

   • Location: C:\WINDOWS\System32\kernel32.dll
   • Address: 0x00007ffc4103257d

4. ntdll!RtlUserThreadStart()

   • Location: C:\WINDOWS\SYSTEM32\ntdll.dll
   • Address: 0x00007ffc4284af28

5. Unknown Function

   • Address: 0x0000000000000000

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

---

Thread 1 (Thread 33828.0xa2a8):

1. MSVCP140!?write@?$basic_ostream@_WU?$char_traits@_W@std@@@std@@QEAAAEAV12@PEB_W_J@Z()

   • Location: C:\Users\camer\anaconda3\envs\pyosmium_testing\msvcp140.dll
   • Address: 0x00007ffc19eb19e7

2. MSVCP140!?_Winerror_message@std@@YAKKPEADK@Z()

   • Location: C:\Users\camer\anaconda3\envs\pyosmium_testing\msvcp140.dll
   • Address: 0x00007ffc19e72780

3. io.cp38-win_amd64!PyInit_io()

   • Location: C:\Users\camer\anaconda3\envs\pyosmium_testing\lib\site-packages\osmium\io.cp38-win_amd64.pyd
   • Address: 0x00007ffc1969c62f

4. io.cp38-win_amd64!PyInit_io()

   • Location: C:\Users\camer\anaconda3\envs\pyosmium_testing\lib\site-packages\osmium\io.cp38-win_amd64.pyd
   • Address: 0x00007ffc196bbea3

5. ucrtbase!.sys_nerr()

   • Location: C:\WINDOWS\System32\ucrtbase.dll
   • Address: 0x00007ffc3fb63a93

6. ucrtbase!_o_free()

   • Location: C:\WINDOWS\System32\ucrtbase.dll
   • Address: 0x00007ffc3fb3042e

7. ucrtbase!_execute_onexit_table()

   • Location: C:\WINDOWS\System32\ucrtbase.dll
   • Address: 0x00007ffc3fb2dddd

8. io.cp38-win_amd64!PyInit_io()

   • Location: C:\Users\camer\anaconda3\envs\pyosmium_testing\lib\site-packages\osmium\io.cp38-win_amd64.pyd
   • Address: 0x00007ffc196b7c39

9. io.cp38-win_amd64!PyInit_io()

   • Location: C:\Users\camer\anaconda3\envs\pyosmium_testing\lib\site-packages\osmium\io.cp38-win_amd64.pyd
   • Address: 0x00007ffc196b7d5e

10. ntdll!RtlActivateActivationContextUnsafeFast()

    • Location: C:\WINDOWS\SYSTEM32\ntdll.dll
    • Address: 0x00007ffc42818b7f

11. ntdll!LdrShutdownProcess()

    • Location: C:\WINDOWS\SYSTEM32\ntdll.dll
    • Address: 0x00007ffc42841586

12. ntdll!RtlExitUserProcess()

    • Location: C:\WINDOWS\SYSTEM32\ntdll.dll
    • Address: 0x00007ffc4284117d

13. KERNEL32!ExitProcess()

    • Location: C:\WINDOWS\System32\kernel32.dll
    • Address: 0x00007ffc41037fab

14. ucrtbase!exit()

    • Location: C:\WINDOWS\System32\ucrtbase.dll
    • Address: 0x00007ffc3fb2bed8

15. ucrtbase!exit()

    • Location: C:\WINDOWS\System32\ucrtbase.dll
    • Address: 0x00007ffc3fb2c099

16. python!OPENSSL_Applink()

    • Address: 0x00007ff67501155f

17. KERNEL32!BaseThreadInitThunk()

    • Location: C:\WINDOWS\System32\kernel32.dll
    • Address: 0x00007ffc4103257d

18. ntdll!RtlUserThreadStart()

    • Location: C:\WINDOWS\SYSTEM32\ntdll.dll
    • Address: 0x00007ffc4284af28

19. Unknown Function

    • Address: 0x0000000000000000

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

I unfortunately am having a lot of trouble getting any debugger to work with both Python and C++. I used your helpful setup.py with the debug option (python setup.py --built_ext --inplace --debug) which generates all of the relevant pdb files. I have python installed with debugger symbols, and I have libosmium and all of its relevant depencies compiled with debugger symbols, but no debugger seems to want to "jump" between the python and the C++. If it starts in python, it won't step into the Reader instantiation. If it starts in C++, the code does not hang at the end.

[cmaloney111]

IMG_0832.2.mov

This also happens (where it terminates sometimes but doesn’t other times with no changes to the file).

[simsevi]

Not a "real" fix but a way to get around the issue in dirty code.
This tracks which threads are started by apply_file and kills them afterwards:

def apply_file_hardcore(self, handler, filepath, use_location):
        import psutil
        def list_thread_ids():
            process = psutil.Process(os.getpid())
            threads = process.threads()
            return {thread.id for thread in threads}
        
        threads_before = list_thread_ids()

        handler.apply_file(filepath,use_location)

        threads_after = list_thread_ids()
        new_threads = threads_after - threads_before
     

        import ctypes

        def kill_thread(thread_id):
            # Access rights for terminating a thread
            THREAD_TERMINATE = 0x0001

            # Open the thread with terminate access
            handle = ctypes.windll.kernel32.OpenThread(THREAD_TERMINATE, False, thread_id)
            if not handle:
                print(f"Failed to open thread {thread_id}")
                return False

            # Terminate the thread
            result = ctypes.windll.kernel32.TerminateThread(handle, 0)
            ctypes.windll.kernel32.CloseHandle(handle)

            if result == 0:
                print(f"Failed to terminate thread {thread_id}")
                return False
            
            return True

        

        # Terminate each new thread
        for thread_id in new_threads:
            kill_thread(thread_id)

The reason for the wrongful termination is loose threads in libosmium. Probably some binding issue that is triggered by a combination of libosmium's threadpool implementation, pybind11 and windows threading model.

[cmaloney111]

Thanks! This is great. The code I was using before was the following which was even worse since it would just kill my entire program instead of just the extra threads started by apply_file.

import subprocess, time
def run_with_timeout(cmd, timeout):
    start = time.time()
    proc = subprocess.Popen(cmd, shell=True)
    while proc.poll() is None and time.time() - start <= timeout: time.sleep(1)
    if proc.poll() is None: proc.terminate()

Your solution is much better! Thank you