[Windows] Embedded Python Memory Leaks #96853 ( Corrected 98%), Need testing

[AlexSoft73]

Hi Python team,

I detected python memory leaks when using it embedded on windows platform, unfortunately the issue I opened was closed.

I decided to correct the issue (though it is not 100% corrected), python now returns most of the memory it takes when finished.

I need test to be done with the modifications I did, especially if it requires any modifications for Unix based systems:

pylifecycle.c --> call function ____Py_ArenaMemoryRelease
obmalloc.c --> Define function ____Py_ArenaMemoryRelease pluss the trackig of memory allocations and releases

Pelase, test it and let me know.

Thanks you all
@AlexSoft73

[iritkatriel]

@AlexSoft73 I don't understand. Which test needs to be done?

[AlexSoft73]

I was working in a CPython copy fixing the issue of memory leaks, these memory leaks are not seeing if python is run as the main application since these memory gets reused by the python interpreter, BUT, if you embed python in applications created in any other language to use its benefits (Python is not the mail application in this case) is where you see the memory leaks, like I showed in the samples I put back to those days, iE: (pardon me, don't take literally the name of the python library function names I am writing from the top of my head and from what I get to recall, but the issue can be seeing) PyInitialize(...) Python gets memory from the system, this is OK! ... ...You run Python code here, OK! ... now you get done with Python and call it to finish: PyFinalize(...) Python never releases the memory it took in the call to PyInitialize, BAD!!!!!!! So, in my branch, you run Python embed and you can see most of the memory being returned back to the System, the modified CPython code that I wrote explains what I did and where the problem was. But, back to those days it seemed to me that CPython community was not interested in fixing this. In that sample I showed ther main reason python was not releasing the memory but the only answers I saw were negatives, felt a bit disappointed and I put aside my participation in the project. But if you are interested in getting CPython improved, here I am and you are very welcomed Thanks for your mail Alexei

…

________________________________ From: Irit Katriel ***@***.***> Sent: Friday, December 1, 2023 09:28 PM To: python/cpython ***@***.***> Cc: AlexSoft73 ***@***.***>; Mention ***@***.***> Subject: Re: [python/cpython] [Windows] Embedded Python Memory Leaks #96853 ( Corrected 98%), Need testing (Issue #98359) @AlexSoft73<https://github.com/AlexSoft73> I don't understand. Which test needs to be done? — Reply to this email directly, view it on GitHub<#98359 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3C7TGHD5LMA3TQOUUAORGLYHJDWBAVCNFSM6AAAAAARHHUALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZWHAYDCNBVGM>. You are receiving this because you were mentioned.

[vstinner]

I don't understand how to reproduce your issue. Which Python version did you test?

Python 3.12 now releases most memory in Py_Finalize(), but not all stdlib extensions have been updated yet.

[vstinner]

detected python memory leaks when using it embedded on windows platform, unfortunately the issue I opened was closed.

Which issue? Do you have a link?

[vstinner]

Ah, wait, there are 2 other issues:

• Py_Finalize() function, why it does not finalise Python?? #98524 -- about numpy: it's not the right bug tracker, we don't develop numpy here.
• [Windows] Embedded Python Memory Leaks #96853 -- I closed this and asked to open a new issue if you have a reproducer: C code to build which reproduce the issue. You didn't provide it and just gave vague information.

[AlexSoft73]

Hi Victor, If you run Python in the Mac world, probably you are right, I am talking running CPython (embeded ) in Windows Platform, have you tried? thanks

…

________________________________ From: Victor Stinner ***@***.***> Sent: Wednesday, December 6, 2023 08:06 PM To: python/cpython ***@***.***> Cc: AlexSoft73 ***@***.***>; Mention ***@***.***> Subject: Re: [python/cpython] [Windows] Embedded Python Memory Leaks #96853 ( Corrected 98%), Need testing (Issue #98359) I don't understand how to reproduce your issue. Which Python version did you test? Python 3.12 now releases most memory in Py_Finalize(), but not all stdlib extensions have been updated yet. — Reply to this email directly, view it on GitHub<#98359 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3C7TGCSYNTI3SIVLQXT5GLYIDF2ZAVCNFSM6AAAAAARHHUALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBTGYYDSNJUGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[AlexSoft73]

If you search GitSub for this you will see, I left a copy with the fix made where 98% of memory is released back to the system, once more, CPython embedded on Windows Platform. Thanks

…

________________________________ From: Victor Stinner ***@***.***> Sent: Wednesday, December 6, 2023 08:07 PM To: python/cpython ***@***.***> Cc: AlexSoft73 ***@***.***>; Mention ***@***.***> Subject: Re: [python/cpython] [Windows] Embedded Python Memory Leaks #96853 ( Corrected 98%), Need testing (Issue #98359) detected python memory leaks when using it embedded on windows platform, unfortunately the issue I opened was closed. Which issue? Do you have a link? — Reply to this email directly, view it on GitHub<#98359 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3C7TGHD2Y5AP3HQMOXMPATYIDF6XAVCNFSM6AAAAAARHHUALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBTGYYTAOJQGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[AlexSoft73]

And Yes, you were one of a few closing the tickets I opened and never paid attention to what I was simply asking. And once more, I am taking about CPython embedded on Windows Platform. Thanks

…

________________________________ From: Victor Stinner ***@***.***> Sent: Wednesday, December 6, 2023 08:23 PM To: python/cpython ***@***.***> Cc: AlexSoft73 ***@***.***>; Mention ***@***.***> Subject: Re: [python/cpython] [Windows] Embedded Python Memory Leaks #96853 ( Corrected 98%), Need testing (Issue #98359) Ah, wait, there are 2 other issues: * #98524<#98524> -- about numpy: it's not the right bug tracker, we don't develop numpy here. * #96853<#96853> -- I closed this and asked to open a new issue if you have a reproducer: C code to build which reproduce the issue. You didn't provide it and just gave vague information. — Reply to this email directly, view it on GitHub<#98359 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3C7TGCPAMUYXKBSISQP7MDYIDH5JAVCNFSM6AAAAAARHHUALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBTGYZTEOJXGI>. You are receiving this because you were mentioned.

[neonene]

Repeating OP's example 2000 times:

#include <Python.h>
int main(int argc, char *argv[])
{
    for (int N = 0; N < 2000; N++) {
        printf("%d\n", N);
        Py_InitializeEx(0);
        Py_FinalizeEx();
    }
    return 0;
}

When running this code on an embedded x64 executable built with MSVC v.1936, the following 3 commits from the left seem to cause serious memory consumption (taken from the Task Manager):

N	3.12.0a1+ 67807cf	3.12.0a7+ ea2c001	3.13.0a0+ 15d4c9f		3.11.x 3.12.0a1
500	330MB	770MB	1000MB		4MB
1000	650MB	1500MB	2000MB		4MB
1500	970MB	2240MB	3000MB		4MB
2000	1290MB	3000MB	4000MB		4MB

I'm not sure if they affect each other.
The python.org official binaries (.lib/.dll) can also be used to confirm.

cc @ericsnowcurrently @eduardo-elizondo @markshannon

[AlexSoft73]

Is this test run tuning Peyton as main application or embedded? Thanks Alexei On Dec 8, 2023, at 1:08 PM, neonene ***@***.***> wrote:  Repeating OP's example<#96853 (comment)> 2000 times: #include <Python.h> int main(int argc, char *argv[]) { for (int N = 0; N < 2000; N++) { printf("%d\n", N); Py_InitializeEx(0); Py_FinalizeEx(); } return 0; } When running this code on an embedded x64 executable built with MSVC v.1936, the following 3 commits from the left seem to cause serious memory consumption (taken from the Task Manager): N 3.12.0a1+ 67807cf<67807cf> 3.12.0a7+ ea2c001<ea2c001> 3.13.0a0+ 15d4c9f<15d4c9f> 3.11.x 3.12.0a1 500 330MB 770MB 1000MB 4MB 1000 650MB 1500MB 2000MB 4MB 1500 970MB 2240MB 3000MB 4MB 2000 1290MB 3000MB 4000MB 4MB I'm not sure if they affect each other. The python.org official binaries (.lib/.dll) can also be used to confirm. cc @ericsnowcurrently<https://github.com/ericsnowcurrently> @eduardo-elizondo<https://github.com/eduardo-elizondo> @markshannon<https://github.com/markshannon> — Reply to this email directly, view it on GitHub<#98359 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3C7TGEPTJLE5PB57FCW6HTYINJQJAVCNFSM6AAAAAARHHUALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBXGYYDSOJYGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[neonene]

Is this test run tuning Peyton as main application or embedded?

As embedded.

[AlexSoft73]

Hi guys and again thanks for your time I am sending you a little sample (created in MSVS2019 compile for Windows) that shows something, once more, I always repeat it since this cannot be seen when python is used as main application The following program showed these results: Memory usage: 4292608------------------------> Before Python was Initialized Memory usage: 10498048----------------------> Python initialized, this is ok, Python got the memory for its interpreter Memory usage: 1000243----------------------> Python is Finalized here, the memory usage should be around 4292608, the usage before Python initializes Memory usage difference between start and end: 5709824, if this is not a leak, then what is it? thanks Alexei //C/c++ sample //-------------------------------------------------------------------------------------------------------------------------------------- #include <windows.h> #include <psapi.h> //#define ALEXDEB _DEBUG #undef _DEBUG #include <Python.h> //#define _DEBUG ALEXDEB void ___print(wchar_t* pToPrint) { OutputDebugString(pToPrint); wprintf(pToPrint); } float PrintMemoryInfo(DWORD processID) { HANDLE hProcess; PROCESS_MEMORY_COUNTERS pmc; double f; hProcess = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, processID); if (NULL == hProcess) return; if (GetProcessMemoryInfo(hProcess, &pmc, sizeof(pmc))) { wchar_t Info[1024]; memset(Info, 0, 1024 * sizeof(wchar_t)); f = pmc.WorkingSetSize; swprintf(Info, 1023, L"Memory usage: %1.0f\n", f); ___print((wchar_t*)Info); } CloseHandle(hProcess); return f; } /****************************** WinMain() *************************** * C program entry point. * * */ int WINAPI wWinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, PWSTR pCmdLine, int nCmdShow) //int main() Substitute by the main function of your c copiler { DWORD I; DWORD ProdId = GetCurrentProcessId(); double BeforePyInit, AfterPyInit, AfterPyFinalized; wchar_t Info[1024]; memset(Info, 0, 1024 * sizeof(wchar_t)); BeforePyInit = PrintMemoryInfo(ProdId);//Memory used by application before First call to Py_Initialize Py_Initialize(0); AfterPyInit = PrintMemoryInfo(ProdId);//Memory allocated by Python on start up Py_Finalize(); AfterPyFinalized = PrintMemoryInfo(ProdId);//At this point the memory used should be around the sized we got printed in the first call swprintf(Info, 1023, L"Memory usage difference between start and end: %1.0f\n", AfterPyFinalized - BeforePyInit);

…

___print(Info); return 0; }

[neonene]

Your example at least needs EmptyWorkingSet() Windows API like:

    hProcess = OpenProcess(
        PROCESS_QUERY_INFORMATION |
        PROCESS_VM_READ |
        PROCESS_SET_QUOTA,  // added
        ...
    );
    ...

    if (EmptyWorkingSet(hProcess)) {  // added
        if (GetProcessMemoryInfo(hProcess, ...)) {
            ...
        }
    }
    CloseHandle(hProcess);

And I repeated the sequence of the main function to see the risk of an out-of-memory.
Then Python 3.11.7 shows:
Then the executable embedded with Python 3.11.7 shows:

(1st)
Memory usage: 217088  // BeforePyInit
Memory usage: 212992  // AfterPyInit 
Memory usage: 204800  // AfterPyFinalized 
Memory usage difference between start and end: -12288
(2nd)
Memory usage: 200704
Memory usage: 212992
Memory usage: 196608
Memory usage difference between start and end: -4096
(3rd)
Memory usage: 196608
Memory usage: 212992
Memory usage: 204800
Memory usage difference between start and end: 8192
(4th)
Memory usage: 200704
Memory usage: 212992
Memory usage: 200704
Memory usage difference between start and end: 0
(5th)
Memory usage: 200704
Memory usage: 212992
Memory usage: 200704
Memory usage difference between start and end: 0
...

If there is any leak, EmptyWorkingSet() cannot reduce the usage constantly like that. Note that GetProcessMemoryInfo() is not recommended for checking a memory leak. See:

• https://forums.wxwidgets.org/viewtopic.php?t=44641
• https://learn.microsoft.com/en-us/windows-hardware/test/wpt/wpa-reference-set