udemy-dl memory leak when skipping downloads

[dragetd]

When the download starts skipping courses, something odd happens: The RAM usage starts increasing over time until the kernel OOM kills it. Maybe some sort of memory leak:

[r0oth3x49]

@dragetd how, can you spot the code part leaking the memory? Because i don't see any.

[dragetd]

No idea. Skipping because of 'fie already exists' does not seem to be the cause. The skips I had were because of 403's from udemy (I wonder how I got these xD).

I noticed the system swapping after some time and at some time killing the process. I then restarted and it worked at first, until I noticed the system swapping again some time later where I took this screenshot.

Maybe it does only get triggered by 4xx server responses. Or maybe it was just a glitch and we simply ignore it, up to you. Just thought I'd post the finding.

[r0oth3x49]

great i 'm closing the issue as invalid i don't see anything related i tried locally.

[irshsheik]

I faced the same problem.
Although not many downloads were skipped.
However, the memory is not released after a file is downloaded, The longer the download time of the course, the higher utilization of the RAM.
Ideally, the program memory consumption should be = memory required for running(which is pretty low) + memory required to download the current file( on 1080p, it won't be more than a 1GB).

Here is the timeline :

1. This was after 5 mins
   
2. After few min
   
3. this was when the program crashed. I have attached the logs also
   

The logs when it crashed completely.

[2020-09-17 23:49:19,665][udemy-dl] WARNI Subtitle : '137 Shift.ro'  (download skipped)
[2020-09-17 23:49:19,665][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:19,724][udemy-dl] WARNI Lecture : '138 Shift Solution'  (download skipped)
[2020-09-17 23:49:19,725][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:19,869][udemy-dl] WARNI Asset : '138 DLL-Shift.js'  (download skipped)
[2020-09-17 23:49:19,870][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:19,911][udemy-dl] WARNI Subtitle : '138 Shift Solution.pt'  (download skipped)
[2020-09-17 23:49:19,911][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:19,946][udemy-dl] WARNI Subtitle : '138 Shift Solution.fr'  (download skipped)
[2020-09-17 23:49:19,947][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:19,987][udemy-dl] WARNI Subtitle : '138 Shift Solution.es'  (download skipped)
[2020-09-17 23:49:19,988][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:20,028][udemy-dl] WARNI Subtitle : '138 Shift Solution.de'  (download skipped)
[2020-09-17 23:49:20,028][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:20,060][udemy-dl] WARNI Subtitle : '138 Shift Solution.id'  (download skipped)
[2020-09-17 23:49:20,060][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:20,089][udemy-dl] WARNI Subtitle : '138 Shift Solution.it'  (download skipped)
[2020-09-17 23:49:20,090][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:20,120][udemy-dl] WARNI Subtitle : '138 Shift Solution.pl'  (download skipped)
[2020-09-17 23:49:20,120][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:20,150][udemy-dl] WARNI Subtitle : '138 Shift Solution.ro'  (download skipped)

[r0oth3x49]

I faced the same problem.
Although not many downloads were skipped.
However, the memory is not released after a file is downloaded, The longer the download time of the course, the higher utilization of the RAM.
Ideally, the program memory consumption should be = memory required for running(which is pretty low) + memory required to download the current file( on 1080p, it won't be more than a 1GB).

Here is the timeline :

1. This was after 5 mins
   ![task after](https://user-images.githubusercontent.com/6179346/93511689-d2ea1200-f940-11ea-8824-0012862f4021.PNG)

2. After few min
   ![processes](https://user-images.githubusercontent.com/6179346/93511701-d7162f80-f940-11ea-9bc0-0938ad447400.PNG)

3. this was when the program crashed. I have attached the logs also
   ![task manager](https://user-images.githubusercontent.com/6179346/93511710-d8dff300-f940-11ea-809d-cccb84ccc46c.png)

The logs when it crashed completely.

[2020-09-17 23:49:19,665][udemy-dl] WARNI Subtitle : '137 Shift.ro'  (download skipped)
[2020-09-17 23:49:19,665][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:19,724][udemy-dl] WARNI Lecture : '138 Shift Solution'  (download skipped)
[2020-09-17 23:49:19,725][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:19,869][udemy-dl] WARNI Asset : '138 DLL-Shift.js'  (download skipped)
[2020-09-17 23:49:19,870][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:19,911][udemy-dl] WARNI Subtitle : '138 Shift Solution.pt'  (download skipped)
[2020-09-17 23:49:19,911][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:19,946][udemy-dl] WARNI Subtitle : '138 Shift Solution.fr'  (download skipped)
[2020-09-17 23:49:19,947][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:19,987][udemy-dl] WARNI Subtitle : '138 Shift Solution.es'  (download skipped)
[2020-09-17 23:49:19,988][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:20,028][udemy-dl] WARNI Subtitle : '138 Shift Solution.de'  (download skipped)
[2020-09-17 23:49:20,028][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:20,060][udemy-dl] WARNI Subtitle : '138 Shift Solution.id'  (download skipped)
[2020-09-17 23:49:20,060][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:20,089][udemy-dl] WARNI Subtitle : '138 Shift Solution.it'  (download skipped)
[2020-09-17 23:49:20,090][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:20,120][udemy-dl] WARNI Subtitle : '138 Shift Solution.pl'  (download skipped)
[2020-09-17 23:49:20,120][udemy-dl] ERROR Reason : [('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
[2020-09-17 23:49:20,150][udemy-dl] WARNI Subtitle : '138 Shift Solution.ro'  (download skipped)

thanks for the logs i will check it seems the issue only occurs when downloading subtitles

[r0oth3x49]

@igagrock hopefully the current commit will resolve the issue.

[irshsheik]

@r0oth3x49 - I am gonna test right away.
thanks

[irshsheik]

It is not fixed.
The issue still remains.

1. the memory pretty much still stacks up as long as the program keeps running.
2. When the file is not video or subtitles, the download simply ignores them and throws the warning of nonType.
3. When the program exits successfully or interrupted using keyboard crtl+c, the memory used is never released and the process is still running.

Also a command like below should only download sub-titles in english. But it does not happen.

python3 udemy-dl/udemy-dl.py https://www.udemy.com/course/js-algorithms-and-data-structures-masterclass --chapter-start 20 --lecture-start 138 --sub-lang en 

In my case, I am using WSL (ubuntu 20.04) on windows10. I am not sure if it is the case with this configuration only or it will happen on any windows/Linux system.