Multiple recognize processes use up more than the maximum amount of cores set

[Ellpeck]

Describe the bug
In the Recognize settings, I set the maximum amount of cores to use for Recognize to 1. This works fine for each individual Recognize process. However, an issue arises when multiple types of recognition are set in the config, at which point multiple Recognize processes seem to start and run simultaneously, causing more than 1 CPU core to be used in total.

I would personally consider this a bug, but it might also boil down to a feature request: I think there should be an option to specify

• how many cores should be used in total or
• whether multiple Recognize processes are allowed to run simultaneously.

Thanks so much for considering this report, as well as for the app in general!

To Reproduce
Steps to reproduce the behavior:

1. Enable multiple types of recognition
2. Set core limit to 1
3. Wait until recognition starts automatically and check your CPU usage

Expected behavior
The "maximum amount of cores" setting is respected in some way, either through all processes sharing the same core (which might be difficult to achieve), or through only one process running simultaneously if the core limit is too low.

Recognize (please complete the following information):

• JS-only mode: No
• Enabled modes: All

Server (please complete the following information):

• Nextcloud: 25.0.0 RC 1
• OS: Ubuntu 18.04
• RAM: 16GB
• Processor Architecture: x64

[marcelklehr]

Hey @Ellpeck

That is a nice catch that we hadn't thought of yet. Nextcloud core apparently wants to run cron.php 5 minutes, but keeps them alive for 15minutes...

[Ellpeck]

Hi! Any update on this? Unfortunately, this issue is making me unable to use Recognize without disabling every recognition type but one, because otherwise it very quickly overloads my server :(

If there's any more details, logs or other info I can provide, do let me know!

[marcelklehr]

Yes, I've checked with the server repository, but changing the cron.php frequency doesn't seem to be an option, so I've changed the frequency of the classifier runs within recognize and set them to time-insensitive, so you can set a 4h window in the night, when recognize should run.

[Ellpeck]

Unfortunately, I'm finding that this issue is still happening, even when using version 3.0.1 of the app.

I also tried to route the node command through a script that sets its niceness to 19, causing it to have the lowest possible priority in terms of CPU scheduling, but the multiple processes still cause the server to struggle immensely.

It seems like most of the processes are actually the movinet classifiers, which seem to run for far longer than the 15 minute timespan that you said was allotted for them. However, even with those turned off, sometimes there will be multiple classification processes running, but they will have a less extreme impact on the overall performance than the movinet ones.

Here's another screenshot for reference. It should also be noted that the first process has been running for more than 15 minutes, and seems to be using up more than 100% CPU, meaning more than one core in this case, even though I have set the core limit to 1.

Edit: I'm now forcing lower CPU usage for the recognize processes by using cpulimit. It works, but it's far less than ideal. It would be great if there was more of an ability to customize the timings and/or CPU usage of the recognize processes.

Please let me know if there's any additional testing I can do, or any additional information I can provide!

[gymnae]

I am also facing this issue and have to disable this app because of this

[marcelklehr]

It's not a real 'fix' but as a workaround you can reduce the batch size of the individual jobs now in version 3.2.0, so they don't take as long anymore.

[Forza-tng]

It's not a real 'fix' but as a workaround you can reduce the batch size of the individual jobs now in version 3.2.0, so they don't take as long anymore.

Yea, this is what I have done so that the batch queue will be shorter, to avoid multiple node runs as much as possible.

In order to minimise the cpu impact, run cron.php with chrt -i 0*. This sets the idle CPU scheduling class on Linux, which is much lower than nice -n19

3-59/5 * * * * chrt -i 0  php -f /var/www/domains/my.domain/htdocs/nextcloud/cron.php >/dev/null 2>&1

https://www.man7.org/linux/man-pages/man1/chrt.1.html

[Forza-tng]

It's not a real 'fix' but as a workaround you can reduce the batch size of the individual jobs now in version 3.2.0, so they don't take as long anymore.

Is it possible to add an internal timer/counter in the classifyer cron code, so that it dynamically runs as many images as possible during 5 minutes? Then we don't have to worry about multiple jobs happening or adjusting the batch size manually.