CPU performance degrades when threads > 4

[ealmonte32]

Hi, I was doing some benchmark testing using sysbench on Oracle Linux 8 on the raspberry pi and noticed that although memory utilization and running processes out of the box in general are very good/impressive with Oracle Linux 8, that when the sysbench tool was given more than 4 threads to run, the Oracle Linux pi performed worse than a lower end pi running Rocky Linux.
For every extra 4 threads the test was given, the worse the pi running Oracle Linux performed.

I was wondering if this is a known issue?
Should this be reported somewhere else?
Below are screenshots, left is Rocky Linux and right is Oracle Linux.

[totalamateurhour]

We have some folks looking into this. One thing that may be part of the root cause is that the the UEK kernel for Oracle Linux on Arm currently uses 64K page size. The Rocky (Red Hat Compatible) kernel is likely using 4K page size.

[totalamateurhour]

Couple follow up comments:

From one of our developers:

I ran few tests with both the OS. higher latency on oracle linux is due to low available memory on Pi3. Sysbench needs more memory than available on RPi3 running with OL8 . As a reason it starts using swap space which causes higher latency to show up in the test. This is not the case with Rocky linux because it's 4k pagesize kernel.

I would not consider it a valid test configuration as Pi3 is not ideal for memory intensive test. Even the Rocky kernel was on the edge of using swap space with sysbench.

Further, the Pi3 has 1GB of RAM correct? In general, OL is aimed at larger systems and isn't necessarily tuned for hardware with such little memory. However, disabling the services that use up the most memory might get Sysbench running at parity with your other platform.

For example:

sudo systemctl disable firewalld
sudo systemctl disable tuned

[ealmonte32]

@totalamateurhour

Sorry I guess was not specific enough about sysbench test, let me clarify:
Yes both systems have 1GB of ram, the swap partition is completely off because I disable swap for testing other stuff since it wears down the SD card, I let OOM just kill the processes whenever I am doing anything that takes up all the memory...
Those services are not running on the OL8, I have not enabled them. Ironically they both are enabled on the Rocky linux system.

The tests were CPU performance tests, not memory, thus during the tests the memory consumption had no effect on the outcome of the results.

As you can see from the screenshot taken during the CPU tests, the memory does not increase much at all and there is plenty of it free.

I tried to see if making changes to the sysctl.conf for kernel.threads-max and other parameters to have similar settings as the other OS made any difference, but it didn't.

Mem before test..

Mem during test..

[totalamateurhour]

Someone on our side is investigating this on a Pi3

[totalamateurhour]

@ealmonte32

We have replicated the issue and found that Oracle Linux is currently missing the Raspberry Pi specific cpufreq module that regulates clock speed, and as a result your OL tests run with a CPU clock that is stuck at 600 MHz. We are working on a kernel update that includes this module and will dynamically scale the clock speed according to the cpufreq governor selected ("performance", by default). In the meantime, as a workaround you can force the clock to run at its maximum safe speed by adding the following to /boot/efi/config.txt:

arm_freq=1200
arm_freq_min=1200

Note that the RPi3 can (and will, in our case) overheat and shut down if you use values higher than 1200. Please refer to the official config.txt documentation for details.

Thanks for reporting the problem and providing the diagnostic information.

[totalamateurhour]

Closing this issue for now.

[ealmonte32]

@totalamateurhour

Thank you for looking into it and finding the culprit.
I can confirm that manually setting the cpu to 1200 (1400 is cpu default) does fix the issue.

[totalamateurhour]

FYI, 5.4.17-2136.305.5.4.el8uek.aarch64 has cpufreq built in.