Standardize the benchmarks I use across multiple boards

[geerlingguy]

Some of the more standard benchmarks I've run include:

• Geekbench 5 (for ARM / RISC-V, in preview)
• disk-benchmark.sh (my script in my pi-cluster project, uses fio and iozone)
• PiBenchmarks.com (community project by @TheRemote, uses fio, hdparm, dd, iozone)
• Top500 / HPL Linpack (compiled on the board — this is nice to test long-term stability and thermals under heavy load)
• Phoronix Test Suite (sometimes, for certain tests... it doesn't work OOTB on ARM64 for a lot of tests, especially for GPU or more exotic CPU tests. See this sparkfun article for some good info)
• Power (testing power consumption over time during HPL and Geekbench, at least)

I've looked at some other tests, too, like @ThomasKaiser's sbc-bench scripts.

I'd like to have a nice standard suite of benchmarks on a basic level for the main things—like CPU, GPU (maybe), RAM, Disk, and Network). Then it would be nice to have a deeper set if I'm trying to get a better understanding of edge cases.

[ThomasKaiser]

I've looked at some other tests, too, like @ThomasKaiser's sbc-bench scripts.

Just to quickly clarify: sbc-bench doesn't invent new/own benchmarks but just fires up some already existing benchmarks and tools to measure RAM latency/bandwidth and real CPU clockspeeds in a controlled environment to generate insights instead of just random numbers.

What are numbers worth if they're too low since too much background activitiy or throttling happened or CPU frequency capping (RPi with input voltage dropping below ~4.65V) or even CPU cores having been shut down due to overheating?

Which conclusion can be drawn about efficiency of a certain SoC or ARM/RISC-V core when we blindly trust into clockspeeds reported via sysfs when in reality they differ: with some Amlogic SoCs we still have a mismatch of -40% comparing 'advertised' clockspeed vs. real and sbc-bench reports that like this:

Cpufreq OPP: 2016    Measured: 1197 (1197.338/1197.311/1197.121)    (-40.6%)
Cpufreq OPP: 1752    Measured: 1197 (1197.392/1197.338/1197.284)    (-31.7%)
Cpufreq OPP: 1536    Measured: 1197 (1197.365/1197.338/1197.202)    (-22.1%)
Cpufreq OPP: 1416    Measured: 1197 (1197.338/1197.338/1197.202)    (-15.5%)
Cpufreq OPP: 1200    Measured: 1197 (1197.256/1197.229/1197.229)
Cpufreq OPP: 1000    Measured:  997    (997.431/997.290/997.266)
Cpufreq OPP:  667    Measured:  664    (664.061/664.045/664.012)
Cpufreq OPP:  500    Measured:  497    (497.358/497.346/497.346)
Cpufreq OPP:  250    Measured:  246    (246.553/246.268/245.968)     (-1.6%)
Cpufreq OPP:  100    Measured:   96       (96.994/96.592/96.208)     (-4.0%)

Same goes for Raspberries with inappropriate powering and here it's even worse: the main OS (ThreadX on the VC) silently drops ARM clockspeeds to 600 MHz while the Linux kernel running on the ARM core still happily reports to run at 1200/1500/1800 MHz. IMO stuff like this needs to be closely monitored so you're able to throw away results that are invalid (and since benchmarking server/storage systems is part of my day job I can confirm that throwing away results instead of relying on them is the 'norm' or what we mostly do since something went wrong we didn't know upfront so we need to adjust setup/methodology and repeat).

In that sense sbc-bench is more of an environment to execute the actual benchmark (see Geekbench and Phoronix 'piggyback' modes).

As for the few individual benchmarks chosen two are more load generators / stressors (cpuminer, stockfish) and there is actually just one that really generates numbers to compare and that's 7-zip focusing on the stuff I'm personally interested in: integer+memory since me being a server guy. This limited focus is far away from generating a general performance recommendation since based on the use case people are using their SBC they need to pay attention to totally different areas (floating point, crypto acceleration, 2D/3D graphics acceleration, video acceleration, even I/O can be way more important than any of the other areas).

But while 7-zip's focus is pretty limited it has one huge advantage: it generates scores that are persistent accross OS and compiler releases (at least true for the 16.02 version that's shipped with Debian/Ubuntu since years). Most other benchmarks generate numbers that will change once you or your OS's maintainers update OS or compiler version. Sometimes it goes into the opposite direction and with more recent GCC the benchmark scores drop (cpuminer for example on Debian 11).

As such that's the huge advantage of those 7-zip scores even if they represent performance metrics the average SBC user can not make lot use of: they represent 'hardware performance' way more precise in situations where software changed significantly and as such it makes sense to compare scores made in different years (not true for the majority of other passive benchmarking tools where numbers fluctuate over time 'by design').

This (ability to compare/validate results made in different years) is also the reason why ramlat and tinymembench are included since they often can explain why the actual benchmark numbers differ. For example RPi benchmark scores usually get better over the years. Even on the same RPi. And this is not since the hardware magically got faster but since RPi Trading Ltd. employees tweaked ThreadX with better DRAM initialisation/timings and once you apt updateafter a reboot memory access improved.

It's important to know why numbers differ and it's also important to realise when numbers made in different years are still valid or not (and that's something ignored in this SBC world almost everywhere - the basic rules of passive benchmarking simply do not apply since it's a true hobbyist's world)

[TheRemote]

You're exactly right and it's actually worse than you're saying. The results aren't invalidated by years. They can be invalidated by days from how fast these apt packages can change. They can be invalidated on the same day if they're running slightly different operating systems or firmware versions.

Having a lot of tests certainly helps get closer to the truth though. Pi Benchmarks is kind of crowd sourcing that for the storage aspect to the entire community.

On Pi Benchmarks we can actually go back and figure out when they changed gcc and the benchmarks started dropping for example. That's because we have tens of thousands of tests to work with (probably a dozen or so per day on average for years across various boards).

That means it's not really a bug that these scores can change over time when we're crowdsourcing the benchmark. It's a feature. Now we can go back and look really carefully at the dates/scores/system configurations and figure out what changed and why.

You're right though that if you are taking individual benchmarks in a bubble and you are trying to compare them with one another that is a totally invalid/useless comparison. Once we have hundreds or thousands of benchmarks though this turns from a weakness into a strength. Now we have enough data to answer some of those more important questions: what changed, when and why.

You're right about everything you're saying though. Crowdsourcing is a way to turn this from a weakness when comparing two individual results into a strength where you're comparing a result against hundreds or thousands of results across all different times / configurations.

[geerlingguy]

My main goal with this repository is to document my benchmarking results for the times I test these boards for a particular purpose (comparison, a review, a new project)—and almost every time I do a comparison, I will re-benchmark using the latest OS/updates since that does make a huge difference a lot of times.

But the secondary purpose is to put up all my numbers (and any surprising notes, like the RK3588S on the Orange Pi being noticeably faster than the RK3588 on the Rock 5 B, both using the vendor Debian images), so others who benchmark or just have an interest in the boards can either confirm or deny the results, or give any feedback if desired (e.g. if some of my methodology is lacking... which it is, sometimes!).

So I always welcome any and all feedback!

[TheRemote]

Beautiful system, that's a great way to investigate the root causes of some of these issues and differences as well.

I will sometimes do head-to-head testing between different boards and I always do fresh tests every time. Like even though I've ran my same CPU benchmarks on the RPI 4 a dozen times I don't just reuse the old numbers each time. I want to make sure they have all the apt updates and firmware updates available as they would be if a user bought the board and set it up for the first time.

I think it's a great methodology. These are all great tools for us to help diagnose performance related issues. Different tools are appropriate for different jobs as well. I love having lots of tools available and that you shared / linked / documented a bunch of different ways to do it!

[ThomasKaiser]

the RK3588S on the Orange Pi being noticeably faster than the RK3588 on the Rock 5 B

Simple reason: Rockchip's Dynamic Memory Interface (DMC) is 'at fault'. Xunlong hasn't discovered this feature/mechanism yet (so DMC is inactive in their kernel config just as it was with Radxa last July). Radxa in the meantime uses a newer/different kernel branch where DMC is enabled, the governor defaults to dmc_ondemand and upthreshold defaults to 40.

This default results in a drop in idle consumption of about 600mW since DRAM is clocked almost all the time now at just 528 MHz instead of 2112 MHz as before! But the rather conservative upthreshold=40 setting results in the memory controller not ramping up DRAM clocks quickly enough for Linux workloads and also... benchmarks.

Once you adjust this to upthreshold=25 the idle consumption remains at this lower level but benchmarks and also real-world tasks as well perform almost identical as if you would set the DMC governor to performance which then behaves like Xunlong's kernel config but there accompanied by a 600mW higher idle consumption since DRAM is clocked at 2112 MHz all the time (all DMC details).

SoCs like RK3588/RK3588s are that complex that we have to deal with trade-offs at various locations. Though that's not entirely new since DMC and other such governors are there since 2015/RK3399.

The DMC governor's impact on RK3588 benchmarks was the main reason why sbc-bench when executed reports now each governor it detects directly to the tester (not only memory is affected but GPU and NPU too of course and with SoCs not available yet maybe something else):

And the results output lists all information about the DMC governor available (the below is from an OPi 5 running with Armbian. The Armbian guys use Radxa's kernel branch for the OPi 5 so DMC is available/adjustable but of course they don't give a sh*t about settings like uptreshold):

SoC guess: Rockchip RK3588/RK3588s (35881000)
  DMC gov: dmc_ondemand (upthreshold: 40)
DT compat: rockchip,rk3588s-orangepi-5
           rockchip,rk3588

The DRAM clockspeed transition table is also listed when available:

Transitions since last boot (9012920ms ago):

/sys/devices/platform/dmc/devfreq/dmc:

     From  :   To
           : 528000000106800000015600000002112000000   time(ms)
* 528000000:         0         0         0        25   8557290
 1068000000:        20         0         0         3    113906
 1560000000:         0        14         0         0      8916
 2112000000:         6         9        14         0    329386
Total transition : 91

Once you set uptreshold=25 on Rock 5B it will most likely slightly outperform your OPi 5 for a reason called PVTM, another dynamic mechanism implemented by Rockchip in all their recent RK35xx line-up. The exact details you'll also find in my Rock 5B review.

I had the impression within the last months that Rockchip seems to put all the 'strong' silicon into RK3588 while the 'weak' silicon ends up in RK3588s SoCs where the A76 then will clock significantly lower than 2400 MHz. That's at least what I thought to observe after checking a bunch of sbc-bench results since my script of course also collects this information after discovering the PVTM mechanisms in addition to measuring real clockspeeds instead of trusting into sysfs values.

But on the aforementioned OPi 5 the PVTM values show a rather 'strong' silicon variant (allowing the A76 cores to clock at ~2315 MHz):

       cpu cpu0: pvtm=1488
       cpu cpu0: pvtm-volt-sel=4
       cpu cpu4: pvtm=1721
       cpu cpu4: pvtm-volt-sel=5
       cpu cpu6: pvtm=1722
       cpu cpu6: pvtm-volt-sel=5

While this OPi 5 shows weaker PVTM values (it's not only silicon variation but also temperature and Xunlong never thinking about appropriate cooling adds to the problem):

       cpu cpu0: pvtm=1401
       cpu cpu0: pvtm-volt-sel=0
       cpu cpu4: pvtm=1632
       cpu cpu4: pvtm-volt-sel=2
       cpu cpu6: pvtm=1640
       cpu cpu6: pvtm-volt-sel=2

I hope this painfully lenghty explanation backs my point of collecting all 'environmental data' known and available while benchmarking since only those provide the answer for 'Why do the numbers look different?!'

[ThomasKaiser]

Pi Benchmarks is kind of crowd sourcing that for the storage aspect to the entire community.

@TheRemote please correct me if I'm wrong. As I understand it, PiBenchmarks.com is all and only about measuring storage performance on different hosts (regardless of host type) and then presenting this as some sort of listings that vaguely differentiate between 'SBC' and 'PC'?

Nevermind, I read through Storage.shin the meantime. Just one question: is this 'total score' you sort the devices on pibenchmarks.com/ exactly the same calculated in the script at the bottom here: https://github.com/TheRemote/PiBenchmarks/blob/master/Storage.sh#L924-L936