for _constraints: memory requirements vary with available # or cores

[DimStar77]

The _constraints system is quite powerful, but every so often I lack a possibility to request:

• I need MB of RAM multiplied with however many cores you offer me

The rationale is rather simple: having a package that does parallel building will use more memory, the more CPU cores it receives.

Practical example, openSUSE:Factory/libtorrent-rasterbar

It requires 3.5G when building with make -j4
It required 1.5GB when building with make -j2
It required ~ 800M when buildnig with make -j1

the .spec file builds using make %{?_smp_mflags} - so the number of workers is injected by the worker / build script.

In this case if would be great to have a way to specify: I need 800M / core
The build script could then dynamically lower the number of parallel build jobs, based on the available RAM

[AdamMajer]

Inability to specify memory per core is causing build failures with g++, especially in packages that have heavy utilization of templates. And simply adding higher memory requirements tends to drive these workloads onto machines with more cores which doesn't solve the problem.

[rguenth]

Duplicate of #4433

[adrianschroeter]

as said in the other report, I don't want to make the constraint system over complicate, since it is already a bottleneck and the next level of problem would be "I just have one compile job which needs 6GB for this generated gsoap file , the others just need 500MB and I need to configure that...".

The only sane way is to decide this per source in the spec file. OBS can not know anything about the need...

[rguenth]

On Wed, 7 Mar 2018, Adrian Schröter wrote: as said in the other report, I don't want to make the constraint system over complicate, since it is already a bottleneck and the next level of problem would be "I just have one compile job which needs 6GB for this generated gsoap file , the others just need 500MB and I need to configure that...". The only sane way is to decide this per source in the spec file. OBS can not know anything about the need...

Which is why we're trying to tell it ... Why not remove _constraints completely?

[marxin]

I would like to refresh discussion about this.
There are currently packages like MozillaFirefox, chromium, libreoffice that define it's own override of _threads. E.g. on firefox one can see:

[  343s] + _threads=16
[  343s] + _core_memory=1900
[  343s] + echo 'Available memory:'
[  343s] Available memory:
[  343s] + cat /proc/meminfo
[  343s] MemTotal:       11996308 kB
...
[  343s] + echo 'System jobs: 16'
[  343s] System jobs: 16
[  343s] + test 16 -gt 1
[  343s] + mem_per_process=1900
[  343s] ++ awk '/MemTotal/ { print $2 }' /proc/meminfo
[  343s] + max_mem=11996308
[  343s] + max_jobs=6
[  343s] + test 16 -gt 6
[  343s] + _threads=6

So a machine with 16 threads is used with 6 only. That's very bad.
Can you please reconsider physicalmemory removal and introduction of core_memory?
It's really painful that we're wasting OBS workers :(

[AdamMajer]

BuildRequires: memory-constraints

# m: memory limit in MBs per core; default is 1000
%limit_build(m:)

You can't really do anything from the _constraints side since I don't believe OBS sorts, but just filters the list anyway. So you can just request more memory if you want more cores active.

[AdamMajer]

Just for sake of completeness here, output from memory-constraints macro is what @marxin is quoting in the output.

[marxin]

I would like to give this issue a new refresh and a deeper analysis. Few weeks ago I had a brief discussion with @adrianschroeter who believes that most of the package builds are dominated by a single process that consumes most of the memory resources.

In the beginning, I would like to notice that openSUSE:Factory was switched to LTO. And so we typically see something like this (I selected a cvise project for demonstration):

So first we see a parallel compilation (g++) that ends with the emission of LTO bytecode. Then there's a single running ld (yellow color) process that invokes GCC WPA phase (blue color). This phase consumer quite some memory and runs in a single process. The phase ends with WPA-stream-out, a short phase where a bit memory is consumed. And the last phase of linking is so called LTRANS phase, again running in parallel where parallelism is inherited from make server. At the end, ld finalizes binary by consuming quite some memory.

Comments about memory usage:

• peak memory is in the parallel compilation in the first phase
• ld + WPA is a fixed consumption (based on code base size and GCC)
• depending on the number of jobs (-j), LTRANS can require more memory than WPA phase

So for such a package, I would like to express constraints in the following manner:

• total memory at least 4 GB - one needs to fit in WPA and ld
• 1 GB per core - both for parallel running compilation and LTRANS phase

That should explain why both of these are important.

Finally, I'm attaching a graph for Firefox where I don't separate individual LTRANS and compilation processes. The graph looks pretty much the same:

I hope it can convince OBS guys that memory per core is something really useful from a packaging perspective.

[adrianschroeter]

hm, so you have a little bit more memory usage during build, but still the single large one linking at the end.

So, when we add per-core memory limit in the constraints it would need to fullfill both, the per core and the total given memory requirement, right?

[marxin]

So, when we add per-core memory limit in the constraints it would need to fullfill both, the per core and the total given memory requirement, right?

Exactly! Both are useful.

[adrianschroeter]

Just for you #10008

[marxin]

Just for you #10008

Great! Thank you for it !!!

[Vogtinator]

There is one downside with this approach. While previously, packages could build on any worker with the minimum of RAM available by just running less jobs in parallel, they lost the ability to scale to such smaller workers, but instead clog up the queue for the bigger ones. This is especially annoying for ARM, where there aren't many powerful hosts.

[michelmno]

examples of packages that are failing with usage of memoryperjob while not before with %limit_build macro in spec for ppc64:
mozjs60 mozjs68 mame blender (not exhaustive list)

[StefanBruens]

Seems like OBS gets the SMP calculation wrong for PPC64(LE). PPC64 is the only one where both cores and treads are specified as QEMU smp parameters, all other archs just use a single core count value.