Environment.ProcessorCount incorrect reporting in containers in 3.1

[amrmahdi]

We are trying to upgrade to .NET Core 3.1. But we that Environment.ProcessorCount reports different values on .NET Core 3.1.

We use the official SDK docker images. I've attached 2 sample projects, 1 for 3.0 and 1 for 3.1

Repro

To repro use the following command to run:

1. docker build . -f Dockerfile 
2. docker run --cpus=1 <image_built_from_#1>

Outcome

3.0

The number of processors on this computer is 1.

3.1

The number of processors on this computer is <actual_node_cpus>.

So if a machine has 8 cores, in 3.0 a container assigned 1 core we still got an outcome of 1, while in 3.1 the outcome is 8.

Is this change by design ?

Were are also seeing much higher cpu consumption on the 3.1 containers, our initial theory that the runtime is thinking it has more cores while it does not.

repro.zip

[stephentoub]

cc: @richlander, @janvorli

[janvorli]

It is by design. The --cpus doesn't limit the number of processors the process can run on. Even with --cpus=1, threads of your process can end up being scheduled on multiple CPUs. See https://docs.docker.com/config/containers/resource_constraints/ for more details. The --cpuset-cpus is the option that sets the specific CPU cores that the container can use.

We used to report the processor count based on the --cpus, but we have realized that in many cases, the Environment.ProcessorCount is used to detect maximum level of parallelism an application can run with and so we have changed it.

[janvorli]

See https://github.com/dotnet/coreclr/issues/26053 for more details.

[amrmahdi]

I see, makes sense. Would this have any impact on the runtime tasks like thread scheduling, GC etc ?

[theolivenbaum]

Just adding my two cents here - got hit hard by this today on updating to netcore3.1 on a 112-core server - where every pod running on this server had CPU limits set on Kubernetes, but now are behaving like they've access to all the 112 cores.

My impression is that decision to change how many cores the runtime considers seems to have been rushed out based on a single-instance benchmark and not on real-world usage of mixed pods in the same server.

Is there any documentation on how to properly set up CPU limits for netcore3.1 under Kubernetes with this change?

[tmds]

In dotnet/coreclr#23398 we discussed this change and did not to merge based on lack of benchmark results.

dotnet/coreclr#26153 then made the change without benchmark results.

[stephentoub]

cc: @VSadov

[jkotas]

benchmark results

We have seen performance results that show improvements both ways. For example, here is a graph that shows significant improvement after the clipping was removed: https://github.com/dotnet/coreclr/issues/22302#issuecomment-519457167.

As @janvorli described, --cpus=1 does not actually limit your app to a single core. It still allows your app to run on all cores, but slower. We have made the decision to remove the clipping based on the documented API behavior.

We may need to introduce new configuration settings or new APIs to address this properly.

[tmds]

CPU quota is the standard parameter of configuring Kubernetes containers.
Before this change the quota got reflected in ProcessorCount (maybe semantically not correct), but it means applications were taking it into account.
With the change, the cpu quota are no longer available and no longer used by .NET Core/.NET Core apps.
The common case of a many containers with low cpu quota on multi-multi-core machine puts your app in a weird configuration (e.g 0.7 quota with ProcessorCount 122).

We may need to introduce new configuration settings or new APIs to address this properly.

+1
The cpu quota should be available in some form.

[mrmartan]

That is not true. CPU quota is OS level construct and .NET process is limited by it as any other process.

There was a discussion to put different variable in https://github.com/dotnet/coreclr/issues/26053#issuecomment-520691729

[mrmartan]

The common case of a many containers with low cpu quota on multi-multi-core machine puts your app in a weird configuration (e.g 0.7 quota with ProcessorCount 122).

I can see that being an issue though