UNCACHED memory allocation in Level Zero - Performance

[jjfumero]

This issue is a suggestion based on our findings.

Some of the examples in Level Zero are using the UNCACHED Flag : https://spec.oneapi.io/level-zero/latest/core/api.html#_CPPv438ZE_DEVICE_MEM_ALLOC_FLAG_BIAS_UNCACHED

For example here:
https://github.com/intel/compute-runtime/blob/master/level_zero/core/test/black_box_tests/zello_timestamp.cpp#L102

In TornadoVM we have run a lot of experiments with CACHED and UNCACHED and we saw that the CACHED version is up to 4x faster than the UNCACHED. As I understand, the UNCACHED flag can be used when buffers are streamed once, and not reused, so there is space in GPU's cache for other reusable buffers. Unfortunately, the Level Zero documentation does not warn about this. From our experience, this is very "error prune" since we were analyzing the number of threads and block of threads deployed, rather than how memory was allocated.

Does it make sense to add some documentation in the Level Zero examples to include this information? As well as in which situations developers may use the UNCACHED vs CACHED flag?

BTW, I am not sure if this is the right repo to file this issue or I should also open one in Level Zero.

[jandres742]

hi @jjfumero

we saw that the CACHED version is up to 4x faster than the UNCACHED

Correct. This is expected.

the UNCACHED flag can be used when buffers are streamed once, and not reused, so there is space in GPU's cache for other reusable buffers.

According to the specification, the flag means that the data would be cached or uncached. Currently, that means that when programming a command containing this allocation, and we dont cache those allocations if UNCACHED has been set. Of course, as you say, if we dont cache those buffers, then the cache would have space to cache cacheable buffers, which is expected.

could you elaborate on what documentation you think it is missing?

[jjfumero]

Hi @jandres742,
thanks for the clarification.

I meant perhaps something like this:

diff --git a/level_zero/core/test/black_box_tests/zello_world_gpu.cpp b/level_zero/core/test/black_box_tests/zello_world_gpu.cpp
index c61455610..9b7c6ec37 100644
--- a/level_zero/core/test/black_box_tests/zello_world_gpu.cpp
+++ b/level_zero/core/test/black_box_tests/zello_world_gpu.cpp
@@ -24,6 +24,12 @@ void executeGpuKernelAndValidate(ze_context_handle_t context, ze_device_handle_t
     // Create two shared buffers
     constexpr size_t allocSize = 4096;
     ze_device_mem_alloc_desc_t deviceDesc = {ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC};
+
+    // Set the flag to UNCACHED buffer. Note that this flag is used when streamed once
+    // to the accelerator (the GPU in this case) and data is not reused. This optimizes
+    // the use of cache for other reusable buffers. However, when using the `UNCACHED`
+    // flag, GPU kernels are expected to run up to 4x slower than the CACHED version.
+    // (ZE_DEVICE_MEM_ALLOC_FLAG_BIAS_CACHED)
     deviceDesc.flags = ZE_DEVICE_MEM_ALLOC_FLAG_BIAS_UNCACHED;
     deviceDesc.ordinal = 0;

I noticed that all the tests and examples provided here https://github.com/intel/compute-runtime/tree/master/level_zero/core/test/black_box_tests uses the UNCACHED version without any warning about performance when invoking kernels multiple times. IMO, better documentation will help developers to be aware of the decisions made to set flags in Level Zero (which are a lot).

It would be good as well to have some information about in which situations the UNCACHED version might be used in the Level Zero spec.

If you agree, I can prepare a PR with this documentation.

[jandres742]

thanks @jjfumero . For the bb tests. Yes, we have the UNCACHED there. There hasn't been any special reason behind it, other than maybe we added it in one test and then we just used that test as base when creating new ones. I guess also we hadn't thought about it since the BB tests are meant to show how to use the APIs, not for performance.

Now, we could add the documentation to the spec, indicating that when using UNCACHED you might see performance drop. But, wouldn't be that expected, that when you dont cache your data, and are constantly using it in the GPU, then you have a larger latency since you need to read from a farther level in the memory hierarchy? This is true not only for GPUs, but also for CPU development.