dpctl.SyclDevice() and *.filter_string are slow

[AlexanderKalistratov]

import dpctl
import time

dpctl.SyclDevice()

for i in range(10):
    start = time.time()
    dpctl.SyclDevice()
    print(time.time() - start)

Output:

0.0028018951416015625
0.0028688907623291016
0.0030078887939453125
0.0030264854431152344
0.002977132797241211
0.00286102294921875
0.0028085708618164062
0.00516057014465332
0.003843069076538086
0.0028009414672851562

import dpctl
import time

d = dpctl.SyclDevice()

for i in range(10):
    start = time.time()
    d.filter_string
    print(time.time() - start)

Output:

0.0021190643310546875
0.002058744430541992
0.0021059513092041016
0.0021822452545166016
0.0024938583374023438
0.0023038387298583984
0.0023317337036132812
0.0023195743560791016
0.002201080322265625
0.002750396728515625

Since filter string is part of type signature in numba-dpex it is calling SyclDevice and *.filter_string at least once per usm/dpnp array parameter on each function call. This generates significant performance overhead.

Related issue:
IntelPython/numba-dpex#945

[samaid]

I doubt this overhead comes from Python. My gut feeling tells it comes from a driver side. Isn't it?

[AlexanderKalistratov]

@samaid

I doubt this overhead comes from Python. My gut feeling tells it comes from a driver side. Isn't it?

99% sure.
But filter_string could be cached. And device too. We must have device information to do compute follows data in numba. So either we are caching device/filter string on numba side or dpctl do it on it's side

[diptorupd]

@AlexanderKalistratov filter_string has a nice UX and visual aspect, but compute follows data has to be based on SyclQueue. The current filter string based compute follows data implementation in numba-dpex is a temporary workaround till numba_dpex/#930 gets resolved.

There are several issues with basing compute follows data on filter string:

1. Filter string computation is slow because it has an inherent ranking of devices.
2. The same device can have different types of queues(in order or out-order, with or without profiling)
3. filter string does not work for sub-devices
4. Lastly, ranking of devices is a problematic concept in itself. If your system has a CUDA and L0 device what maps to gpu or gpu:0?

[AlexanderKalistratov]

@diptorupd

but compute follows data has to be based on SyclQueue

It is based on SyclQueue. filter_string doesn't affect that.

Filter string computation is slow because it has an inherent ranking of devices.

It could be cached for the device. You don't need to compute it every time. It is from 1 to 3 lines of code for fix.

The same device can have different types of queues(in order or out-order, with or without profiling)

It doesn't have any affect on compilation. Device - does.

filter string does not work for sub-devices

Kernel compiled for device could be run on sub-device. No recompilation needed.

Lastly, ranking of devices is a problematic concept in itself. If your system has a CUDA and L0 device what maps to gpu or gpu:0?

Something dpctl to deal with anyway.

[AlexanderKalistratov]

@diptorupd as IntelPython/numba-dpex#930 resolved you still need to know for which device you are compiling. You need to generate different SPIRV for different devices. Datatype of the result could be different for different devices. So, at the moment of Numba function compilation you must know for which device you are compiling.

I don't see how you are going to overcome these limitations.

[diptorupd]

@AlexanderKalistratov I have all that information from the queue. What is your argument against getting device information from a queue?

[AlexanderKalistratov]

@diptorupd

I have all that information from the queue. What is your argument against getting device information from a queue?

You need to get this information from the queue at compile time. Which means, it should be part of SyclQueue siganture. SyclQueue would be part of DpnpArray, which means this information would be part of DpnpArray signature too.
So, no arguments against getting this information from queue. It simply doesn't change anything.

[diptorupd]

@AlexanderKalistratov I feel you are mixing two different concerns:

a) Compiling a kernel
b) Implementing compute follows data programming model that requires us to figure out what queue should execution take place.

I am talking about b), for the programming model we will not use filter string because of the points I presented above.

As for compiling a kernel for a specific device:

This is the flow:
i. numba-dpex evaluates all array arg types for a kernel and first figures if compute follows data is satisfied.
ii. If compute follows data is satisfied, we get the queue object.
iii. Then we compile the kernel based on the device type and backend that is extracted from the queue.

Does that help clarify?

[diptorupd]

You need to get this information from the queue at compile time. Which means, it should be part of SyclQueue siganture. SyclQueue would be part of DpnpArray, which means this information would be part of DpnpArray signature too.
So, no arguments against getting this information from queue. It simply doesn't change anything.

And I keep telling you that SyclQueue is part of the type information available at static type. Look at https://github.com/IntelPython/numba-dpex/blob/main/numba_dpex/core/types/usm_ndarray_type.py

[AlexanderKalistratov]

And I keep telling you that SyclQueue is part of the type information available at static type. Look at

And I keep telling you that you need to put all the same information stored in filter_string to SyclQueue type. You can reinvent filter_string from the scratch, or just get one from dpctl.

[diptorupd]

And I keep telling you that you need to put all the same information stored in filter_string to SyclQueue type. You can reinvent filter_string from the scratch, or just get one from dpctl.

No, you only keep telling me that filter string is needed. You need to first start by explaining why you need it and why what I have explained till now does not meet the requirement.

Moreover, just to be clear are you fully aware what a filter string is? Can you explain why you need device rank?

[diptorupd]

In any case, I suggest we move this to a discussion topic as the bug report is not really related to the discussion we are having.

[diptorupd]

Let's move the topic to https://github.com/IntelPython/dpctl/discussions/1107

[oleksandr-pavlyk]

I have opened a PR to cache filter_string property.