[SYCL] Events caching for in-order queues in the L0 plugin #6643

againull · 2022-08-25T10:17:43Z

For in-order queues we can reuse events even before they are completed and released. Simplified scheme looks like this:
submit command1 [] -> pi_event1 (ze_event1)
submit command2 [dep pi_event1] -> pi_event2 (ze_event2)
submit reset ze_event1
submit command3 [dep pi_event2] -> pi_event3 (ze_event1)
So, in this example same native handle ze_event1 is used for command 1 and command3.

There are several levels of caching supported:

Caching disabled.
Only discarded device-scope events are cached and reused for in-order queues.
All device-scope events are cached and reused for in-order queues.

For 3. external ref count is used to track number of external references. Event can be reused only when this number turns to zero.

E2E tests: intel/llvm-test-suite#1263

sycl/plugins/level_zero/pi_level_zero.hpp

againull · 2022-08-25T16:44:54Z

Fix for SYCL/Plugin/level_zero_device_scope_events.cpp test is here: intel/llvm-test-suite#1179

…ished

…zation for all in-order queues

smaslov-intel · 2022-09-01T13:33:43Z

sycl/plugins/level_zero/pi_level_zero.cpp

+                                       Event->IsDiscarded) &&
+      !Event->hasExternalRefs()) {
+    CandidateForReuse =
+        new _pi_event(Event->ZeEvent, Event->ZeEventPool, Context,


where is this one going to be deleted? why do you need a to allocate a whole new PI event?

where is this one going to be deleted?

Note. I will try to get rid of CandidateForReuse and use only LastCommandEvent per comment below.

pi_event objects get deleted in piEventRelease, we reuse only native handles. pi_event objects are only wrappers which get created and deleted in piEventRelease while ze_event handles are reused.

why do you need a to allocate a whole new PI event?

We need to allocate a whole pi_event when caching in the queue level. pi_event object encapsulates data associated with a particular command (kernel, mapped buffer, list of dependent events, etc), we need to take some actions on this data when the event is completed (release the kernel, free allocated memory, cleanup list of dependent events appropriately).

But as you've summarized in this comment #6643 (comment)
we reuse native event handles before pi_event is completed (for in-order queue we have more info about dependency between commands). We can't reuse whole pi_event object before its completion and we need to re-create pi_event using same native handles.

Example,

kernelA -> pi_event1 {ze_event1, ptr to kernelA, deplist1}

kernelB-> pi_event2 {ze_event2, ptr to kernelB, deplist2}

reset ze_event1 -> at this point pi_event1 is not completed yet, we can't reuse whole pi_event1, because we still going to need info in pi_event1 like event list - deplist1. So we create another object pi_event3 with the same native handle ze_event1.

kernelC-> pi_event3 {ze_event1, ptr to kernelC, deplist3}

As you see pi_event1 and pi_event3 have same native handle - ze_event1 but different data because they are associated with different commands and we need to keep all this info because neither pi_event1 nor pi_event3 is not completed yet and we are going to need this info when they are completed.

sycl/plugins/level_zero/pi_level_zero.hpp

smaslov-intel · 2022-09-01T13:48:38Z

sycl/plugins/level_zero/pi_level_zero.cpp

+  if (DisableEventsCaching)
+    return false;
+
+  bool ProfilingEnabled = (Properties & PI_QUEUE_PROFILING_ENABLE) != 0;


Why can't we reuse events that have no external references to even if queue is with profiling?

I guess we can reuse, I will fix.

* Get rid of unnecessary storage of events. * Make sure there are no leaks of created copies. * Lock event when accessing the data members * Reset event before reusing * Bump external reference count of the proxy event for each event which is externally visible. * Some clarification comments

…ntime

againull · 2022-09-08T06:01:48Z

HIP backend AtomicRef failures are unrelated.

* Don't reuse host-visible events for now - a lot of corner cases which need to be worked on separately * Some cleanup and fix

againull · 2022-09-14T06:54:17Z

Hi @steffenlarsen, @sergey-semenov, could you please review these changes (as far as I know you contributed to L0 plugin) by any chance because Sergey Maslov is in sick leave.

sycl/plugins/level_zero/pi_level_zero.cpp

* Use stoi instead of atoi. * Get rid of piEventReleaseExternal, add comments to other event release functions. * Return queue lock which was removed accidentally.

smaslov-intel · 2022-09-14T19:04:31Z

sycl/doc/EnvironmentVariables.md

@@ -187,6 +187,7 @@ variables in production code.</span>
 | `SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS` | Any(\*) | Enable support of device-scope events whose state is not visible to the host. If enabled mode is SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS=1 the Level Zero plugin would create all events having device-scope only and create proxy host-visible events for them when their status is needed (wait/query) on the host. If enabled mode is SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS=2 the Level Zero plugin would create all events having device-scope and add proxy host-visible event at the end of each command-list submission. The default is 2, meaning only the last event in a batch is host-visible. |
 | `SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS` | Integer | When set to a positive value enables use of Level Zero immediate commandlists, which means there is no batching and all commands are immediately submitted for execution. Default is 0. Note: When immediate commandlist usage is enabled it is necessary to also set SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS to either 0 or 1.  |
 | `SYCL_PI_LEVEL_ZERO_USE_MULTIPLE_COMMANDLIST_BARRIERS` | Integer | When set to a positive value enables use of multiple Level Zero commandlists when submitting barriers. Default is 0. |
+| `SYCL_PI_LEVEL_ZERO_INORDER_QUEUE_REUSE_EVENTS` | Integer | This environment variable controls an optimization for in-order queues which allows to reuse uncompleted Level Zero events in scope of the same queue based on the dependency chain between commands. When set to 1 the plugin will not perform this optimization. When set to 2 the plugin will reuse only explicitely discarded device-scope events for in-order queues. When set to 3 the plugin will try to reuse all device-scope events based on reference counting. Default is 3. |


When set to 1 the plugin will not perform this optimization.

I suggest make 0 to mean no optimization.

Looks like implementation is already such

Yes, thanks for noticing, fixed.

smaslov-intel · 2022-09-14T19:07:46Z

sycl/plugins/level_zero/pi_level_zero.cpp

+// Returns bool value indicating whether queue supports in-order optimization
+// for provided type of event.
+bool _pi_queue::supportsInOrderQueueOptimization(bool HostVisible,
+                                                 bool IsDiscarded) {


There can be multiple different optimization of in-order queues, so maybe call this one more specifically?

Renamed to be more specific.

smaslov-intel · 2022-09-14T22:36:24Z

sycl/plugins/level_zero/pi_level_zero.hpp

+  // Copy of the last command event which is suitable for reuse.
+  // It will be put into the cache when new command is submitted to the in-order
+  // queue.
+  pi_event CandidateForReuse = nullptr;


Suggested change

pi_event CandidateForReuse = nullptr;

pi_event InOrderEventCandidateForReuse = nullptr;

sycl/plugins/level_zero/pi_level_zero.hpp

smaslov-intel · 2022-09-15T00:52:12Z

sycl/plugins/level_zero/pi_level_zero.cpp

+// Controls the level of reusing events for in-order queues.
+static const enum CachingLevel {


Suggested change

// Controls the level of reusing events for in-order queues.

static const enum CachingLevel {

// Controls the level of reusing events for in-order queues.

static const enum InOrderQueueReuseEventsLevel {

steffenlarsen · 2022-09-15T11:34:44Z

sycl/doc/EnvironmentVariables.md

@@ -187,6 +187,7 @@ variables in production code.</span>
 | `SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS` | Any(\*) | Enable support of device-scope events whose state is not visible to the host. If enabled mode is SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS=1 the Level Zero plugin would create all events having device-scope only and create proxy host-visible events for them when their status is needed (wait/query) on the host. If enabled mode is SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS=2 the Level Zero plugin would create all events having device-scope and add proxy host-visible event at the end of each command-list submission. The default is 2, meaning only the last event in a batch is host-visible. |
 | `SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS` | Integer | When set to a positive value enables use of Level Zero immediate commandlists, which means there is no batching and all commands are immediately submitted for execution. Default is 0. Note: When immediate commandlist usage is enabled it is necessary to also set SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS to either 0 or 1.  |
 | `SYCL_PI_LEVEL_ZERO_USE_MULTIPLE_COMMANDLIST_BARRIERS` | Integer | When set to a positive value enables use of multiple Level Zero commandlists when submitting barriers. Default is 0. |
+| `SYCL_PI_LEVEL_ZERO_INORDER_QUEUE_REUSE_EVENTS` | Integer | This environment variable controls an optimization for in-order queues which allows to reuse uncompleted Level Zero events in scope of the same queue based on the dependency chain between commands. When set to 0 the plugin will not perform this optimization. When set to 1 the plugin will reuse only explicitely discarded device-scope events for in-order queues. When set to 2 the plugin will try to reuse all device-scope events based on reference counting. Default is 2. |


Suggested change

| `SYCL_PI_LEVEL_ZERO_INORDER_QUEUE_REUSE_EVENTS` | Integer | This environment variable controls an optimization for in-order queues which allows to reuse uncompleted Level Zero events in scope of the same queue based on the dependency chain between commands. When set to 0 the plugin will not perform this optimization. When set to 1 the plugin will reuse only explicitely discarded device-scope events for in-order queues. When set to 2 the plugin will try to reuse all device-scope events based on reference counting. Default is 2. |

| `SYCL_PI_LEVEL_ZERO_INORDER_QUEUE_REUSE_EVENTS` | Integer | This environment variable controls an optimization for in-order queues which allows the reuse of uncompleted Level Zero events in scope of the same queue based on the dependency chain between commands. When set to 0 the plugin will not perform this optimization. When set to 1 the plugin will reuse only explicitly discarded device-scope events for in-order queues. When set to 2 the plugin will try to reuse all device-scope events based on reference counting. Default is 2. |

[SYCL] Events caching for in-order queues in the L0 plugin

7ce4d4b

againull requested a review from a team as a code owner August 25, 2022 10:17

againull requested a review from smaslov-intel August 25, 2022 10:18

smaslov-intel reviewed Aug 25, 2022

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.hpp Show resolved Hide resolved

smaslov-intel reviewed Aug 25, 2022

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.hpp Show resolved Hide resolved

smaslov-intel reviewed Aug 25, 2022

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.hpp Outdated Show resolved Hide resolved

againull mentioned this pull request Aug 25, 2022

[SYCL] Events caching for in-order queues with discard_events property #6574

Closed

againull added 3 commits August 30, 2022 12:11

Merge remote-tracking branch 'origin/sycl' into advanced_caching_publ…

77bc051

…ished

Add comments describing the optimization

c308a1e

Fix after merge with immediate command list changes and enable optimi…

5c2c3f8

…zation for all in-order queues

smaslov-intel reviewed Sep 1, 2022

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.hpp Outdated Show resolved Hide resolved

smaslov-intel reviewed Sep 1, 2022

View reviewed changes

againull added 2 commits September 7, 2022 21:13

Can't enable all events caching yet because of the issue in the L0 ru…

a4e88ef

…ntime

againull requested a review from smaslov-intel September 8, 2022 04:28

againull added 2 commits September 7, 2022 21:33

Merge remote-tracking branch 'origin/sycl' into advanced_caching

a4192c2

Allow caching when profiling enabled

c83f3ad

againull added 2 commits September 13, 2022 10:40

Merge remote-tracking branch 'origin/sycl' into advanced_caching

88814e6

Updates

1388ba4

* Don't reuse host-visible events for now - a lot of corner cases which need to be worked on separately * Some cleanup and fix

againull requested review from sergey-semenov and steffenlarsen September 14, 2022 06:45

Remove loc

be08d55

steffenlarsen reviewed Sep 14, 2022

View reviewed changes

Address review comments

ce0c80e

* Use stoi instead of atoi. * Get rid of piEventReleaseExternal, add comments to other event release functions. * Return queue lock which was removed accidentally.

againull requested a review from a team as a code owner September 14, 2022 16:35

againull requested a review from steffenlarsen September 14, 2022 16:41

smaslov-intel reviewed Sep 14, 2022

View reviewed changes

Fix env variable values and rename function

3d1484d

againull requested a review from smaslov-intel September 14, 2022 20:13

smaslov-intel reviewed Sep 14, 2022

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.hpp Show resolved Hide resolved

Rename candidate

c4e7b14

againull requested a review from smaslov-intel September 14, 2022 22:44

smaslov-intel reviewed Sep 14, 2022

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.hpp Show resolved Hide resolved

againull requested a review from smaslov-intel September 14, 2022 23:30

smaslov-intel reviewed Sep 15, 2022

View reviewed changes

Renaming

b2610ed

againull requested a review from smaslov-intel September 15, 2022 01:45

steffenlarsen reviewed Sep 15, 2022

View reviewed changes

againull closed this Nov 2, 2022

againull deleted the advanced_caching branch December 3, 2022 00:10

	pi_event CandidateForReuse = nullptr;
	pi_event InOrderEventCandidateForReuse = nullptr;

		// Controls the level of reusing events for in-order queues.
		static const enum CachingLevel {

[SYCL] Events caching for in-order queues in the L0 plugin #6643

[SYCL] Events caching for in-order queues in the L0 plugin #6643

Uh oh!

Conversation

againull commented Aug 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

againull commented Aug 25, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

againull Sep 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

againull commented Sep 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

againull commented Sep 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

againull commented Aug 25, 2022 •

edited

Loading

againull Sep 7, 2022 •

edited

Loading

againull commented Sep 8, 2022 •

edited

Loading

againull commented Sep 14, 2022 •

edited

Loading