[OpenCL] Add support for cslice partitioning #7963

asudarsa · 2023-01-09T16:57:17Z

Recently, an extension was added here: #7513
This extension allows partitioning a device by "cslice" (aka CCS-es).
With this change, on PVC sub-sub-devices now require info::partition_property::ext_intel_partition_by_cslice instead of info::partition_property::partition_by_affinity_domain that wasn't quite accurately describing the actual scheme.
Level Zero backend support was added here: #7626
In this change, we are adding support in OpenCL plugin.

This change makes use of the cl_intel_command_queue_families recently added.

Thanks
Signed-off-by: Arvind Sudarsanam arvind.sudarsanam@intel.com

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

asudarsa · 2023-01-09T16:57:51Z

Marking it as a draft to allow for initial review.

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

aelovikov-intel · 2023-01-09T17:32:24Z

Do you have a test for this already?

aelovikov-intel · 2023-01-09T17:34:44Z

sycl/plugins/opencl/pi_opencl.cpp

+  cl_device_id parentId = nullptr;
+  clGetDeviceInfo(cast<cl_device_id>(device), CL_DEVICE_PARENT_DEVICE,
+                  sizeof(cl_device_id), &parentId, NULL);
+  if (parentId == nullptr)
+    return 0;
+  cl_device_id parentParentId = nullptr;
+  clGetDeviceInfo(parentId, CL_DEVICE_PARENT_DEVICE, sizeof(cl_device_id),
+                  &parentParentId, NULL);
+  if (parentParentId == nullptr)
+    return 1;
+  cl_device_id parentParentParentId = nullptr;
+  clGetDeviceInfo(parentParentId, CL_DEVICE_PARENT_DEVICE, sizeof(cl_device_id),
+                  &parentParentParentId, NULL);
+  if (parentParentParentId == nullptr)
+    return 2;
+  return -1;


nit: I'd rather see a generic implementation (i.e., while loop) with an assert that the result is <=2.

I changed the levels to use an enum. So, I am leaving code as is (no while loop). Hope that's ok. Thanks

aelovikov-intel · 2023-01-09T17:36:37Z

sycl/plugins/opencl/pi_opencl.cpp

+    if (device->subLevel == -1)
+      device->subLevel = getSubLevel(device);


This makes little sense to me. Why don't we report an error immediately for the invalid level? Also, I've realized that I'd like to see a named const/constexpr variable for "-1".

Here, -1 (or INVALID in the upcoming patch) means that we have not yet queried the backend for the device level info. We will make a call to 'getLevel' only once.

aelovikov-intel · 2023-01-09T17:37:52Z

sycl/plugins/opencl/pi_opencl.cpp

+      clGetDeviceInfo(
+          cast<cl_device_id>(device), CL_DEVICE_PARTITION_MAX_SUB_DEVICES,
+          sizeof(partitionMaxSubDevices), &partitionMaxSubDevices, nullptr);
+    } else if (device->isSubDevice()) {


What else can it be if it's not a root device (false at line 393)?

It can be a sub-device or sub-sub device created using calls to CreateSubDevice by user

aelovikov-intel · 2023-01-09T17:38:32Z

sycl/plugins/opencl/pi_opencl.cpp

+          cast<cl_device_id>(device), CL_DEVICE_PARTITION_MAX_SUB_DEVICES,
+          sizeof(partitionMaxSubDevices), &partitionMaxSubDevices, nullptr);
+    } else if (device->isSubDevice()) {
+      // find out number of CCSes


What if it's already a sub-sub-device ?

Then it will just return 0 in out_num_devices

I think I'd prefer to be more explicit about this. Something like

switch (level) { case 2: return no_further_partitioning case 0: return delegate_to_opencl case 1: { our logic } default: assert/PI_INVALID_ERROR/etc. }

Modified in upcoming patch. Thanks

aelovikov-intel · 2023-01-09T17:39:26Z

sycl/plugins/opencl/pi_opencl.cpp

+      device->subLevel = getSubLevel(device);
+    if (device->isRootDevice()) {
+      clGetDeviceInfo(
+          cast<cl_device_id>(device), CL_DEVICE_PARTITION_MAX_SUB_DEVICES,


cast<cl_device_id>(device) - I'm confused here. It looks like the handle is from the OpenCL RT and is opaque for us, yet we do device->subLevel above.

This handle (device) is a 'pi_device' handle. This way of passing the pi_device into OpenCL APIs seems like a standard way. I am not too sure about the 'internals' of OpenCL runtime. @bashbaug, can you please comment? Thanks

aelovikov-intel · 2023-01-09T17:49:29Z

sycl/plugins/opencl/pi_opencl.cpp

+      clGetDeviceInfo(
+          cast<cl_device_id>(device), CL_DEVICE_QUEUE_FAMILY_PROPERTIES_INTEL,
+          3*sizeof(cl_queue_family_properties_intel), qfprops, &qsize);
+      qsize = qsize/sizeof(cl_queue_family_properties_intel);


Not clang-formated, I think.

I ran clang-format before submission. Which line do you think may be misformatted? Thanks

Maybe it's specific clang-format configuration for the OpenCL plugin. The rest of the codebase would have spaces around / and no spaces after ( and before ) in the for loop below.

I made a manual change for these. Not sure how I can catch them otherwise. Thanks

aelovikov-intel · 2023-01-09T17:49:41Z

sycl/plugins/opencl/pi_opencl.cpp

+          cast<cl_device_id>(device), CL_DEVICE_QUEUE_FAMILY_PROPERTIES_INTEL,
+          3*sizeof(cl_queue_family_properties_intel), qfprops, &qsize);
+      qsize = qsize/sizeof(cl_queue_family_properties_intel);
+      for ( size_t q = 0; q < qsize; q++ ) {


I ran clang-format before submission. Which line do you think may be misformatted? Thanks

aelovikov-intel · 2023-01-09T17:50:32Z

sycl/plugins/opencl/pi_opencl.cpp

+        }
+      }
+    } else {


Any chance we can employ early return above to reduce this nesting?

Will take a look. Thanks

I think changing to switch/case reduced the nesting level. Thanks

aelovikov-intel · 2023-01-09T17:51:28Z

sycl/plugins/opencl/pi_opencl.cpp

+        return ReturnValue(pi_device_partition_property{0});
+      }
+      return ReturnHelper(PI_DEVICE_PARTITION_BY_AFFINITY_DOMAIN);
+    } else if (device->isSubDevice()) {


No need for else as the above if early-returns.

Will change. Thanks

aelovikov-intel · 2023-01-09T17:55:09Z

sycl/plugins/opencl/pi_opencl.cpp

+      }
+      for (uint32_t i = 0; i < *out_num_devices; ++i) {


Why can't it be a single loop?

I think it can be. Just an incremental add I think. I will fuse them.

Fused. Thanks

aelovikov-intel

Sorry, I misclicked in the previous review accidentally selecting "approved". Hopefully this will reset it.

asudarsa · 2023-01-09T19:03:05Z

Sorry, I misclicked in the previous review accidentally selecting "approved". Hopefully this will reset it.

Got it. Thanks so much for the feedback Andrei. Will make changes soon.

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

asudarsa · 2023-01-10T00:16:45Z

I will add a test very soon. Thanks

aelovikov-intel · 2023-01-09T21:36:39Z

sycl/plugins/opencl/pi_opencl.cpp

+    pi_uint32 numSubDevices = 0;
+    if (device->level == _pi_device::INVALID) // level not yet updated
+      device->level = getLevel(device);


Why is this lazy? Can we update when device is created?

We actually do update this when device is created. So, getLevel will mostly not be called here. I am just using this as a fail-safe. But, thinking of it, I am not able to formulate a scenario where device->level can be invalid once it has been created. I can take out this code if needed. Thanks

I'd rather have an assert here then. The laziness complicates the code unnecessarily.

aelovikov-intel · 2023-01-10T00:34:46Z

sycl/plugins/opencl/pi_opencl.cpp

+    if (device->level == _pi_device::INVALID) // level not updated yet
+      device->level = getLevel(device);
+    // For root-device there is no partitioning to report.
+    if (device->isRootDevice())


IMO, we should either drop lines 468/469 (getLevel) or convert ifs into a switch.

Ah..good catch. I will drop lines 468/469.

aelovikov-intel · 2023-01-10T00:35:32Z

sycl/plugins/opencl/pi_opencl.cpp

+        return ReturnHelper(PI_DEVICE_PARTITION_BY_AFFINITY_DOMAIN);
+      }
+      case _pi_device::SUBDEVICE: {
+        // find out number of CCSes


Suggested change

// find out number of CCSes

// Find out number of CCSes.

Will be done in next patch. Thanks

aelovikov-intel · 2023-01-10T00:35:54Z

sycl/plugins/opencl/pi_opencl.cpp

+          std::cout
+              << "This device does not support cl_intel_command_queue_families"
+              << std::endl;


Seems like those debug prints need to be removed/guarded by debug macros.

aelovikov-intel · 2023-01-10T00:38:24Z

sycl/plugins/opencl/pi_opencl.cpp

+        qsize = qsize / sizeof(cl_queue_family_properties_intel);
+        for (size_t q = 0; q < qsize; q++) {
+          if (qfprops[q].capabilities == CL_QUEUE_DEFAULT_CAPABILITIES_INTEL &&
+              qfprops[q].count > numSubDevices) {


What does this condition mean? Can we have numSubDevices not equal to zero?

This is covering a scenario where multiple families have 'DEFAULT' capabilities and we are trying to get the handle to the family with maximum number of dub-devices.

aelovikov-intel · 2023-01-10T00:55:59Z

sycl/plugins/opencl/pi_opencl.cpp

+  }
+  // Absorb the CL_DEVICE_NOT_FOUND and just return 0 in out_num_devices
+  if (result == CL_DEVICE_NOT_FOUND) {
+    std::cout << "Device not found\n";


More debug prints that need a cleanup.

Sorry. removed it. Thanks

aelovikov-intel · 2023-01-10T00:56:48Z

sycl/plugins/opencl/pi_opencl.cpp

@@ -381,6 +583,9 @@ pi_result piDevicesGet(pi_platform platform, pi_device_type device_type,
      cast<cl_uint>(num_entries), cast<cl_device_id *>(devices),
      cast<cl_uint *>(num_devices));

+  for (pi_uint32 i = 0; i < num_entries; ++i) {
+    devices[i]->level = _pi_device::ROOTDEVICE;


With this in place I'm really surprised we need lazy initialization for this ->level in the code earlier in this PR.

I agree. However, earlier in the code, we will actually not call the getLevel function as we are already setting the level here.

I have actually removed getLevel function. We can populate the level info at the time of device creation.

aelovikov-intel · 2023-01-10T00:57:58Z

sycl/plugins/opencl/pi_opencl.hpp

+// A single-threaded app has an opportunity to enable this mode to avoid
+// overhead from mutex locking. Default value is 0 which means that single
+// thread mode is disabled.
+static const bool SingleThreadMode = [] {
+  const char *Ret = std::getenv("SYCL_PI_OPENCL_SINGLE_THREAD_MODE");
+  const bool RetVal = Ret ? std::stoi(Ret) : 0;
+  return RetVal;
+}();
+
+// Class which acts like shared_mutex if SingleThreadMode variable is not set.
+// If SingleThreadMode variable is set then mutex operations are turned into
+// nop.
+class pi_shared_mutex_ocl {
+  std::shared_mutex Mutex;
+


Is this brand new or copy-pasted from level zero plugin?

Removed the copy-paste code and added the header file from unified runtime.

aelovikov-intel · 2023-01-10T00:59:30Z

sycl/plugins/opencl/pi_opencl.hpp

+    ROOTDEVICE = 0,
+    SUBDEVICE = 1,
+    SUBSUBDEVICE = 2,


I think we can do some magic/arithmetic inside getLevel in the code at the top of the PR's diff and then just cast it back to the enum. Not insisting though.

I suppose you are referring to nextLevel? I am not sure how getLevel can be improved.

Both, actually. We can calculate the depth of the device hierarchy in a loop and then cast it to this enum.

I would advise to keep the code as is for better readability. Hope that's ok.

aelovikov-intel · 2023-01-10T01:00:09Z

sycl/plugins/opencl/pi_opencl.hpp

+  _pi_device(pi_platform Plt) : Platform{Plt} {
+    level = INVALID;
+    family = index = 0;
+    // NOTE: one must additionally call initialize() to complete


I don't see _pi_device::initialize here.

Ah..the comment needs to be deleted.

aelovikov-intel · 2023-01-10T01:02:04Z

I'm not sure how I feel about directly copy-pasting level zero plugin's infrastructure here. Adding @smaslov-intel for his thoughts.

asudarsa · 2023-01-19T23:52:56Z

sycl/plugins/opencl/pi_utils.hpp

@@ -0,0 +1,79 @@
+//===--------- pi_utils.hpp - Plugin Utility Functions -------------------===//


pi_utils.hpp will be removed. This code is available from unified runtime. Thanks.

aelovikov-intel · 2023-01-20T00:15:27Z

sycl/plugins/opencl/pi_opencl.hpp

+  pi_uint32 index;  // SYCL queue index inside a given family of queues
+
+  bool isRootDevice(void) { return level == ROOTDEVICE; }
+  bool isSubDevice(void) { return level == SUBDEVICE; }


It's been more than a week since my review, so I've forgotten some amount of context, but looking into the conversation at #7963 (comment) I'm realizing that I don't particularly like this interface.

sub-sub-device->isSubDevice is ambiguous in its meaning. It is a "sub-device" of a "sub-device", after all.

Ah. I agree. I was not able to think of any clearer/easier way to represent this. May be a comment can help?

changed isSubSubDevice to isCCS.

asudarsa · 2023-01-20T02:49:59Z

Do you have a test for this already?

Just added for review.

asudarsa · 2023-01-20T02:50:15Z

/verify with intel/llvm-test-suite#1530

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

sycl/plugins/opencl/pi_opencl.cpp

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

asudarsa · 2023-02-01T14:49:49Z

sycl/plugins/opencl/pi_opencl.cpp

@@ -1747,8 +2116,20 @@ pi_result piextKernelGetNativeHandle(pi_kernel kernel,
 // This API is called by Sycl RT to notify the end of the plugin lifetime.
 // TODO: add a global variable lifetime management code here (see
 // pi_level_zero.cpp for reference) Currently this is just a NOOP.
+// We clear all the 'map' variables here.


I added the code to clean up all the maps that we are populating in the plugin. I also delete the pi_device handles in the cslice_devices list if they have not been deleted yet.
Thanks

Ah..the code cleanup needs to be done in a different way. I will address this in the next PR. Thanks

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

bashbaug

There are additional enhancements we should make but I'm fine adding them in a subsequent PR. Thanks!

aelovikov-intel · 2023-02-03T00:12:36Z

sycl/plugins/opencl/pi_opencl.cpp

+                          paramValueSize, paramValue, paramValueSizeRet);
+      return static_cast<pi_result>(result);
+    } else {
+      // For root-device there is no partitioning to report.


I don't see much similarities between the plugins in this code actually, but if Ben is fine with postponing the cleanup, so am I.

aelovikov-intel · 2023-02-03T00:16:18Z

sycl/plugins/opencl/pi_opencl.cpp

@@ -815,11 +1087,16 @@ pi_result piContextCreate(const pi_context_properties *properties,
                                             size_t cb, void *user_data1),
                          void *user_data, pi_context *retcontext) {
  pi_result ret = PI_ERROR_INVALID_OPERATION;
+  std::vector<cl_device_id> cl_devices = getClDevices(num_devices, devices);


Is it ok to have duplicates here? Although I think the helper itself has to be changed, if it's not ok.

aelovikov-intel · 2023-02-03T00:18:01Z

sycl/plugins/opencl/pi_opencl.cpp

+    std::vector<pi_device> device_list_vec(num_devices);
+    for (size_t i = 0; i < num_devices; ++i)
+      device_list_vec[i] = devices[i];


nit: I think

template< class InputIt > vector( InputIt first, InputIt last, const Allocator& alloc = Allocator() );

ctor should work just fine here.

aelovikov-intel · 2023-02-03T00:18:16Z

sycl/plugins/opencl/pi_opencl.cpp

+  std::vector<pi_device> device_list_vec(num_devices);
+  for (size_t i = 0; i < num_devices; ++i)
+    device_list_vec[i] = devices[i];


nit: Likewise.

aelovikov-intel · 2023-02-03T00:19:39Z

sycl/plugins/opencl/pi_opencl.cpp

+    if (context2devlist.find(context) != context2devlist.end()) {
+      auto devlist = context2devlist[context];


nit

Suggested change

if (context2devlist.find(context) != context2devlist.end()) {

auto devlist = context2devlist[context];

if (auto it = context2devlist.find(context); it != context2devlist.end()) {

auto devlist = *it;

asudarsa · 2023-02-07T16:33:49Z

Hi @againull

Can you please help to merge this? latest comments will be addressed in an upcoming PR.

Thanks

sycl/plugins/opencl/pi_opencl.hpp

dm-vodopyanov · 2023-02-08T15:43:08Z

@asudarsa this patch fails post-commit on Windows and macOS. Can you please fix this ASAP?

https://github.com/intel/llvm/actions/runs/4116506385/jobs/7106640407
https://github.com/intel/llvm/actions/runs/4116506385/jobs/7106643190

@bader FYI, revert might be needed in case these failures came from unified runtime - to return post-commit to (partially) green state as soon as possible.

This reverts commit dcbd6e5.

asudarsa · 2023-02-08T17:51:15Z

@asudarsa this patch fails post-commit on Windows and macOS. Can you please fix this ASAP?

https://github.com/intel/llvm/actions/runs/4116506385/jobs/7106640407 https://github.com/intel/llvm/actions/runs/4116506385/jobs/7106643190

@bader FYI, revert might be needed in case these failures came from unified runtime - to return post-commit to (partially) green state as soon as possible.

Hi Dmitry,

Some of the fails seem unrelated to my change. The only issue that seems to be related is 'zer_api.h' not being found. I am not sure why that issue does not occur in Linux. Can you please verify, if possible, if the post-commit builds and tests pass without my change, before actually reverting it?

Thanks

bader · 2023-02-08T18:16:18Z

@asudarsa, these post-commit steps were clean before your commit and stably fail after it. I also looked at the error message and it points to the code modified by this patch.
I'm 100% sure that fails are caused by this patch.

bader · 2023-02-09T01:05:57Z

@asudarsa, do you have any ETA for resolving post-commit test issues?
@dm-vodopyanov created #8251 to revert this change and @cperkinsintel approved it. We can revert it and you can push fixed version of the patch later, but I prefer to push the fix if it's ready.

cperkinsintel · 2023-02-09T01:09:02Z

This PR is also resulting in a failure in llvm-test-suite/SYCL/Plugin/interop-opencl.cpp in the ACC/FPGA case. Our GHA testing doesn't include an ACC case so this was missed.

Running llvm-test-suite would have detected both that failure and the windows building one.

bader · 2023-02-09T02:40:11Z

Okay, I think we should revert the change and do more testing locally.
@dm-vodopyanov, @cperkinsintel, thanks for helping with this.

This reverts commit dcbd6e5.

[OpenCL] Add support for cslice partitioning

2c31fa4

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

asudarsa requested a review from a team as a code owner January 9, 2023 16:57

asudarsa requested a review from aelovikov-intel January 9, 2023 16:57

asudarsa marked this pull request as draft January 9, 2023 16:57

asudarsa requested a review from bashbaug January 9, 2023 16:58

asudarsa temporarily deployed to aws January 9, 2023 17:05 — with GitHub Actions Inactive

Fix type issue

3ba4ade

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

aelovikov-intel approved these changes Jan 9, 2023

View reviewed changes

aelovikov-intel requested changes Jan 9, 2023

View reviewed changes

asudarsa temporarily deployed to aws January 9, 2023 18:36 — with GitHub Actions Inactive

asudarsa added 2 commits January 9, 2023 11:55

Fix test fail

f7ce8f6

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

Addressing review comments

9600fe6

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

asudarsa requested a review from aelovikov-intel January 10, 2023 00:17

aelovikov-intel reviewed Jan 10, 2023

View reviewed changes

aelovikov-intel requested a review from smaslov-intel January 10, 2023 01:01

asudarsa temporarily deployed to aws January 10, 2023 01:33 — with GitHub Actions Inactive

asudarsa temporarily deployed to aws January 10, 2023 03:49 — with GitHub Actions Inactive

asudarsa commented Jan 19, 2023

View reviewed changes

aelovikov-intel reviewed Jan 20, 2023

View reviewed changes

asudarsa mentioned this pull request Jan 20, 2023

[PI][OpenCL] Test support for exposing compute slices intel/llvm-test-suite#1530

Open

asudarsa added 2 commits January 19, 2023 19:04

Address review comments

59da75e

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

Minor format issue

59f5741

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

asudarsa requested a review from aelovikov-intel January 20, 2023 03:07

asudarsa temporarily deployed to aws January 31, 2023 18:15 — with GitHub Actions Inactive

aelovikov-intel reviewed Jan 31, 2023

View reviewed changes

sycl/plugins/opencl/pi_opencl.cpp Show resolved Hide resolved

sycl/plugins/opencl/pi_opencl.cpp Show resolved Hide resolved

sycl/plugins/opencl/pi_opencl.cpp Show resolved Hide resolved

asudarsa temporarily deployed to aws February 1, 2023 02:22 — with GitHub Actions Inactive

asudarsa temporarily deployed to aws February 1, 2023 02:54 — with GitHub Actions Inactive

Address more review comments

ad3d1da

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

asudarsa commented Feb 1, 2023

View reviewed changes

asudarsa temporarily deployed to aws February 1, 2023 14:59 — with GitHub Actions Inactive

Address Werror issues

daf8f73

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

asudarsa temporarily deployed to aws February 1, 2023 16:07 — with GitHub Actions Inactive

asudarsa temporarily deployed to aws February 1, 2023 17:27 — with GitHub Actions Inactive

Address Werror issues

142fa31

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

asudarsa temporarily deployed to aws February 1, 2023 18:41 — with GitHub Actions Inactive

asudarsa temporarily deployed to aws February 1, 2023 19:19 — with GitHub Actions Inactive

asudarsa requested review from aelovikov-intel and bashbaug February 2, 2023 16:26

bashbaug approved these changes Feb 2, 2023

View reviewed changes

aelovikov-intel approved these changes Feb 3, 2023

View reviewed changes

againull merged commit dcbd6e5 into intel:sycl Feb 7, 2023

smaslov-intel reviewed Feb 7, 2023

View reviewed changes

sycl/plugins/opencl/pi_opencl.hpp Show resolved Hide resolved

dm-vodopyanov added a commit that referenced this pull request Feb 8, 2023

Revert "[OpenCL] Add support for cslice partitioning (#7963)"

af6a71d

This reverts commit dcbd6e5.

bader pushed a commit that referenced this pull request Feb 9, 2023

Revert "[OpenCL] Add support for cslice partitioning (#7963)" (#8251)

fa8ce20

This reverts commit dcbd6e5.

		if (device->subLevel == -1)
		device->subLevel = getSubLevel(device);

		@@ -0,0 +1,79 @@
		//===--------- pi_utils.hpp - Plugin Utility Functions -------------------===//

		if (context2devlist.find(context) != context2devlist.end()) {
		auto devlist = context2devlist[context];

[OpenCL] Add support for cslice partitioning #7963

[OpenCL] Add support for cslice partitioning #7963

Uh oh!

Conversation

asudarsa commented Jan 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asudarsa commented Jan 9, 2023

Uh oh!

aelovikov-intel commented Jan 9, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asudarsa Jan 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aelovikov-intel left a comment

Choose a reason for hiding this comment

Uh oh!

asudarsa commented Jan 9, 2023

Uh oh!

asudarsa commented Jan 10, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asudarsa commented Jan 9, 2023 •

edited

Loading

asudarsa Jan 9, 2023 •

edited

Loading