Skip to content

[SYCL][DOC] Update the SYCL Runtime Interface document to give some design updates #680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 22, 2020
Merged

[SYCL][DOC] Update the SYCL Runtime Interface document to give some design updates #680

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b0b8895
[SYCL] Update SYCLPluginInterface.md with further design details.
garimagu Sep 26, 2019
8ed0146
New changes.
garimagu Apr 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 98 additions & 24 deletions sycl/doc/PluginInterface.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
# The DPC++ Runtime Plugin Interface.


## Overview
The DPC++ Runtime Plugin Interface (PI) is the interface layer between
device-agnostic part of the DPC++ runtime and the device-specific runtime layers
The DPC++ Runtime Plugin Interface (PI) is an interface layer between the
device-agnostic part of DPC++ runtime and the device-specific runtime layers
which control execution on devices. It employs the “plugin” mechanism to bind
to the device specific runtime layers similarly to what is used by libomptarget
to the device specific runtime layers similar to what is used by libomptarget
or OpenCL.

The picture below illustrates the placement of the PI within the overall DPC++
The picture below illustrates the placement of PI within the overall DPC++
runtime stack. Dotted lines show components or paths which are not yet available
in the runtime, but are likely to be developed.
![PI in DPC++ runtime architecture](images/RuntimeArchitecture.svg)
Expand All @@ -22,31 +21,85 @@ points implementing the PI interface. The DPC++ runtime collects those function
pointers into a PI interface dispatch table - one per plugin - and uses this
table to dispatch to the device(s) covered by the corresponding plugin.

PI is based on a subset of OpenCL 1.2 runtime specification, it follows its
platform, execution and memory models in all aspects except those explicitly
mentioned in this document. A part of PI API types and functions have exact
PI is based on a subset of OpenCL 1.2 runtime specification, it follows OpenCL's
platform, execution and memory models in all aspects except for those explicitly
mentioned in this document. Some of PI API types and functions have exact
matches in OpenCL. Whenever there is such a match, the semantics also fully
matches unless the differences are explicitly specified in this document. While
match unless the differences are explicitly specified in this document. While
PI has roots in OpenCL, it does have many differences, and the gap is likely
to grow, for example in the areas of memory model and management, program
to grow, for example in areas of memory model and management, program
management.

## Discovery and linkage of PI implementations

![PI implementation discovery](images/PluginDiscovery.svg)

Device discovery phase enumerates all available devices and their features by
querying underlying plugins found in the system. This process is only performed
once before any actual offload is attempted.
querying underlying plugins found in the system. This process is performed when
all attached platforms or devices are queried in an application; for example,
during device selection.

### Plugin discovery

Plugins are physically dynamic libraries stored somewhere in the system where
the DPC++ runtime runs. TBD - design and describe the process in details.
Plugins are physically dynamic libraries or shared objects.
The process to discover plugins follows the following guidelines.

The DPC++ Runtime reads the names of the plugins from a configuration file
at a predetermined location (TBD - Add this location). These plugins are
searched at locations in env LD_LIBRARY_PATH on Linux and env PATH on Windows.
(TBD - Extend to search the plugins at a path relative to the SYCL Runtime
installation directory by using DT_RPATH on Linux. Similar functionality can be
achieved on Windows using SetDllDirectory. This will help avoiding extra setting
of LD_LIBRARY_PATH.)
To avoid any issues with read-only access, an environment variable SYCL_PI_CONFIG
can be set to point to the configuration file which lists the Plugin names. The
enviroment variable if set overrides the predetermined location's config file.
These Plugins are then be searched in LD_LIBRARY_PATH locations.
It is the developer's responsibility to include the plugin names from the
predetermined location's config file to enable discovery of all plugins.
(TBD - Extend to support search in DT_RPATH as above.)
In the current implementation the plugin names are hardcoded in the library.
Configuration file or env SYCL_PI_CONFIG is currently not being considered.

A trace mechanism is provided using env SYCL_PI_TRACE to log the discovery/
binding/ device enumeration process. Different levels of tracing can be achieved
with different values of SYCL_PI_TRACE.
SYCL_PI_TRACE=0x01 provides basic trace of plugins discovered and bound. It also
lists the device selector's selected device information.
SYCL_PI_TRACE=0x02 provides trace of all PI calls made from the DPC++ runtime
with arguments and returned values.
SYCL_PI_TRACE=-1 lists all PI Traces above and more debug messages.

#### Plugin binary interface
TBD - list and describe all the symbols plugin must export in order to be picked
up by the DPC++ runtime for offload.
Plugins should implement all the Interface APIs required for the PI Version
it supports. There is [pi.def](../include/CL/sycl/detail/pi.def)/
[pi.h](../include/CL/sycl/detail/pi.h) file listing all PI API names that can be
called by the specific version of Plugin Interface.
It exports a function - "piPluginInit" that returns the plugin details and function pointer
table containing the list of pointers to implemented Interface Functions defined in pi.h.
In the future, this document will list the minimum set of Interface APIs
to be supported by Plugins. This will also require adding functionality to SYCL
Runtime to work with such limited functionality plugins.

(TBD - list and describe the symbols that a plugin must implement in order to
be picked up by the DPC++ runtime for offload.)

#### Binding a Plugin
The DPC++ Runtime loads all discovered Plugins and tries to bind them by calling
piPluginInit API for each loaded Plugin. The Plugins return the information of
supported PI version and the list of implemented PI API Function pointers.
(TBD - Use the PI API Version information and check for compatibility.
Extend to support version compatibility checks without loading the library.
Eg:Changing the plugin name to reflect the supported Plugin Interface version.)
The information of compatible plugins (with the Function Pointer Table) is
stored in the associated platforms during platform object construction.
The PI API calls are later forwarded using this information.
A plugin is said to "bind" after this process completes with no errors.
During device selection, the user can prefer selection of a device from a
specific Plugin or Backend using the env SYCL_BE. The correspondence between
a plugin and a SYCL_BE value is currently hardcoded in the runtime.
( TBD: Make this a part of configuration file).
Eg: SYCL_BE=PI_OPENCL corresponds to OpenCL Plugin.

#### OpenCL plugin

Expand All @@ -56,12 +109,18 @@ OpenCL implementations. They can be installed either in the standard Khronos
ICD-compatible way (e.g. listed in files under /etc/OpenCL/vendors on
Linux) or not, and the OpenCL plugin can hook up with both.

TBD describe the nested OpenCL implementation discovery process performed by
TBD - implement and describe the nested OpenCL implementation discovery process performed by
the OpenCL plugin

### Device enumeration by plugins
Devices from all bound plugins are queried and listed as and when required, eg: during device selection in device_selector.
The trace shows the PI API calls made when using SYCL_PI_TRACE=-1.
(TBD - Add the trace to list all available devices when plugins are successfully bound.)

### Plugin Unloading
The plugins not chosen to be connected to should be unloaded.

TBD
TBD - Unloading a bound plugin.

## PI API Specification

Expand All @@ -71,11 +130,11 @@ able to operate on the corresponding device. The core API further breaks down
into
- **OpenCL-based** APIs which have OpenCL origin and semantics
- **Extension** APIs which don't have counterparts in the OpenCL
- **Interoperability API** which allows interoperability with underlying APIs
- **Interoperability API** which allows interoperability with underlying runtimes
such as OpenCL.

See [pi.h](../include/CL/sycl/detail/pi.h) header for the full list and
descriptions of PI APIs. [TBD: link to pi.h doxygen here]
descriptions of PI APIs.

### The Core OpenCL-based PI APIs

Expand Down Expand Up @@ -114,10 +173,10 @@ in a data section.
### The Interoperability PI APIs

These are APIs needed to implement DPC++ runtime interoperability with
underlying "native" device runtimes such as OpenCL. Currently there are only
OpenCL interoperability APIs, which is to be implemented by the OpenCL PI
plugin only. These APIs match semantics of the corresponding OpenCL APIs
exactly.
underlying "native" device runtimes such as OpenCL.
There are some OpenCL interoperability APIs, which are to be implemented
by the OpenCL PI plugin only. These APIs match semantics of the corresponding
OpenCL APIs exactly.
For example:

```
Expand All @@ -129,6 +188,21 @@ pi_result piclProgramCreateWithSource(
pi_program * ret_program);
```

Some interoperability extension APIs have been added to get native runtime
handles from the backend-agnostic PI Objects or to create PI Objects using the
native handles. Eg:

```
pi_result piextDeviceGetNativeHandle(
pi_device device,
pi_native_handle *nativeHandle);

pi_result piextDeviceCreateWithNativeHandle(
pi_native_handle nativeHandle,
pi_device *device);

```

### PI Extension mechanism

TBD This section describes a mechanism for DPC++ or other runtimes to detect
Expand Down