Skip to content

[CI] Update drivers installation process into containers #7730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Dec 15, 2022
Merged

[CI] Update drivers installation process into containers #7730

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3b662af
[CI] Install level_zero runtime as part of the drivers installation
Dec 10, 2022
7237928
Fix issues
Dec 13, 2022
697d74c
Fix more issues
Dec 13, 2022
b4f91bd
Minor formatting
Dec 13, 2022
a3fee45
Update update_drivers.py with level_zero
Dec 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 8 additions & 14 deletions .github/workflows/sycl_containers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ jobs:
build-args: |
compute_runtime_tag=${{fromJson(steps.deps.outputs.deps).linux.compute_runtime.github_tag}}
igc_tag=${{fromJson(steps.deps.outputs.deps).linux.igc.github_tag}}
cm_tag=${{fromJson(steps.deps.outputs.deps).linux.cm.github_tag}}
level_zero_tag=${{fromJson(steps.deps.outputs.deps).linux.level_zero.github_tag}}
tbb_tag=${{fromJson(steps.deps.outputs.deps).linux.tbb.github_tag}}
fpgaemu_tag=${{fromJson(steps.deps.outputs.deps).linux.fpgaemu.github_tag}}
cpu_tag=${{fromJson(steps.deps.outputs.deps).linux.oclcpu.github_tag}}
Expand Down Expand Up @@ -128,12 +130,8 @@ jobs:
tags: |
ghcr.io/${{ github.repository }}/ubuntu2004_intel_drivers:unstable-${{ github.sha }}
ghcr.io/${{ github.repository }}/ubuntu2004_intel_drivers:unstable
build-args: |
compute_runtime_tag=${{fromJson(steps.deps.outputs.deps).linux_staging.compute_runtime.github_tag}}
igc_tag=${{fromJson(steps.deps.outputs.deps).linux_staging.igc.github_tag}}
tbb_tag=${{fromJson(steps.deps.outputs.deps).linux_staging.tbb.github_tag}}
fpgaemu_tag=${{fromJson(steps.deps.outputs.deps).linux_staging.fpgaemu.github_tag}}
cpu_tag=${{fromJson(steps.deps.outputs.deps).linux_staging.oclcpu.github_tag}}
# build-args:empty , so we automatically take the "latest" tags as
# unstable

base_image_ubuntu2204:
if: github.repository == 'intel/llvm'
Expand Down Expand Up @@ -207,10 +205,11 @@ jobs:
build-args: |
compute_runtime_tag=${{fromJson(steps.deps.outputs.deps).linux.compute_runtime.github_tag}}
igc_tag=${{fromJson(steps.deps.outputs.deps).linux.igc.github_tag}}
cm_tag=${{fromJson(steps.deps.outputs.deps).linux.cm.github_tag}}
level_zero_tag=${{fromJson(steps.deps.outputs.deps).linux.level_zero.github_tag}}
tbb_tag=${{fromJson(steps.deps.outputs.deps).linux.tbb.github_tag}}
fpgaemu_tag=${{fromJson(steps.deps.outputs.deps).linux.fpgaemu.github_tag}}
cpu_tag=${{fromJson(steps.deps.outputs.deps).linux.oclcpu.github_tag}}

# This job produces a Docker container with the latest versions of Intel
# drivers, that can be found on GitHub.
drivers_image_ubuntu2204_unstable:
Expand Down Expand Up @@ -241,10 +240,5 @@ jobs:
tags: |
ghcr.io/${{ github.repository }}/ubuntu2204_intel_drivers:unstable-${{ github.sha }}
ghcr.io/${{ github.repository }}/ubuntu2204_intel_drivers:unstable
build-args: |
compute_runtime_tag=${{fromJson(steps.deps.outputs.deps).linux_staging.compute_runtime.github_tag}}
igc_tag=${{fromJson(steps.deps.outputs.deps).linux_staging.igc.github_tag}}
tbb_tag=${{fromJson(steps.deps.outputs.deps).linux_staging.tbb.github_tag}}
fpgaemu_tag=${{fromJson(steps.deps.outputs.deps).linux_staging.fpgaemu.github_tag}}
cpu_tag=${{fromJson(steps.deps.outputs.deps).linux_staging.oclcpu.github_tag}}

# build-args:empty, so we automatically take the "latest" tags as
# unstable
1 change: 1 addition & 0 deletions devops/containers/ubuntu2004_intel_drivers.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ ENV DEBIAN_FRONTEND=noninteractive
ARG compute_runtime_tag=latest
ARG igc_tag=latest
ARG cm_tag=latest
ARG level_zero_tag=latest
ARG tbb_tag=latest
ARG fpgaemu_tag=latest
ARG cpu_tag=latest
Expand Down
1 change: 1 addition & 0 deletions devops/containers/ubuntu2204_intel_drivers.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ ENV DEBIAN_FRONTEND=noninteractive
ARG compute_runtime_tag=latest
ARG igc_tag=latest
ARG cm_tag=latest
ARG level_zero_tag=latest
ARG tbb_tag=latest
ARG fpgaemu_tag=latest
ARG cpu_tag=latest
Expand Down
49 changes: 7 additions & 42 deletions devops/dependencies.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@
"url": "https://github.com/intel/cm-compiler/releases/tag/cmclang-1.0.144",
"root": "{DEPS_ROOT}/opencl/runtime/linux/oclgpu"
},
"level_zero": {
"github_tag": "v1.8.8",
"version": "v1.8.8",
"url": "https://github.com/oneapi-src/level-zero/releases/tag/v1.8.8",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This URL points to Level Zero loader and I think the runtime is installed as part of compute-runtime package. Isn't loader installed with compute-runtime as well?
If not, I'm okay with installing the loader, but PR title should be corrected.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loader is not installed as part of the compute-runtime.
Here is content for the record. Here is what is installed with compute-runtime package:

dpkg-query -L intel-level-zero-gpu
/.                                                        
/usr                                                      
/usr/include                                              
/usr/lib                                                  
/usr/lib/x86_64-linux-gnu                                 
/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.24278    
/usr/share                                                
/usr/share/doc                                            
/usr/share/doc/intel-level-zero-gpu                       
/usr/share/doc/intel-level-zero-gpu/changelog.gz          
/usr/share/doc/intel-level-zero-gpu/copyright             
/var                                                      
/var/lib                                                  
/var/lib/libze_intel_gpu                                  
/var/lib/libze_intel_gpu/pci_bind_status_file             
/var/lib/libze_intel_gpu/wedged_file                      
/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1

And here is what this PR adds:

dpkg-query -L level-zero                       
/usr                                                                   
/usr/lib                                                               
/usr/lib/x86_64-linux-gnu                                              
/usr/lib/x86_64-linux-gnu/libze_loader.so.1.8.8                        
/usr/lib/x86_64-linux-gnu/libze_tracing_layer.so.1.8.8                 
/usr/lib/x86_64-linux-gnu/libze_validation_layer.so.1.8.8              
/usr/lib/x86_64-linux-gnu/libze_loader.so.1                            
/usr/lib/x86_64-linux-gnu/libze_tracing_layer.so.1                     
/usr/lib/x86_64-linux-gnu/libze_validation_layer.so.1                  

dpkg-query -L level-zero-devel                 
/usr                                                                   
/usr/include                                                           
/usr/include/level_zero                                                
/usr/include/level_zero/layers                                         
/usr/include/level_zero/layers/zel_tracing_api.h                       
/usr/include/level_zero/layers/zel_tracing_ddi.h                       
/usr/include/level_zero/layers/zel_tracing_register_cb.h               
/usr/include/level_zero/loader                                         
/usr/include/level_zero/loader/ze_loader.h                             
/usr/include/level_zero/ze_api.h                                       
/usr/include/level_zero/ze_ddi.h                                       
/usr/include/level_zero/zes_api.h                                      
/usr/include/level_zero/zes_ddi.h                                      
/usr/include/level_zero/zet_api.h                                      
/usr/include/level_zero/zet_ddi.h                                      
/usr/lib                                                               
/usr/lib/x86_64-linux-gnu                                              
/usr/lib/x86_64-linux-gnu/pkgconfig                                    
/usr/lib/x86_64-linux-gnu/pkgconfig/level-zero.pc                      
/usr/lib/x86_64-linux-gnu/pkgconfig/libze_loader.pc                    
/usr/lib/x86_64-linux-gnu/libze_loader.so                              
/usr/lib/x86_64-linux-gnu/libze_tracing_layer.so                       
/usr/lib/x86_64-linux-gnu/libze_validation_layer.so

"root": "{DEPS_ROOT}/opencl/runtime/linux/oclgpu"
},
"tbb": {
"github_tag": "v2021.5.0",
"version": "2021.5.0",
Expand All @@ -40,47 +46,6 @@
"root": "{ARCHIVE_ROOT}/comp/oclfpga/linux"
}
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nice thing about staging is that we know the versions of the dependencies. How do we know what "latest" is?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

latest is a moving tag.
Exact versions can be found in a container build logs or from the inside of the container through package manager commands, like:

# apt list --installed | grep intel-igc-cm                                                                                        
intel-igc-cm/now 1.0.119 amd64 [installed,local]

"linux_staging": {
"compute_runtime": {
"github_tag": "22.31.23852",
"version": "22.31.23852",
"url": "https://github.com/intel/compute-runtime/releases/tag/22.31.23852",
"root": "{DEPS_ROOT}/opencl/runtime/linux/oclgpu"
},
"igc": {
"github_tag": "igc-1.0.11485",
"version": "1.0.11485",
"url": "https://github.com/intel/intel-graphics-compiler/releases/tag/igc-1.0.11485",
"root": "{DEPS_ROOT}/opencl/runtime/linux/oclgpu"
},
"cm": {
"github_tag": "cmclang-1.0.144",
"version": "1.0.144",
"url": "https://github.com/intel/cm-compiler/releases/tag/cmclang-1.0.144",
"root": "{DEPS_ROOT}/opencl/runtime/linux/oclgpu"
},
"tbb": {
"github_tag": "v2021.5.0",
"version": "2021.5.0",
"url": "https://github.com/oneapi-src/oneTBB/releases/download/v2021.5.0/oneapi-tbb-2021.5.0-lin.tgz",
"root": "{DEPS_ROOT}/tbb/lin"
},
"oclcpu": {
"github_tag": "2021-WW50",
"version": "2021.13.11.0.23",
"url": "https://github.com/intel/llvm/releases/download/2021-WW50/oclcpuexp-2021.13.11.0.23_rel.tar.gz",
"root": "{DEPS_ROOT}/opencl/runtime/linux/oclcpu"
},
"fpgaemu": {
"github_tag": "2021-WW50",
"version": "2021.13.11.0.23",
"url": "https://github.com/intel/llvm/releases/download/2021-WW50/fpgaemu-2021.13.11.0.23_rel.tar.gz",
"root": "{DEPS_ROOT}/opencl/runtime/linux/oclfpgaemu"
},
"fpga": {
"root": "{ARCHIVE_ROOT}/comp/oclfpga/linux"
}
},
"windows": {
"compute_runtime": {
"version": "101.1191",
Expand Down Expand Up @@ -109,4 +74,4 @@
"root": "{ARCHIVE_ROOT}/comp/oclfpga/win"
}
}
}
}
4 changes: 4 additions & 0 deletions devops/scripts/generate_test_matrix.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ module.exports = ({core, process}) => {
driverOld["linux"]["igc"]["version"] ||
driverNew["linux"]["cm"]["version"] !==
driverOld["linux"]["cm"]["version"] ||
driverNew["linux"]["level_zero"]["version"] !==
driverOld["linux"]["level_zero"]["version"] ||
driverNew["linux"]["tbb"]["version"] !==
driverOld["linux"]["tbb"]["version"] ||
driverNew["linux"]["oclcpu"]["version"] !==
Expand All @@ -37,6 +39,7 @@ module.exports = ({core, process}) => {
driverNew["linux"]["compute_runtime"]["github_tag"],
"igc_tag" : driverNew["linux"]["igc"]["github_tag"],
"cm_tag" : driverNew["linux"]["cm"]["github_tag"],
"level_zero_tag" : driverNew["linux"]["level_zero"]["github_tag"],
"tbb_tag" : driverNew["linux"]["tbb"]["github_tag"],
"cpu_tag" : driverNew["linux"]["oclcpu"]["github_tag"],
"fpgaemu_tag" : driverNew["linux"]["fpgaemu"]["github_tag"],
Expand Down Expand Up @@ -82,6 +85,7 @@ module.exports = ({core, process}) => {
driverNew["linux"]["compute_runtime"]["github_tag"],
"igc_tag" : driverNew["linux"]["igc"]["github_tag"],
"cm_tag" : driverNew["linux"]["cm"]["github_tag"],
"level_zero_tag" : driverNew["linux"]["level_zero"]["github_tag"],
"tbb_tag" : driverNew["linux"]["tbb"]["github_tag"],
"cpu_tag" : driverNew["linux"]["oclcpu"]["github_tag"],
"fpgaemu_tag" : driverNew["linux"]["fpgaemu"]["github_tag"],
Expand Down
6 changes: 6 additions & 0 deletions devops/scripts/install_drivers.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
CR_TAG=$compute_runtime_tag
IGC_TAG=$igc_tag
CM_TAG=$cm_tag
L0_TAG=$level_zero_tag
TBB_TAG=$tbb_tag
FPGA_TAG=$fpgaemu_tag
CPU_TAG=$cpu_tag
Expand Down Expand Up @@ -39,6 +40,7 @@ InstallIGFX () {
echo "Compute Runtime version $CR_TAG"
echo "IGC version $IGC_TAG"
echo "CM compiler version $CM_TAG"
echo "Level Zero version $L0_TAG"
python3 $LOCATION/get_release.py intel/intel-graphics-compiler $IGC_TAG \
| grep ".*deb" \
| wget -qi -
Expand All @@ -47,6 +49,10 @@ InstallIGFX () {
| wget -qi -
sha256sum -c *.sum && \
python3 $LOCATION/get_release.py intel/cm-compiler $CM_TAG \
| grep ".*deb" \
| grep -v "u18" \
| wget -qi -
python3 $LOCATION/get_release.py oneapi-src/level-zero $L0_TAG \
| grep ".*deb" \
| wget -qi -
dpkg -i *.deb && rm *.deb *.sum
Expand Down
7 changes: 6 additions & 1 deletion devops/scripts/update_drivers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,11 @@ def uplift_linux_igfx_driver(config, platform_tag):
config[platform_tag]['cm']['version'] = cm['tag_name'].replace('cmclang-', '')
config[platform_tag]['cm']['url'] = 'https://github.com/intel/cm-compiler/releases/tag/' + cm['tag_name']

level_zero = get_latest_release('oneapi-src/level-zero')
config[platform_tag]['level_zero']['github_tag'] = level_zero['tag_name']
config[platform_tag]['level_zero']['version'] = level_zero['tag_name']
config[platform_tag]['level_zero']['url'] = 'https://github.com/oneapi-src/level-zero/releases/tag/' + level_zero['tag_name']

return config


Expand All @@ -52,5 +57,5 @@ def main(platform_tag):


if __name__ == '__main__':
platform_tag = sys.argv[1] if len(sys.argv) > 1 else 'linux_staging'
platform_tag = sys.argv[1] if len(sys.argv) > 1 else "ERROR_PLATFORM"
sys.stdout.write(main(platform_tag) + '\n')