Skip to content

Add a workflow to install nightly intel/llvm + drivers #621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 6, 2021

Conversation

oleksandr-pavlyk
Copy link
Contributor

@oleksandr-pavlyk oleksandr-pavlyk commented Oct 5, 2021

Build dpctl with Open Source DPC++ toolchain.

  1. Download and unpack nightly build of SYCL bundle
  2. Download and unpack OpenCL CPU driver
  3. Download and unpack OpenCL FPGA emulator
  4. Download and unpack oneAPI TBB
  5. Set-up LD_LIBRARY_PATH and PATH, CPATH
  6. Set-up OCL_ICD_FILENAMES so that SYCL OpenCL plug-in knows what library to load
  7. Build dpctl and run pytest.

@oleksandr-pavlyk oleksandr-pavlyk force-pushed the build-with-os-llvm-sycl branch 3 times, most recently from bcf6ddc to 7e043bb Compare October 5, 2021 15:54
@coveralls
Copy link
Collaborator

coveralls commented Oct 5, 2021

Coverage Status

Coverage remained the same at 74.413% when pulling f6de191 on build-with-os-llvm-sycl into 112b6be on master.

@oleksandr-pavlyk oleksandr-pavlyk force-pushed the build-with-os-llvm-sycl branch 2 times, most recently from e1740f3 to 8870ef4 Compare October 5, 2021 16:23
export NIGHTLY_TAG=$(python3 -c "import sys, urllib.parse as ul; print (ul.quote_plus(sys.argv[1]))" $(git tag | sort -V | tail -1))
cd ..
export DOWNLOAD_URL_PREFIX=https://github.com/intel/llvm/releases/download
wget ${DOWNLOAD_URL_PREFIX}/${NIGHTLY_TAG}/dpcpp-compiler.tar.gz
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can cache these downloaded copies to speed up the workflow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be great. I just need to know where to look for an example. GTest installation was being cached, so I will look there.

Copy link
Contributor

@diptorupd diptorupd Oct 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see couple of issues here: Where are you checking if the cache is hit? I do not think caching the sycl-bundle repo based on all files in it is going to work. We need to ascertain if the cached folder is the one we want to use.

I suggest the following:
a) Create a conf file inside sycl-bundle that stores the SHA of the nightly that was used by the workflow previously.
b) Hash the sycl-bundle folder as before to ensure we have all the needed files.
c) Use the following git command (I found using internet search) to get the SHA of the latest nightly:

git -c 'versionsort.suffix=-' ls-remote --tags --sort='v:refname' git@github.com:intel/llvm.git| tail --lines=1 | awk '{print $1}'

Then check if a cached entry exists

     - name: Download nightly and components
       if: steps.cache-sycl-bundle.outputs.cache-hit != 'true'

Also check if the SHA matches. (I do not know how to do that)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You do not need a conf file. As we are storing the cloned repo we might as well run git tag | sort -V | tail -1 and compare the SHA with what the command I pasted returns.

@oleksandr-pavlyk oleksandr-pavlyk force-pushed the build-with-os-llvm-sycl branch 7 times, most recently from 8ed7f1c to aa51900 Compare October 5, 2021 19:54
Build dpctl with that

Use OCL_ICD_FILENAMES to inform plugin of CPU driver location
@oleksandr-pavlyk oleksandr-pavlyk force-pushed the build-with-os-llvm-sycl branch from aa51900 to 37d97ab Compare October 5, 2021 20:07
Save the nightly tag into bundle_id.txt and use its content to build the cache
@oleksandr-pavlyk oleksandr-pavlyk force-pushed the build-with-os-llvm-sycl branch from 06937d5 to 696ab05 Compare October 6, 2021 12:50
cd sycl_bundle
export LATEST_LLVM_TAG=$(git -c 'versionsort.suffix=-' ls-remote --tags --sort='v:refname' https://github.com/intel/llvm.git | tail --lines=1)
export LATEST_LLVM_TAG_SHA=$(echo ${LATEST_LLVM_TAG} | awk '{print $1}')
export NIGHTLY_TAG=$(python3 -c "import sys, urllib.parse as ul; print (ul.quote_plus(sys.argv[1]))" $(echo ${LATEST_LLVM_TAG} | awk '{gsub(/^refs\/tags\//, "", $2)} {print $2}'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also need to check if the NIGHLY_TAG matches with the cached bundle_id.

@oleksandr-pavlyk oleksandr-pavlyk force-pushed the build-with-os-llvm-sycl branch from 4a88d6e to 8346a0d Compare October 6, 2021 16:11
@oleksandr-pavlyk oleksandr-pavlyk merged commit 782340f into master Oct 6, 2021
@oleksandr-pavlyk oleksandr-pavlyk deleted the build-with-os-llvm-sycl branch October 6, 2021 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants