Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] review dev CI docs part2 #22850

Merged
merged 15 commits into from
Mar 13, 2024
Prev Previous commit
Next Next commit
review changes
  • Loading branch information
tsavina committed Feb 16, 2024
commit 75699c7ef8a6c25e91d2c3df780d087d2213da2f
2 changes: 1 addition & 1 deletion docs/dev/ci/github_actions/adding_tests.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# How to add Tests to the OpenVINO GitHub Actions CI
# How to add Tests

The OpenVINO repository has [many workflows](./../../../../.github/workflows), which contain
jobs for building and testing OpenVINO. Their general and structural overview is
tsavina marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
17 changes: 12 additions & 5 deletions docs/dev/ci/github_actions/caches.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Overview of Caches used in the OpenVINO GitHub Actions CI
# Caches

OpenVINO uses caches to accelerate builds and tests while minimizing network usage.

Expand All @@ -23,9 +23,14 @@ Jobs in the workflows utilize these caches based on their requirements.

## GitHub Actions Cache

This cache is used for sharing small dependencies or artifacts between runs. Refer to the [GitHub Actions official documentation](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows) for a complete reference.
This cache is used for sharing small dependencies or artifacts between runs.
Refer to the [GitHub Actions official documentation](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows)
for a complete reference.

The `CPU functional tests` job in the [`linux.yml`](./../../../../.github/workflows/linux.yml) workflow uses this cache for sharing tests execution time to speed up the subsequent runs. First, the artifacts are saved with `actions/cache/save` with a particular key `${{ runner.os }}-${{ runner.arch }}-tests-functional-cpu-stamp-${{ github.sha }}`:
The `CPU functional tests` job in the [`linux.yml`](./../../../../.github/workflows/linux.yml)
workflow uses this cache for sharing tests execution time to speed up the subsequent runs.
First, the artifacts are saved with `actions/cache/save` with a particular
key `${{ runner.os }}-${{ runner.arch }}-tests-functional-cpu-stamp-${{ github.sha }}`:
```yaml
CPU_Functional_Tests:
name: CPU functional tests
Expand All @@ -43,7 +48,8 @@ CPU_Functional_Tests:
Then it appears in the [repository's cache](https://github.com/openvinotoolkit/openvino/actions/caches):
![gha_cache_example](../../../sphinx_setup/_static/images/ci/gha_cache_example.png)

The next runs can download the artifact from the repository's cache with `actions/cache/restore` and use it:
The next runs can download the artifact from the repository's cache with `actions/cache/restore`
and use it:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "the following" instead of "next?"

tsavina marked this conversation as resolved.
Show resolved Hide resolved
```yaml
CPU_Functional_Tests:
name: CPU functional tests
Expand Down Expand Up @@ -150,7 +156,8 @@ Build:

### Providing `sccache` Prefix

The folder on the remote storage where the cache for the OS/architecture will be saved is provided via the `SCCACHE_AZURE_KEY_PREFIX` environment variable under the job's `env` key:
The folder on the remote storage where the cache for the OS/architecture will be saved is
provided via the `SCCACHE_AZURE_KEY_PREFIX` environment variable under the job's `env` key:
```yaml
Build:
...
Expand Down
27 changes: 14 additions & 13 deletions docs/dev/ci/github_actions/custom_actions.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,24 @@
# Overview of Custom GitHub Actions used in the OpenVINO GitHub Actions CI
# Custom GitHub Actions

Several actions are written specifically for the needs of the OpenVINO workflows.
Custom actions, written specifically for OpenVINO workflows, are located
in the [actions](../../../../.github/actions) folder along with their source code.

You can find all the custom actions and their source code [here](../../../../.github/actions).

Refer to the official [custom GitHub Action documentation](https://docs.github.com/en/actions/creating-actions/about-custom-actions) for more information.
Refer to the official [custom GitHub Action documentation](https://docs.github.com/en/actions/creating-actions/about-custom-actions)
for more information.

## Available Custom Actions

* [Setup Python](#setup-python)
* [System Info Print](#system-info-print)
* Smart CI (see details: [feature documentation](./smart_ci.md))

## Setup Python
## Set up Python

This custom action installs the required Python version and environment variables on the runner.

Under the hood it uses the GitHub-provided `actions/setup-python` and community-provided `deadsnakes-action` depending on the machine architecture.
`actions/setup-python` does not work on the Linux ARM64 machines so `deadsnakes-action` is used instead.
Under the hood it uses the GitHub-provided `actions/setup-python`, and community-provided
`deadsnakes-action`, depending on the machine architecture.
Since `actions/setup-python` does not work on the Linux ARM64 machines,
`deadsnakes-action` is used instead.

### Usage
```yaml
Expand All @@ -32,10 +33,10 @@ Under the hood it uses the GitHub-provided `actions/setup-python` and community-
```
where:
* `version` - the Python version to install in the `MAJOR.MINOR` format
* `pip-cache-path` - the path to the `pip` cache on the mounted share. Read more about shares and caches [here](./caches.md)
* `should-setup-pip-paths` - whether the action should set up the `PIP_CACHE_DIR` and `PIP_INSTALL_PATH` environment variables for later usage
* `self-hosted-runner` - whether the runner is self-hosted. Read more about available runners [here](./runners.md)
* `show-cache-info` - whether the action should show the share space occupied by the `pip` cache
* `pip-cache-path` - the path to the `pip` cache on the mounted share. Read more in the [shares and caches](./caches.md) documentation
* `should-setup-pip-paths` - indicates whether the action should set up the `PIP_CACHE_DIR` and `PIP_INSTALL_PATH` environment variables for later usage
* `self-hosted-runner` - indicates whether the runner is self-hosted. Learn more about [available runners](./runners.md)
* `show-cache-info` - indicates whether the action should show the share space occupied by the `pip` cache

## System Info Print

Expand Down
64 changes: 34 additions & 30 deletions docs/dev/ci/github_actions/docker_images.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
# Overview of the Docker Images used in the OpenVINO GitHub Actions CI
# Docker Images

Most of the workflows in the OpenVINO GHA CI are using [self-hosted machines with dynamically spawned runners](./runners.md) to execute jobs.
Most of the workflows in the OpenVINO GHA CI use [self-hosted machines with dynamically spawned runners](./runners.md)
to execute jobs.

To avoid corruption of the runners and machines, the workflows utilize various Docker images that introduce a layer of protection for the self-hosted machines.
To avoid the corruption of runners and machines, workflows use various Docker images
which introduce an additional layer of protection for self-hosted machines.

The Docker images are specified for each job using the `container` key. See the [GHA documentation](https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container) for the syntax reference.
Docker images are specified for each job using the `container` key. See the
[GHA documentation](https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container)
for syntax reference.

An example `Build` job from the [`linux.yml`](./../../../../.github/workflows/linux.yml) workflow:
```yaml
Expand All @@ -19,42 +23,39 @@ Build:

The `openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04` Docker image is used for this job.

Additionally, it is possible to make the caches available in the Docker containers using the `volumes` key.
Read more about the available caches and how to choose one [here](./caches.md).
Additionally, you can make the caches available in Docker containers using the `volumes` key.
For more details, refer to the [caches](./caches.md) page.

## Available Docker Images

The jobs in the workflows utilize appropriate Docker images based on a job's needs.
The jobs in the workflows use Docker images based on each job's requirements.

As the self-hosted machines are hosted in [Microsoft Azure](https://azure.microsoft.com/en-us),
it is optimal to use the Docker images hosted in an instance of [Azure Container Registry (ACR)](https://azure.microsoft.com/en-us/products/container-registry).
Since self-hosted machines are hosted in [Microsoft Azure](https://azure.microsoft.com/en-us),
it is recommended to use Docker images hosted in an instance of [Azure Container Registry (ACR)](https://azure.microsoft.com/en-us/products/container-registry).

The ACR used for the OpenVINO GHA CI is `openvinogithubactions.azurecr.io`.
The ACR used for OpenVINO GHA CI is `openvinogithubactions.azurecr.io`.

Some pros and cons of having own container registry are:
* pros:
* No pull limits for the images
* There are [limits](https://docs.docker.com/docker-hub/download-rate-limit/) for the pulls from Docker Hub
Here are some pros and cons of having your own container registry:
* Pros:
* No pull limits for images: Docker Hub imposes [limits](https://docs.docker.com/docker-hub/download-rate-limit/) on image pulls
* Fast pulling speeds
* cons:
* The registry should be populated with needed images before usage
* The registry does not mirror the images available on Docker Hub automatically
* The needed images should be added manually to the registry
* Cons:
* The registry needs to be populated with required images before usage
* The registry does not automatically mirror images available on Docker Hub
tsavina marked this conversation as resolved.
Show resolved Hide resolved
* The necessary images must be manually added to the registry

As the number of enabled workflows grew, so did the number of available Docker images.

The available Docker images are using the following pattern for their names: `openvinogithubactions.azurecr.io/dockerhub/<image-name>:<image-tag>`.
The available Docker images use the following pattern for their names: `openvinogithubactions.azurecr.io/dockerhub/<image-name>:<image-tag>`.

Most of the images on the OpenVINO ACR are mirrors of the images with the same names on Docker Hub.

The examples:
Examples:
* `openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04` corresponds to `ubuntu:20.04` from Docker Hub
* `openvinogithubactions.azurecr.io/dockerhub/ubuntu:22.04` corresponds to `ubuntu:22.04` from Docker Hub
* `openvinogithubactions.azurecr.io/dockerhub/nvidia/cuda:11.8.0-runtime-ubuntu20.04` corresponds to `nvidia/cuda:11.8.0-runtime-ubuntu20.04` from Docker Hub

## How to choose an Image
## How to Choose an Image

The Docker image required for a job stems from the nature of the job and configuration that is being tested.
The Docker image required for a job depends on the nature of the job and the configuration being tested.

tsavina marked this conversation as resolved.
Show resolved Hide resolved
An example `Build` job from the [`linux.yml`](./../../../../.github/workflows/linux.yml) workflow:
```yaml
Expand All @@ -67,12 +68,15 @@ Build:
...
```

The `openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04` Docker image is used for this job and **for other jobs in the workflow**.
Usually, if one Docker image is used for the building job, the other jobs would use the same image for testing.
The `openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04` Docker image is used for this job
and **for other jobs in the workflow**.
Usually, if one Docker image is used for the building job, other jobs use the same image for testing.

If the tests do not require any specific OS or distribution, it would be best to use the already available images: e.g., `openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04`.
If the tests do not require any specific OS or distribution, it is recommended to use
the available images, for example, `openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04`.

If the plan is to test some specific OS or distribution (e.g., `fedora`),
the Docker image for this flavour should be first uploaded to the OpenVINO ACR and only then used in a workflow.
If testing requires a specific OS or distribution (for example, `fedora`),
the Docker image for this flavour should be first uploaded to the OpenVINO ACR and
then used in a workflow.

Contact someone from the CI team for assistance with the image uploading.
Contact a member of the CI team for assistance with the image uploading.
Loading
Loading