Skip to content

Commit

Permalink
Add contribute page about CI (apache#9906)
Browse files Browse the repository at this point in the history 
* Add contribute page about CI

This adds some docs with a description of the TVM CI and some usage instructions to both make contributing more friendly and educate existing developers about how CI runs. Bikeshedding on content is welcome

Note: The TODOs are blocked on some other PRs and will be done before landing

* docker instructions

* Comments

* Rebase

Co-authored-by: driazati <driazati@users.noreply.github.com>
  • Loading branch information
@driazati @ylc
2 people authored and ylc committed Feb 16, 2022
1 parent 251c819 commit c1c2a43
Show file tree
Hide file tree
Showing 7 changed files with 236 additions and 85 deletions.
176 changes: 176 additions & 0 deletions docs/contribute/ci.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
.. _ci_guide:

Using TVM's CI
==============

TVM uses Jenkins for running Linux continuous integration (CI) tests on
`branches <https://ci.tlcpack.ai/job/tvm/>`_ and
`pull requests <https://ci.tlcpack.ai/job/tvm/view/change-requests/>`_ through a
build configuration specified in a `Jenkinsfile <https://github.com/apache/tvm/blob/main/Jenkinsfile>`_.
Non-critical jobs run in GitHub Actions for Windows and MacOS jobs.

A standard CI run looks something like this viewed in `Jenkins' BlueOcean viewer <https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/activity>`_.
CI runs usually take several hours to complete and pull requests (PRs) cannot be merged before CI
has successfully completed. To diagnose failing steps, click through to the failing
pipeline stage then to the failing step to see the output logs.

.. image:: https://github.com/tlc-pack/web-data/raw/main/images/contribute/ci.png
:width: 800
:alt: The Jenkins UI for a CI run


Debugging Failures
******************

When CI fails for some reason, there are several methods to diagnose the issue.

Jenkins Logs
------------

.. |pytest| replace:: ``pytest``
.. _pytest: https://docs.pytest.org/en/6.2.x/

The first place to look for a failure is in the CI logs, follow the red Xs on
the failing job to view the logs. Note:

* Jenkins does not display the full log by default, at the top of the log viewer
is a button "Show complete log" which will take you to a plaintext version of the log
* |pytest|_ failures are summarized at the bottom of the log but you will likely
need to scroll up to view the actual failure.

Reproduce Failures
------------------

Most TVM Python tests run under |pytest|_ and
can be run as described in :ref:`pr-testing`. For a closer environment to the one
than runs in CI you can run the docker images directly, build TVM, and execute
tests inside the container. See :ref:`docker_images` for details.

Keeping CI Green
****************

Developers rely on the TVM CI to get signal on their PRs before merging.
Occasionally breakages slip through and break ``main``, which in turn causes
the same error to show up on an PR that is based on the broken commit(s). Broken
commits can be identified `through GitHub <https://github.com/apache/tvm/commits/main>`_
via the commit status icon or via `Jenkins <https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/activity?branch=main>`_.
In these situations it is possible to either revert the offending commit or
submit a forward fix to address the issue. It is up to the committer and commit
author which option to choose, keeping in mind that a broken CI affects all TVM
developers and should be fixed as soon as possible.

Skip CI for Reverts
-------------------

For reverts and trivial forward fixes, adding ``[skip ci]`` to the revert's
commit message will cause CI to shortcut and only run lint. Committers should
take care that they only merge CI-skipped PRs to fix a failure on ``main`` and
not in cases where the submitter wants to shortcut CI to merge a change faster.

.. code:: bash
# Revert HEAD commit, make sure to insert '[skip ci]' at the beginning of
# the commit subject
git revert HEAD
git checkout -b my_fix
# After you have pushed your branch, create a PR as usual.
git push my_repo
# Example: Skip CI on a branch with an existing PR
# Adding this commit to an existing branch will cause a new CI run where
# Jenkins is skipped
git commit --allow-empty --message "[skip ci] Trigger skipped CI"
git push my_repo
Handling Flaky Failures
***********************

.. https://stackoverflow.com/questions/4743845/format-text-in-a-link-in-restructuredtext/4836544#4836544
.. |pytest's @xfail decorator| replace:: pytest's ``@xfail`` decorator
.. _pytest's @xfail decorator: https://docs.pytest.org/en/6.2.x/skipping.html#xfail-mark-test-functions-as-expected-to-fail
.. |strict=True| replace:: ``strict=True``
.. _strict=True: https://docs.pytest.org/en/6.2.x/skipping.html#strict-parameter

If you notice a failure on your PR that seems unrelated to your change, you should
search `recent GitHub issues related to flaky tests <https://github.com/apache/tvm/issues?q=is%3Aissue+%5BCI+Problem%5D+Flaky+>`_ and
`file a new issue <https://github.com/apache/tvm/issues/new?assignees=&labels=&template=ci-problem.md&title=%5BCI+Problem%5D+>`_
if you don't see any reports of the failure. If a certain test or class of tests affects
several PRs or commits on ``main`` with flaky failures, the test should be disabled via
|pytest's @xfail decorator|_ with |strict=True|_ and the relevant issue linked in the
disabling PR.

.. code:: python
@pytest.mark.xfail(strict=False, reason="Flaky test: https://github.com/apache/tvm/issues/1234
def test_something_flaky():
pass
``ci-docker-staging``
*********************
The `ci-docker-staging <https://github.com/apache/tvm/tree/ci-docker-staging>`_
branch is used to test updates to Docker images and ``Jenkinsfile`` changes. When
running a build for a normal PR from a forked repository, Jenkins uses the code
from the PR except for the ``Jenkinsfile`` itself, which comes from the base branch.
When branches are built, the ``Jenkinsfile`` in the branch is used, so a committer
with write access must push PRs to a branch in apache/tvm to properly test
``Jenkinsfile`` changes. If your PR makes changes to the ``Jenkinsfile``, make sure
to @ a `committer <https://github.com/apache/tvm/blob/main/CONTRIBUTORS.md>`_
and ask them to push your PR as a branch to test the changes.
.. _docker_images:
Docker Images
*************
.. |top_of_the_Jenkinsfile| replace:: top of the ``Jenkinsfile``
.. _top_of_the_Jenkinsfile: https://github.com/apache/tvm/blob/7481a297740f073b193a3f09b3e27f056e8c7f2e/Jenkinsfile#L48-L54
Each CI job runs most of its work inside a Docker container, built from files
in the `docker/ <https://github.com/apache/tvm/tree/main/docker>`_ folder. These
files are built nightly in Jenkins via the `docker-images-ci <https://ci.tlcpack.ai/job/docker-images-ci/>`_ job.
The images for these containers are hosted in the `tlcpack Docker Hub <https://hub.docker.com/u/tlcpack>`_
and referenced at the |top_of_the_Jenkinsfile|_. These can be inspected and run
locally via standard Docker commands.
.. code:: bash
# Beware: CI images can be several GB in size
# Get a bare docker shell in the ci-gpu container
docker run -it tlcpack/ci-gpu:v0.78 /bin/bash
``docker/bash.sh`` will automatically grab the latest image from the ``Jenkinsfile``
and help in mounting your current directory.
.. code:: bash
# Run the ci_cpu image specified in Jenkinsfile
cd tvm
bash docker/bash.sh ci_cpu
# the tvm directory is automatically mounted
# example: build tvm (note: this will overrwrite build/)
$ ./tests/scripts/task_config_build_cpu.sh
$ ./tests/scripts/task_build.sh build -j32
Reporting Issues
****************
Issues with CI should be `reported on GitHub <https://github.com/apache/tvm/issues/new?assignees=&labels=&template=ci-problem.md&title=%5BCI+Problem%5D+>`_
with a link to the relevant jobs, commits, or PRs.
36 changes: 1 addition & 35 deletions docs/contribute/committer_guide.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ when they actively manage outstanding PRs,
but watch the community less frequently in the rest of the time.

Remember that your merit will never go away, so please
take your time and pace when contributing to the project:)
take your time and pace when contributing to the project :)


Broad Collaboration
Expand All @@ -101,37 +101,3 @@ Sometimes, we tend to only interact with people we know.
However, broad collaborations are necessary to the success of the project.
Try to keep that in mind, shepherd PRs for, and request code reviews from
community members who you do not interact physically.


Keeping CI Green
----------------
Developers rely on the TVM CI to get signal on their PRs before merging.
Occasionally breakges slip through and break ``main``, which in turn causes
the same error to show up on an PR that is based on the broken commit(s).
In these situations it is possible to either revert the offending commit or
submit a forward fix to address the issue. It is up to the committer and commit
author which option to choose, keeping in mind that a broken CI affects all TVM
developers and should be fixed as soon as possible.

For reverts and trivial forward fixes, adding ``[skip ci]`` to the revert's
commit message will cause CI to shortcut and only run lint. Committers should
take care that they only merge CI-skipped PRs to fix a failure on ``main`` and
not in cases where the submitter wants to shortcut CI to merge a change faster.

.. code:: bash
# Example: Skip CI on a revert
# Revert HEAD commit, make sure to insert '[skip ci]' at the beginning of
# the commit subject
git revert HEAD
git checkout -b my_fix
# After you have pushed your branch, create a PR as usual.
git push my_repo
# Example: Skip CI on a branch with an existing PR
# Adding this commit to an existing branch will cause a new CI run where
# Jenkins is skipped
git commit --allow-empty --message "[skip ci] Trigger skipped CI"
git push my_repo
6 changes: 3 additions & 3 deletions docs/contribute/community.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@
.. _community_guide:

TVM Community Guideline
=======================
TVM Community Guidelines
========================

TVM adopts the Apache style model and governs by merit. We believe that it is important to create an inclusive community where everyone can use, contribute to, and influence the direction of the project. See `CONTRIBUTORS.md <https://github.com/apache/tvm/blob/main/CONTRIBUTORS.md>`_ for the current list of contributors.

Expand All @@ -42,7 +42,7 @@ Committers are individuals who are granted the write access to the project. A co
- Quality of contributions: High-quality, readable code contributions indicated by pull requests that can be merged without a substantial code review. History of creating clean, maintainable code and including good test cases. Informative code reviews to help other contributors that adhere to a good standard.
- Community involvement: active participation in the discussion forum, promote the projects via tutorials, talks and outreach. We encourage committers to collaborate broadly, e.g. do code reviews and discuss designs with community members that they do not interact physically.

The Project Management Committee(PMC) consists group of active committers that moderate the discussion, manage the project release, and proposes new committer/PMC members. Potential candidates are usually proposed via an internal discussion among PMCs, followed by a consensus approval, i.e. least 3 +1 votes, and no vetoes. Any veto must be accompanied by reasoning. PMCs should serve the community by upholding the community practices and guidelines TVM a better community for everyone. PMCs should strive to only nominate new candidates outside of their own organization.
The `Project Management Committee (PMC) <https://projects.apache.org/committee.html?tvm>`_ consists group of active committers that moderate the discussion, manage the project release, and proposes new committer/PMC members. Potential candidates are usually proposed via an internal discussion among PMCs, followed by a consensus approval, (i.e. at least 3 +1 votes, and no vetoes). Any veto must be accompanied by reasoning. PMCs should serve the community by upholding the community practices and guidelines TVM a better community for everyone. PMCs should strive to only nominate new candidates outside of their own organization.


Reviewers
Expand Down
53 changes: 29 additions & 24 deletions docs/contribute/document.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ it is a "simple, comprehensive and nearly universally-applicable scheme. It is
proven in practice across a wide variety of fields and applications."

This document describes the organization of TVM documentation, and how to write
new documentation.
new documentation. See `docs/README.md <https://github.com/apache/tvm/tree/main/docs#build-locally>`_
for instructions on building the docs.

The Four Document Types
***********************
Expand All @@ -40,8 +41,8 @@ without necessarily explaining why the software works the way it does. Those
explanations can be saved for other document types. An introductory tutorial
focuses on a successful first experience. These are the most important docs to
turning newcomers into new users and developers. A fully end-to-end
tutorial&mdash; from installing TVM and supporting ML software, to creating and
training a model, to compiling to different architectures&mdash;will give a new
tutorial — from installing TVM and supporting ML software, to creating and
training a model, to compiling to different architectures — will give a new
user the opportunity to use TVM in the most efficient way possible. A tutorial
teaches a beginner something they need to know. This is in contrast with a
how-to, which is meant to be an answer to a question that a user with some
Expand Down Expand Up @@ -92,7 +93,7 @@ Within these documents you can explore contradictory and conflicting position,
and help the reader make sense of how and why the software was built the way it
is. It's not the place for how-tos and descriptions on how to accomplish tasks.
They instead focus on higher level concepts that help with the understanding of
the project. Generally these are written by the architects and developers of
the project. Generally these are written by the architects and developers of
the project, but can useful to help both users and developers to have a deeper
understanding of why the software works the way it does, and how to contribute
to it in ways that are consistent with the underlying design principles.
Expand Down Expand Up @@ -124,18 +125,22 @@ Technical Details
*****************

We use the `Sphinx <http://sphinx-doc.org>`_ for the main documentation.
Sphinx support both the reStructuredText and markdown. When possible, we
encourage to use reStructuredText as it has richer features. Note that the
python doc-string and tutorials allow you to embed reStructuredText syntax.
Sphinx supports both reStructuredText and markdown. When possible, we
encourage reStructuredText as it has richer features. Note that the
Python doc-string and tutorials allow you to embed reStructuredText syntax.

See
`docs/README.md <https://github.com/apache/tvm/tree/main/docs#build-locally>`_
for instructions on building the docs.


Python Reference Documentation
------------------------------

We use `numpydoc <https://numpydoc.readthedocs.io/en/latest/>`_ format to
document the function and classes. The following snippet gives an example
docstring. We always document all the public functions, when necessary,
provide an usage example of the features we support(as shown below).
We use the `numpydoc <https://numpydoc.readthedocs.io/en/latest/>`_ format to
document the function and classes. The following snippet gives an example
docstring. We always document all the public functions, when necessary,
provide an usage example of the features we support (as shown below).

.. code:: python
Expand Down Expand Up @@ -167,19 +172,19 @@ provide an usage example of the features we support(as shown below).
"""
return rv1
Be careful to leave blank lines between sections of your documents. In the
above case, there has to be a blank line before `Parameters`, `Returns` and
`Examples` in order for the doc to be built correctly. To add a new function to
the doc, we need to add the `sphinx.autodoc
<http://www.sphinx-doc.org/en/master/ext/autodoc.html>`_ rules to the
`docs/api/python <https://github.com/apache/tvm/tree/main/docs/api/python>`_).
Be careful to leave blank lines between sections of your documents. In the
above case, there has to be a blank line before ``Parameters``, ``Returns`` and
``Examples`` in order for the doc to be built correctly. To add a new function to
the docs, we need to add the `sphinx.autodoc
<http://www.sphinx-doc.org/en/master/ext/autodoc.html>`_ rules to
`docs/reference/api/python <https://github.com/apache/tvm/tree/main/docs/reference/api/python>`_).
You can refer to the existing files under this folder on how to add the
functions.

C++ Reference Documentation
---------------------------

We use the doxgen format to document c++ functions. The following snippet
We use the doxygen format to document c++ functions. The following snippet
shows an example of c++ docstring.

.. code:: c++
Expand All @@ -200,15 +205,15 @@ add comments about code logics to improve readability.
Sphinx Gallery How-Tos
----------------------

We use the `sphinx-gallery <https://sphinx-gallery.github.io/>`_ to build many
python how-tos. You can find the source code under `gallery
<https://github.com/apache/tvm/tree/main/gallery>`_ quite self explanatory.
We use `sphinx-gallery <https://sphinx-gallery.github.io/>`_ to build many
Python how-tos. You can find the source code under `gallery
<https://github.com/apache/tvm/tree/main/gallery>`_.
One thing that worth noting is that the comment blocks are written in
reStructuredText instead of markdown so be aware of the syntax.

The how-to code will run on our build server to generate the document page. So
The how-to code will run on our build server to generate the document page. So
we may have a restriction like not being able to access a remote Raspberry Pi,
in such case add a flag variable to the tutorial (e.g. `use_rasp`) and allow
in such case add a flag variable to the tutorial (e.g. ``use_rasp``) and allow
users to easily switch to the real device by changing one flag. Then use the
existing environment to demonstrate the usage.

Expand All @@ -218,7 +223,7 @@ If you add a new categorization of how-to, you will need to add references to

Refer to Another Location in the Document
-----------------------------------------
Please use sphinx's `:ref:` markup to refer to another location in the same doc.
Please use sphinx's ``:ref:`` markup to refer to another location in the same doc.

.. code-block:: rst
Expand Down
Loading

0 comments on commit c1c2a43

Please sign in to comment.