[ci] Split up Jenkins into platform-specific jobs

[driazati]

Right now all CI (excluding GitHub Actions, which this doesn't address at all) goes through 1 indirection to get any useful information outside of a basic pass/fail, the tvm-ci/pr-head job here:

The tvm-ci/pr-head job then mixes all the tests together, which with sharding means dozens of tests in a long vertical column where it's hard to find exactly what failed. driazati#38 shows an alternative where each platform (cpu, gpu, arm, etc) has its own job that reports from Jenkins to GitHub independently. To implement this, (1) Jenkins needs to be configured with job definitions for each of these platforms and (2) the Jenkinsfile in apache/tvm needs to be broken up

• [ci] Split Jenkinsfile into platform-specific jobs #13300 splits up the Jenkinsfile into multiple ones per platform, which each of the jobs from (1) are set up to read. This doesn't do anything on its own.
• for (1) that happens in https://github.com/tlc-pack/ci with the jenkins jobs here: https://github.com/tlc-pack/ci/blob/main/jenkins/jenkins-jobs/prod/tvm.yaml. Move temp jenkins jobs to apache/tvm tlc-pack/ci#58 has the changes to move them over from my test repo to the main repo. Once merged this will start reporting the new statuses to GitHub
• Once the previous step is working we can merge [ci] Remove Jenkinsfile for migration to platform-specific jobs #13316 which will make the new statuses required and get rid of the old tvm-ci/pr-head requirement.
• Another PR in tlc-pack/ci to remove the tvm-ci/pr-head will stop it reporting to GitHub: tlc-pack/ci@20bdc59

Follow up fixes:

• Make tvm-bot rerun aware of the new jobs: [ci] Make tvm-bot aware of platform specific jobs #13571
• Cancel the remaining jobs when any fails (or if just lint fails)
• Re-balance shards (e.g. for hexagon / cortexm)
• Fix the docs deploy: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm-gpu/detail/main/3/ [ci] Fix docs deploy #13570

This will remove the Jenkinsfile at the top level, so Docker image updates would happen now in the ci/jenkins/data.py file which has the source data for the Jenkinsfile templates.

cc @Mousius @areusch @gigiblender @leandron

[driazati]

This is done now

[quic-sanirudh]

@driazati I know this question is probably quite late, but I see that in many PRs, when CI lint fails, the other jobs keep running. I see that there was a task above to fail other jobs when one fails (Cancel the remaining jobs when any fails (or if just lint fails)).

I wonder if this is possible to be enabled now, as I see PRs that have one job failed, but others are still running, and they take up resources causing many other PRs to be waiting (especially for GPU resources).

Would it be possible to kill the other jobs when one fails, or could we enable that at least for lint failures? Could there be other issues because of doing something like this?