Separate Push CI images from Scheduled CI

[ydshieh]

What does this PR do?

⚠️ Before merge, I need to build the new push CI images that have the new tags.

Currently, if setup.py is changed, Push CI will re-build the CI images before running tests.

transformers/.github/workflows/self-push-caller.yml

Lines 39 to 43 in 7e84723

	build-docker-containers:
	needs: check-for-setup
	if: (github.event_name == 'push') && (needs.check-for-setup.outputs.changed == '1')
	uses: ./.github/workflows/build-docker-images.yml
	secrets: inherit

However, this may cause different jobs in a scheduled CI workflow run to use images with different versions.

Recently, when tokenizers is changed to 0.13, some jobs failed in the scheduled CI due to the new image (with tokenizers 0.13) but the transformers code in those runs still required tokenizers < 0.13.

This PR separates the push CI images from scheduled CI.

[muellerzr]

Great solution! Makes sense, lg2m

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[LysandreJik]

Good idea! Let's document that somewhere :)

[ydshieh]

Added in Transformers testing design, internal document on Notion.

Text version

The CI for a push event (to main branch) will check if setup.py is changed. If yes, it will launch the docker image build CI before launching the actual tests. This is to make sure the tests will run against the specified package versions in setup.py. In order to avoid the conflict with the daily schedule CI, which should use the same image version for all jobs during a workflow run, we separate the CI images used for push events and schedule CI. The docker images used for push events start with the tag of the images used in the corresponding jobs in scheduled CI, but with a postfix push-ci. For example, transformers-all-latest-gpu in schedule CI will be transformers-all-latest-gpu-push-ci in push CI.