container_push targets are not robust to transient issues

[fejta]

Would you accept a PR for me to add retries here?

We're seeing flakiness when trying to push a lot of images concurrently:

https://testgrid.k8s.io/sig-testing-prow#push-prow

[nlopezgi]

hi @fetja,
do you have ideas about why the push is flaky (e.g. is it connection issues?) I think adding retries with proper exponential backoff and limits would be fine. But likely those changes will have to go to in https://github.com/google/containerregistry. Let mr know if you want to pursue this to guide you how to send changes to that repo (the SoT for their code is in google3)

[fejta]

Click on any red cell in the testgrid link I gave above (https://testgrid.k8s.io/sig-testing-prow#push-prow&width=20)

Most common is:

/root/.cache/bazel/_bazel_root/05618a594cb3499a4b912817865867cd/execroot/io_k8s_test_infra/bazel-out/k8-fastbuild/bin/prow/release-push: line 116: wait: pid 17196 is not a child of this shell

[ixdy]

The particular failure @fejta linked

.../release-push: line 116: wait: pid 17196 is not a child of this shell

appears to be described by moby/moby#36864 (comment) - basically, it is an issue in bash 4.3 when run in environments where the user process limit (ulimit -u) is set to unlimited, such as inside a docker container. Upgrading to bash 4.4 seems to resolve the issue.

https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1590001 offers a test script which replicates the problem easily under docker when using bash 4.3; I was not able to reproduce when using bash 4.4 under docker.

[ixdy]

and that failure is manifesting in the contrib/push-all.sh.tpl script:

rules_docker/contrib/push-all.sh.tpl

Lines 39 to 44 in 7f61e98

	# Wait for all of the subprocesses, failing the script if any of them failed.
	if [ "${#PIDS[@]}" != 0 ]; then
	for pid in ${PIDS[@]}; do
	wait ${pid}
	done
	fi

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_docker!

[github-actions]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"