Speed up remote file download cache checks in iree_tests

[ScottTodd]

The iree_tests/download_remote_files.py script is used to download files listed in test case JSON files like this one:

SHARK-TestSuite/iree_tests/pytorch/models/resnet50/test_cases.json

Lines 1 to 19 in 6fd8fd0

	{
	"file_format": "test_cases_v0",
	"test_cases": [
	{
	"name": "splats",
	"runtime_flagfile": "splat_data_flags.txt",
	"remote_files": []
	},
	{
	"name": "real_weights",
	"runtime_flagfile": "real_weights_data_flags.txt",
	"remote_files": [
	"https://sharkpublic.blob.core.windows.net/sharkpublic/scotttodd/iree_tests/2024-03-12/resnet50/inference_input.0.bin",
	"https://sharkpublic.blob.core.windows.net/sharkpublic/scotttodd/iree_tests/2024-03-12/resnet50/inference_output.0.bin",
	"https://sharkpublic.blob.core.windows.net/sharkpublic/scotttodd/iree_tests/2024-03-12/resnet50/real_weights.irpa"
	]
	}
	]
	}

The script is used from CI like here:

SHARK-TestSuite/.github/workflows/test_iree.yml

Lines 146 to 152 in 6fd8fd0

	# Download remote files.
	- name: "Downloading remote files for real weight model tests"
	run: |
	source ${VENV_DIR}/bin/activate
	python3 iree_tests/download_remote_files.py --root-dir iree_special_models
	python3 iree_tests/download_remote_files.py --root-dir iree_tests/pytorch/models
	python3 iree_tests/download_remote_files.py --root-dir iree_tests/sharktank

Sample runs:

• https://github.com/nod-ai/SHARK-TestSuite/actions/runs/9859499558/job/27223405047#step:6:15
• https://github.com/nod-ai/SHARK-TestSuite/actions/runs/9843893693/job/27176121463#step:6:15
• https://github.com/iree-org/iree/actions/runs/9876851075/job/27277633589#step:9:24
• https://github.com/iree-org/iree/actions/runs/9875391039/job/27272561557#step:9:24

(mi300 machine seems to be slower than mi250 machine? 30s vs 2 minutes)

We should be able to optimize this cache checking and file downloading so it takes < 10 seconds, instead of 1-3 minutes.

Ideas:

• Use a thread pool to parallelize the checks and downloads
• Zip the remote files for each test that are always used together (e.g. .zip, .tar, or .tar.gz) so only one local hash calculation, remote hash query, and download request is needed instead of 2-50.
• Switch more files from Azure to Hugging Face