vecbench: download dataset as separate files

[andy-kimball]

Update DatasetLoader (used by vecbench and the vecann workload) to download the train, test, and neighbors vector sets as separate files rather than as a single file. Separating the files will allow huge train datasets to be separated into multiple files. For the new files, use the .fbin and .ibin formats:

.fbin
[num_vectors (uint32), vector_dim (uint32), vector_array (float32)]

.ibin
[num_vectors (uint32), num_neighbors_per_vector (uint32), neighbor_array (int32)]

Epic: CRDB-42943
Release note: None

[cockroach-teamcity]

This change is 

[DrewKimball]

Reviewed 7 of 7 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @golgeek, @herkolategan, and @mw5h)

---

pkg/workload/vecann/datasets.go line 145 at r1 (raw file):

		trainCount = trainSet.Count
	} else {
		// Only read the header to get the train vector count.

Maybe there should be a single metadata file with the counts + dimensions?

[andy-kimball]

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @DrewKimball, @golgeek, @herkolategan, and @mw5h)

---

pkg/workload/vecann/datasets.go line 145 at r1 (raw file):

Previously, DrewKimball (Drew Kimball) wrote…

Maybe there should be a single metadata file with the counts + dimensions?

I think that would add some complexity that wouldn't be worth the benefit. It's easy and efficient to read the first few bytes of the file. We can revisit if we find some good reason to do that, though.

[andy-kimball]

bors r=drewkimball

[craig]

Build succeeded:

• check_generated_code
• local_roachtest_fips
• examples_orms
• lint
• linux_amd64_build
• acceptance
• local_roachtest
• linux_amd64_fips_build
• docker_image_amd64
• unit_tests