Relatedness, no Dataproc

[MattWellie]

Concept PR, related to #869

Takes the Relatedness Stage of the large_cohort pipeline, and tries to remove it from Dataproc

The Relatedness Stage as it sits in main is a little convoluted

• one entrypoint (run) takes a MT and a HT as input, and makes 2 HTs as output
• The PCRelate stage runs first (uses one MT, makes one HT), and is optionally skipped if the result HT already exists as a checkpoint (here)
• The Sample Flagging stage runs next, taking the PCRelate HT and a SampleQC HT as input, making another HT

This PR splits this into two Stages - RelatednessPCRelate and RelatednessFlag, each running the separate part of that larger process.

For each of these steps there's a separate script, and a Stage.
The script sets up a Hail runtime, runs the code, and writes the output locally
The Stage wrapper creates a VM, copys the data into the VM, runs the script, and copys the output back.

This could all be one Stage, just as it's implemented at the moment, but breaking it up this way seems logical enough as the two methods create distinct output, and we keep both outputs separately in GCP. It also makes the interface easier, instead of having to copy multiple tables in/out of one job.

[michael-harper]

Just tried kicking off a test batch which failed because tshirt_mt_sizing returns a string, so 50Gi * 2 becomes 50Gi50Gi 😅

Suggested change

	tshirt_mt_sizing(
	sequencing_type=config_retrieve(['workflow', 'sequencing_type']),
	cohort_size=len(cohort.get_sequencing_group_ids()),
	)
	* 2
	t_shirt_size_value = tshirt_mt_sizing(
	sequencing_type=config_retrieve(['workflow', 'sequencing_type']),
	cohort_size=len(cohort.get_sequencing_group_ids()),
	).split('Gi')[0]
	required_storage_value = int(t_shirt_size_value) * 2
	required_storage = f'{required_storage_value}Gi'

[michael-harper]

Ran some tests using tenk10k data in bioheart-test. The results seem to be the same between DP and no-DP
Relatedness PC Relate batch and Relatedness Flag (noting that metamist registration did not work)

[MattWellie]

metamist registration did not work

I assumed a 'mt' analysisType existed, but... maybe it doesn't. That seems to be the foreign key error it's failing on