Support missing data in LS copying process

[jeromekelleher]

We want to be able to support small amounts of missing data in the LS copying process. This will be indicated by a -1 in the genotype for a given sample at a give site. As implemented in tskit-dev/tskit#489, we can update the copying process quite simply to accommodate missing data. We can then either impute (the default, I'd imagine) or preserve the missingness by snipping the edges where the haplotype for a given sample is missing.

Any thoughts @HyanWon?

Related to #169

[hyanwong]

We discussed the idea of imputing everything (easiest), then having a separate routine that masks out sites missing from the SampleData file by snipping out edges. This seems sensible to me. One question is whether the normal inference process has an "impute_missing" flag which runs this routine as a final step.

[hyanwong]

To do the snipping, we need access to the original SampleData file as well as the inferred TS. Given that the SampleData file is a necessary input to match_samples, it seems sensible to have a flag on that method that also runs the "missingness-snipping" routine as a final step.

[jeromekelleher]

We can do it inline as well during match_samples --- it's quite straightforward to do actually, so probably simplest to add it as an option impute_missing=True to match_samples I think.

[hyanwong]

A follow-up to this issue: we might like to allow missing data within an ancestor (e.g. if we are using an ancient sample as an ancestor). From this comment in the test suite, it looks as if a site is all missing when reconstructing normally (i.e. without ancients), we set then site value to 0 rather than -1. Yet @awohns thinks that sites marked as missing in the middle of an ancestor are allowed, in which case I assume we should be marking these as -1?

Can you confirm whether sites marked as tskit.MISSING_DATA within the middle of an otherwise intact ancestor are just skipped over (or equivalent) by the matching process, @jeromekelleher ? If so I think we can (a) close this issue and (b) alter the building algorithm to put tskit.MISSING_DATA where all sites are missing, rather than 0.

[jeromekelleher]

Missing data should be treated the same in samples and within ancestor haplotypes (ie., in the middle) for the copying process, so I don't see any reason this wouldn't work. I think we can close this issue all right, this functionality was implemented a while ago.

[hyanwong]

So it looks like we should make the following change in the ancestor builder, @jeromekelleher:

hyanwong@7590f49

Do you agree? If so I'll change the tests and also add it to the C version. This could make a difference to Wilder's HGDP-with-missing-data inferences

[jeromekelleher]

Probably - will need to look at it in more detail though.