Allow truncated samples (with missing data to left and right)

[hyanwong]

I think it shouldn't be too hard to allow samples with missing data, presumably marked as a variant whose allelic state is set to tskit.NULL using the current notation this would be an allelic state equal to constants.UNKNOWN_ALLELE e.g.

sample_data.add_site(
    position=1234, genotypes=[0, 0, 1, tsinfer.constants.UNKNOWN_ALLELE], alleles=["G", "C"])

Allowing missing sites in the zarr storage would require setting sites.genotypes to dtype=np.int8 rather than np.uint8, but I guess that's fine. It would need a bit of hacking to pass the various error checks, and to skip matching that bit of the sample against the ancestors. But we have the code to do that for ancestor matching anyway, so I assume it would be a reasonably easy change.

(modified to reflect the fact that we currently use tsinfer.constants.UNKNOWN_ALLELE == 255)

[hyanwong]

I'm happy to code this up, if @jeromekelleher gives it the go-ahead.

One other thing we would need to do is to adjust the generate_ancestors() step to account for missing values. I don't think this would be too difficult though.

[jeromekelleher]

There's a general issue about how to represent missing data in tskit over here. We'd want to use the same encoding here whichever way we do it.

But yes, definitely something we want to support; thanks!

[hyanwong]

Just some notes about my current implementation (at https://github.com/hyanwong/tsinfer/tree/allow-UNKNOWN_ALLELE-in-samples).

1. We need to allow constants.UNKNOWN_ALLELE in add_sites in the SampleData method, without having to specify it in the alleles array. This should work whether the value for constants.UNKNOWN_ALLELE is negative or positive. DONE, including tests
2. The ancestor building process needs to know about constants.UNKNOWN_ALLELE when deciding on consensus variants - this includes how to deal with them in the error buffer. PARTIALLY DONE
3. Any ancestral haplotypes that contain a break, such that the focal sites are separated by a region where none of the derived samples have data, should be split into separate ancestral haplotypes. (on consultation with Jerome, I think we don't need to do this. If the user presents us with samples that have missing chunks in the middle of an otherwise contiguous haplotype, we should simply assume that there is good reason to assume they are indeed on the same haplotype)
4. We need to check that the Li & Stevens process allows missing data, both (a) in the reference panel against which matches occur, and (b) in the haplotype which we are trying to match. NOT DONE
5. When matching the samples, we should only match in the region from the leftmost known allele to the rightmost known allele (unknown sites in-between are tackled by 2. above). DONE with tests, using the trim_region function
6. When mapping the inference sites back to original sites (in get_samples_tree_sequence()), we need to trim the edges in the produced tree sequence such that if there is an entirely unknown region to the left or right (no data for any of the samples), then there is no tree in that region. I think it should, however, be OK to have a tree in a leftmost region if there are non-inference sites there. DONE, including tests

[jeromekelleher]

Sounds great, thanks Yan!

[hyanwong]

Replaced with #224