Chop off edges before first site and after last site

[awohns]

Material before the first site and after the last site is uninformative so we should probably prevent edges from spanning those regions. This can be an issue for downstream analyses when telomeres are large (in practice we manually remove these regions with tskit.keep_intervals anyway).
Making this automatic in tsinfer involve minimal changes and prevent downstream problems

[hyanwong]

I'm not sure we should do this by default? E.g. it seems weird to have missing sections before and after in cases where we're only running on a fraction of the chromosome. I guess we could have a flag if necessary. I think it might need a bit of thought - after all, people won't always be running tsinfer in the same way we are doing.

[jeromekelleher]

Seems to me like it's more likely to cause errors by leaving these long, meaningless trees at the start and end of the chromosome. This sort of stuff has repeatedly caught people out. I'd rather people be a bit confused than have wrong answers.

[hyanwong]

I'm not sure if your comment above is voting for removing them by default or not @jeromekelleher ? A more complex thing might be to remove them (should we also issue a warning?) if they are, say, greater than 10 times the average distance between sites? That's a bit more complicated, but it seems reasonable to have a parameter with None implementing this by default:

param bool omit_ends: if True, do not extend trees before the first (inference?) site or after the last.
    If ``None`` (default), omit ends when they are 10 times the average distance between (inference?) sites

[jeromekelleher]

I'm strongly advocating for chopping off the ends, and not making it an option. There's no good reason to keep these meaningless trees at the ends of chromosomes, I think. I don't think we should try to make it clever.

[hyanwong]

I'm strongly advocating for chopping off the ends, and not making it an option. There's no good reason to keep these meaningless trees at the ends of chromosomes, I think. I don't think we should try to make it clever.

Hmm. OK. I'm not so sure. It might seem odd for e.g. viral genomes or other genomes that don't have telomeres per se. As a user, I would not immediately expect to have blank sections at the start end end of my sequence, if I'm only inferring a snippet from the middle of the chromosome (which I suspect will be a common use-case). Once it was explained to me, I guess I would be OK with it. But it would be a surprise, initially.

Either way, I strongly suggest that we want to keep non-inference sites to the right of the last inference site and the left of the first inference site. If so, we would be doing some extending from the inference sites by default anyway.

[jeromekelleher]

As a user, I would not immediately expect to have blank sections at the start end end of my sequence, if I'm only inferring a snippet from the middle of the chromosome (which I suspect will be a common use-case).

Surely it's even more important in this use case - we want to have missing data on the bits of the chromosome where trees haven't actively been inferred. Branch-length stats computed on the trees-spanning-the-whole-genome (but actually only referring to a little chunk in the middle) would be totally different to the trees for the region that we've inferred. This would definitely catch people out - we've been stung by the same issue repeatedly in stdpopsim.

Either way, I strongly suggest that we want to keep non-inference sites to the right of the last inference site and the left of the first inference site. If so, we would be doing some extending from the inference sites by default anyway.

Keeping non-inference sites seems fine - the user has told us to put these sites into a tree, and to use parsimony to map them on. Extending the first/last tree is the only reasonable way we have to do this.

[hyanwong]

Ah, ISWYM: that is a good point. I was assuming people would trim a chromosomal fragment down, but then they would lose the actual positions, so we might want to encourage people not to trim.

So I take it back - removing the ends is perhaps the behaviour we want be default, although I do think we should make it very clear in the docs, and perhaps we should have a way to extend the TS to cover the ends, if required? I can't think of any way to do this without simply hacking the tables.

[hyanwong]

Chatted to Jerome about this. He pointed out that the way to preserve the entire TS is to place an invariant site at the start and end of the genome. So as long as this is documented (and presumably tested) I think it would be fine to shorten the leftmost and rightmost edges by default.

If the rightmost site is the last inference site, and creates a new tree, how long should the new edges be in that tree? Are we assuming integer site positions here, in which case I assume the new edges should be of length 1?

[hyanwong]

Is this a task to add to the the new post_process routine, do you think?

[jeromekelleher]

Agreed, let's get it in.

[hyanwong]

What do we do for the last tree, if it only spans the last site? Presumably we won't have an infinitesimally short tree? I assume we make the right position of the last tree min(last_pos + 1, sequence_length)?