Ensure source generator nodes keep track of removed entries

[jjonescz]

Originally posted by @chsienki in #66992 (comment).

There should probably be an invariant that the table size going in is the same as what comes out, before compaction. In practice this isn't true today, as we implicitly track empty slots, meaning tables can (and do) get shorter. We might want to think about changing that, as I think it's probably been the root cause of a lot of these removal bugs.

Related:

• SyntaxNodeProvider should keep track of removed items #67633
• Track state table removal due to input separately from removal due to selection #66643
• [17.6] Cache source generator node tables only if their state counts match #66992
• Preserve empty NodeStateTable entries #66963
• Fix inserting new values with source generators #68451

[jcouv]

@jjonescz I put this issue in 17.7, but since it sounds more like engineering debt than an observable issue, it can probably wait until next MQ. I'll let you decide on priority. Thanks

[jaredpar]

@jjonescz, @chsienki do we have any more work on this? There aren't any outstanding SG bugs that I'm tracking. So wanted to make sure this still had scenarios attached to it.

[chsienki]

@jaredpar this is really just tracking technical debt, as I think we should refactor the way we handle state tables.

I actually did a little bit of this when I was working on #68830 but haven't had time to pick it back up yet.

I'll assign it over to me and move it to the backlog for now.