ChunkTeacher no longer requires num_examples/num_episodes to be correct

[stephenroller]

Patch description
For as long as we've had ChunkTeacher, it's required that the number of episodes and examples be exactly reported correctly, or else it would cause a hang. With this patch, we loosen that requirement.

Methodology:

• Upgrade some of ChunkTeacher's internal logic to use DatatypeHelper to check about shuffling/cycling. Some of this is equivalent but more explicit and makes a bit more sense. The one logical change here is train:ordered was previously cycling, when it shouldn't.
• FixedDialogTeacher keeps track of the current episode and sees if we'll go over our limit, and starts emitting padding. Modify ChunkTeacher to always lie about its "next episode" so that check never triggers.
• Epoch done was previously computed in FixedDialogTeacher.next_example, checking the counts. Override this behavior in ChunkTeacher to have it simply test whether it's seeing a pad or not. This means we'll go one batch further than we need to, but that's okay -- everything filters the pads correctly and it doesn't affect counts or examples seen.

Testing steps
Manual testing.

New CI tests that stand up chunk teachers that lie about their number examples/episodes, and check that the number of unique messages seen is exactly correct.

[emilydinan]

awesome. should we require a user to implement num_examples now? (setting it as an abstractmethod requires the implementation)

[emilydinan]

I assume this works OK with dynamic batch padding, since that occurs at a level higher than the teacher?

[stephenroller]

Yes, this has been tested with all variants of batching and ensured that all examples are seen.

[stephenroller]

You still have to implement num_examples, as that determines the behavior of -eps and -veps etc. It just doesn't have to line up with data

[klshuster]

awesome