LdaModel trains beyond size of corpus when using an iterable

[gochristoph]

Problem description

When streaming documents/bag of words to LdaModel via a custom iterable, LdaModel will train beyond the size of the corpus, with output like 19-07-05 22:53:43 PROGRESS: pass 0, at document #178000/50000 -- where the number left to the / is higher than the number right to it.

Steps/code/corpus to reproduce

from gensim.models import LdaModel
import logging
logging.basicConfig(format='%(asctime)s  %(message)s', \
    datefmt='%y-%m-%d %H:%M:%S', level=logging.INFO)

class TestIterable:
    def __init__(self):
        self.bag_of_words = [(0,2), (3,1), (6,1), (100,2)]
        self.cursor = 0

    def __iter__(self):
        self.cursor = 0
        logging.info('TestIterable() __iter__ was called')
        return self

    def __next__(self):
        if self.cursor < 50000:
            self.cursor += 1
            return self.bag_of_words
        else:
            logging.info('TestIterable() returned StopIteration')
            raise StopIteration


corpus = TestIterable()
# uncommenting this part will make a list out of the corpus
# corpus = [document for document in corpus]

logging.info('performing lda training')
trained_model = LdaModel(corpus, num_topics=2)

Using the TestIterable() will result in LdaModel training indefinitively. Converting the TestIterable() corpus to a list will lead to the expected result of a proper training.

I have not written too many iterables so far, and of course there could be a problem there. But as far as I could infer from the LdaModel documentation, all that is required is an interable -- and to the best of my knowledge, corpus = TestIterable() is a proper iterable, and iterator as well.

Thanks a lot!

Versions

Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
Python 3.6.4 (default, Apr 10 2018, 07:54:00)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
NumPy 1.14.2
SciPy 1.0.1
gensim 3.7.3
FAST_VERSION 0

[piskvorky]

Hm, that's pretty weird. Thanks for the detailed report.

I'll try to have a look what's going on, but probably not very soon, sorry :(

[gochristoph]

Hi Radim, thanks a lot -- looking forward to your take on it!

[Hiyorimi]

Is this still available?

[piskvorky]

@Hiyorimi yes! Sorry to @gochristoph , I never got around to this.

[Hiyorimi]

@mpenkov I believe that just getting the list won't do the trick, since it will slow down the training ?

[mpenkov]

Perhaps you want to iterate and return the first element instead of converting to a list?

[piskvorky]

@Hiyorimi the issue here is that LdaModel is reported to train indefinitely, on the given TestIterable object.

I can't imagine why it would do that – LdaModel only uses the iterable via the standard Python for x in iterable: … construct, IIRC. So I suspect the issue is with the iterable, not with LdaModel. But I can't see anything wrong at a glance, so the next step will be to replicate the issue and investigate.

It has nothing to do with converting to a list, that's just a way of debugging the iterable.

[Hiyorimi]

I added simple test case in separate branch.

Problem is that here we get wrapped chunk as :

wrapped_chunk = [list(itertools.islice(it, int(chunksize)))]

And called on TestIterable (or ProblematicIterable in tests), it will iterate forever.

It happens because TestIterable sets self.cursor on iter call. Being perfectly valid for getting sequence length, this setting triggers infinite chunk generation, once islice() being called on iterator.

I am not sure if that it is the valid case for LDAModel, probably @gochristoph can clarify, if that resolves the matter.

I can think only on one simple check, which is suggested in PR #2633