Limitation with Parameter 'k'

[vietpham1911]

I am using ColBERT for passage retrieval in my question-answering system and I have encountered a limitation that I would like to understand better:

I have set the parameter k to 1000, for getting the top 1000 passages. However, I have noticed that only the top 64 passages are returned instead of the expected 1000 even though ColBERT was going through over one million passages. Could you please help me understand the reason behind this limitation? You also can see it in the picture below.

[okhat]

This is indeed really weird. It should return more. But I’m guessing you may have to tweak the parameters ncells (and others) in the notebook. Increase ncells to 8 or even 16.

[vietpham1911]

Now, it is returning the top 1000 passages even though I did not change anything... Thanks for your reply!