Twint doesn't get all followers list

[mmosleh]

It seems Twint doesn't get the list of all followers for accounts with large number of followers and stoped abruptly at some random number. For example tried twint -u nasa --followers and each time the script stopped at some random number with few thousand screen_names.

• Python version is 3.6;
• Updated Twint with pip3 install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;
• I have searched the issues and there are no duplicates of this issue/question/request.

[pielco11]

It could be possible that Twitter stops returning new entities because you (as everyone else) in that case requested too many queries.

[mmosleh]

Thanks @pielco11 , I was wondering what is the way around it. e.g., If it raises an error and it can continue pulling the information from that point. or controlling the requests within some limit so that doesn't happen. thanks

[pielco11]

I'll look deeper (can't determine a dead-line), for now I can say that the issue does not seem to have an unique pattern. I
tried a couple of queries and got a long list, ~100k users. Plus when one stopped I started a new one, and this lasted a long. So I guess that's not Twitter that's blocking you, for what I tested I think that using a VPN will not get you around the issue.

A solution could be to re-try the query when it fails, anyway the code should be changed after a deeper look at what-is-going-on

[mmosleh]

Thank you @pielco11 !

[castrovictor]

I'll look deeper (can't determine a dead-line), for now I can say that the issue does not seem to have an unique pattern. I
tried a couple of queries and got a long list, ~100k users. Plus when one stopped I started a new one, and this lasted a long. So I guess that's not Twitter that's blocking you, for what I tested I think that using a VPN will not get you around the issue.

A solution could be to re-try the query when it fails, anyway the code should be changed after a deeper look at what-is-going-on

Hi, first of all, thanks for doing such amazing tool and public the code, I am sure I will learn a lot from your work. I tried getting a long list, ~130k and it stopped in random number of followers in each query.
On the other hand, I am making a script to get all tweet links of a user, because I think your tool does not do it. Without log in and without using the API, but in some way, after doing a lot of querys (with my script), twitter blocks your user's tweets search. Using an VPN, the problem was solved. This is just to give you some information.

Finally, If you have a PayPal account, I would like to buy you a coffe for posting the source code, because as I said, I would like to learn who do you did the tool, which would be impossible without the source code.

[pielco11]

~130k followers are a lot so Twitter might be blocking requests at a random time

For the second point, that's why Twitter blocks an IP if it makes too many requests, that's why using a VPN solves the problem.

What we could try is handling that "followers count" issue and ask the user to change the IP and then retry the query, and see if this solves the issue.

Unfortunately I do not have enough time to solve every issue, so the patch will be delayed. Every kind of help in the development is widely accepted

[pielco11]

@mmosleh Here is what's going on

In the first case there is a show more, Twint extracts that link and does a new request. Then that button vanishes so Twint is not able to make a new request.
If I get the last cursor-id and make a new request changing the IP and stuff, nothing changes

I think that we found the origin of the issue and sadly we can't do anything, at least for now

[mmosleh]

@pielco11 I made a quick dirty patch into the previous version of Twint (the one with a single file). Just few retrial on the last curser-id when receive the error massage. I managed to download all 32M NASA followers this way. (I'm not familiar the code base on the new version though)

[pielco11]

@mmosleh oh, nice... may you provide me the commit id? git rev-parse HEAD