Twitter download quantity incomplete issue.

[ee092884]

Describe the bug
A clear and concise description of what the bug is.

Expected behavior
Hi, I am using version 2.11.2.421 and found that the number of downloads on Twitter is incorrect.

For example: The total number of tweets by a user is 600, with a total of 200 media photos/videos/tweets (including 4 images in a single tweet).

If there are a total of 400 files including images, videos, and GIFs,the final downloaded files only amount to 100.

I have logged into my Twitter account in the download settings and checked all available options in the blog section for downloading.

After starting the download for some time, I noticed that the remaining space shown in the progress suddenly fills up instantly and completes the download.

However, there is a significant discrepancy in file quantity; it does not fully download.

Even after removing the downloaded user account and closing TumblThree before reopening and importing again for another download attempt, I still encounter the same result.

Is there any way to fix or resolve this issue?

Desktop (please complete the following information):

• TumblThree version: [e.g. 2.11.2.421]
• OS: [e.g. Windows 11 Pro]
• Browser [e.g. Chrome]
• Version [e.g. 117.0.5938.63]

Additional context
Add any other context about the problem here.

[thomas694]

Yes, there were still some flaws since the modifications for the last data structure changes.
Some bug fixes were still in the works. Please try the newest version.

[ee092884]

I tested the new version, and I noticed that the download scan page is fixed at 20. When I manually change it and add it to the download queue, it gets forcefully reset to 20. This scanning speed is too slow. Before encountering issues with downloading on Twitter, I used version 2.10.3.407. Despite facing incomplete downloads or downloads getting stuck at times (where re-downloading the same file would also get stuck), I found that deleting the problematic tweet and moving to the next one allowed for much faster downloads (more than 10 images per second), completely ignoring Twitter's limitations.

Additionally, regarding the file naming rules, I used the old version (2.10.3.407), and my rule was %x%b-%i-%e-%n. The downloaded file names would consist of the Twitter ID (display name, not numeric ID) - tweet ID - tweet timestamp - image number. However, in the new version, it only uses numeric IDs, which makes organizing and searching a bit cumbersome. Is there a way to address this?

[thomas694]

The maximum valid value is 20 (but setting lower values doesn't make much sense), actually they are then returning up to 20 posts depending on different factors. It also scans at the maximum speed possible. If it was too fast, the server enforces a limit ("waiting until"). Yes, old versions or the old API to be precise had higher limits. I doubt that you can get around their current limitations (without at least paying).

Regarding the file naming rules, yes, there was a wrong assignment of a field. Thanks for mentioning it. It should be fixed in the next version.
Wouldn't it be better to put %x further behind so that they are sorted by blog name in all cases?

[ee092884]

Regarding the issue of bypassing limitations, I didn't employ any methods to circumvent restrictions, nor did I subscribe to any Twitter premium memberships to increase API access limits.
What I did was use the latest version at the time (2.11.2.421) as a stepping stone.
I logged into Twitter to obtain a Cookies.json file and then utilized the older version (2.10.3.407) to download Twitter content.
This approach allowed me to avoid download limitations (including the 20-scan limit) because the older version (2.10.3.407) does not have an option to log in to Twitter.
Therefore, I used the new version (2.11.2.421) to obtain the Cookies.json file and then used the older version (2.10.3.407) for execution.

[thomas694]

Thanks, that made it more clear.

[ee092884]

That's great, and I appreciate your understanding.

I have another question regarding Twitter.
Twitter allows users to reply to their own tweets and continue posting images or videos in subsequent tweets.
Currently, the default options for downloading images and videos do not cover downloading all media items within a Twitter thread.
There is an option called "Download replies," but it is not selected by default in the settings.
This means that for each new download task, I have to manually check this option.
Is there a way to include this functionality in the settings so that I can customize the default selection?

[thomas694]

In the next version there will be a new option Download replies in the settings dialog.

[ee092884]

Regarding the issue of bypassing limitations, I didn't employ any methods to circumvent restrictions, nor did I subscribe to any Twitter premium memberships to increase API access limits. What I did was use the latest version at the time (2.11.2.421) as a stepping stone. I logged into Twitter to obtain a Cookies.json file and then utilized the older version (2.10.3.407) to download Twitter content. This approach allowed me to avoid download limitations (including the 20-scan limit) because the older version (2.10.3.407) does not have an option to log in to Twitter. Therefore, I used the new version (2.11.2.421) to obtain the Cookies.json file and then used the older version (2.10.3.407) for execution.

Recently encountered issues with version 2.11.4.435 during testing
Using this version, there is a +8 minute pause during the download process before it resumes
Also, each post is locked at 20

Previously mentioned that using version 2.10.3.407 allows for normal download speed unaffected by API limitations
However, there may be incomplete downloads and freezing issues
Can we use the download mode of version 2.10.3.407 in the next release? 🥹

[thomas694]

The "+08:00" refers to your time zone, the waiting time is the difference between shown and your current time.

Does the "old" download actually work for you? For the few blogs I tested it against, it downloaded no more than up to around 5% of the available contents, the rest wasn't even offered, so no chance to download it.