Next.js app that finds deleted tweets that have been archived by the wayback machine
- Install Node.js and npm
git clone https://github.com/kulgg/tweetfisher.git
cd tweetfisher
npm install
npm run dev
Initially, I wanted to perform all requests on the client-side to avoid throttling problems and allow for easy hosting for users. However, I soon realized that this was not possible. Both archive.org and twitter.com do not set the required CORS headers, making it impossible to access response contents from the frontend code, even when using the 'no-cors' mode for requests. Furthermore, the HTTP HEAD requests to twitter.com that check for deleted tweets must imitate search engine headers to function properly, but this cannot be done in frontend requests as the browser adds additional headers.
As a solution, all requests have been wrapped by Next.js API routes. Unfortunately, this makes it difficult to host this app for widespread use due to throttling constraints.
- Consider account state (Active, Inactive, Suspended)
- Include all possible archived tweets
- Parse quote tweets
- Adaptive throttling
- Backend should request multiple times incase of failure
- Do requests concurrently with ability to set max TPS
- Fetch deleted tweets parallel to status checking process
- Show deleted tweets fetching process
- Support parsing of Jul 2022 and later version
- Use throttled queue instead
- Fix emojs in tweets
- Support image parsing
- Support "replies to"
- Fix parsing bugs
- Add scrollup button
- Add popon header
- Add throttle settings PopUp UI
- Finish sticky header
- Validate that urls are of an actual status
- Queues should instantly use updated TPS values
- Fix Index state mess
- Fix timeout useEffect queue (ongoing fetches are not invalidates when a new username is entered)