TikTok archiver (ttarc)

NOTE: As of 2020-04-23 5PM CET, the trending endpoint returns an empty list only. Hence, as is, this program does not fetch any data.

---

Inspiration: tiktok-feed by CorentinB.

Minimalistic TikTok archival tool using https://m.tiktok.com/node/share/trending.

Collects the trending JSON document and linked (mp4) video clips and puts them into WARC via wget (1.14 or higher).

Install

$ go get github.com/miku/cmd/ttarc/...

or use some Linux packages (there's an armhf version for an SBC)

Run

$ ttarc
$ ls -lah
...
ttarc-trending-20200312214721.cdx
ttarc-trending-20200312214721.warc
...

This can be put into cron, e.g. to be run every 15 minutes.

*/15 * * * * /usr/local/bin/ttarc -P /tmp/ttarc -log /tmp/ttarc/ttarc.log

Usage

Usage of ttarc:
  -P string
        output directory (default ".")
  -b    ignore wget errors, just log them
  -f string
        basename for warc file (default "ttarc-trending-20200312224313")
  -log string
        log to stdout, if empty
  -ua string
        user agent (default "Mozilla/5.0 (Windows NT 10.0; ... )
  -verbose
        be verbose
  -version
        show version and exit

TODO

• broader scope
• optimal tile
• save user profiles, e.g. anna
• the discover list
• cycle through top 5000 english words and make a frequency list
• video id, 6840824663585639686

Stills

Extracted via warcio, ffmpeg and imagemagick.