Zip bulk download: Recreates new bulk file on each request / Gateway Timeout

[jvwong]

We advertise bulk data downloads via /api/document/zip. However, I noticed that:

1. Browser responds with a Gateway Timeout error when visiting the link
2. Looking at the cached file in the factoid container at downloads/factoid_bulk it appears to be recreated each time a request is made

This creates a barrier to data download.

• Cannot reproduce on unstable instance (works fine here)
• Able to reproduce on beta instance: Gateway Time-out (1); cannot reproduce (2)

---

Aside:

• I wonder if this endpoint is being hit by bots/Google
• We should probably not generate this file on the fly, other optimizations

[jvwong]

With regards to the recreation, the following

factoid/src/server/routes/api/document/index.js

Line 631 in 3e5c230

let fileDate = fs.statSync(filePath).birthtimeMs;

should actually reference the time the file was modified (mtime) rather than the file creation time (birthtimeMs), which is a fixed time (in the past).

[jvwong]

Aside from above bug regarding recreate time stamp, the /api/document/zip endpoint doesn't provide an HTTP response until it's finished dumping the bulk data - even with some optimizations, this takes a while (couple of minutes).

One idea is to have this endpoint instead:

• serve existing files
• ensure the files
  • exist on boot / db restore
  • are updated on some regular interval 'offline' (e.g. cron), not lazily

Also should not expose any public route that would trigger a dump, or disable bots from hitting any such endpoint