Skip to content

Conversation

@jvwong
Copy link
Member

@jvwong jvwong commented Jan 6, 2022

This update attempts to make creating files with Biofactoid data and downloading them more efficient.

  • File downloading
    • The /api/document/zip and /api/document/biopax/zip endpoints just serve file(s) that are already created
  • File creation
    • Listens for database events that either create or modify documents with status 'public'
    • Events fire off a request to exportToZip as before to create the required file
    • Requests for file creation are delayed ad batched within a specified time window (EXPORT_BULK_DELAY_HOURS)

Refs #1032

@jvwong
Copy link
Member Author

jvwong commented Jan 6, 2022

  • File creation on app reboot

Copy link
Member

@maxkfranz maxkfranz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some notes

const MS_PER_SEC = 1000;
const SEC_PER_MIN = 60;
const MIN_PER_HOUR = 60;
const baseUrl = `http://localhost:${PORT}`; // ever not localhost?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really needed anyway, since you could call the corresponding function directly instead of going through the route. It works either way, though going the routes could slow things down.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense - this requires extracting the logic from the route handlers:

  • /api/document/ and also
  • /api/document/biopax,
  • /api/document/sbgn
  • /api/document/text

I'll try, but if it gets hairy I'll just leave it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Refactoring converter functions
    • Stripped out the logic from the endpoints and reused them directly, avoiding the web service routes.
  • Fixed up the behaviour on boot:
    • When NODE_ENV is production, always create exports
    • When NODE_ENV is not production (e.g. dev), create the export zips if any are missing (so the web service routes work), otherwise do nothing

@metincansiper
Copy link
Contributor

Looks good to me. Just wondered one thing why there is a delay shift for biopax exports (https://github.com/PathwayCommons/factoid/pull/1034/files#diff-b0e82ba0bf6ce7365597b983e42f14dcca6f038f4297df4c1f811a28f88d035dR190)? Is there a specific reason of why you want to start biopax exports after bulk exports?

@jvwong
Copy link
Member Author

jvwong commented Jan 26, 2022

Looks good to me. Just wondered one thing why there is a delay shift for biopax exports (https://github.com/PathwayCommons/factoid/pull/1034/files#diff-b0e82ba0bf6ce7365597b983e42f14dcca6f038f4297df4c1f811a28f88d035dR190)? Is there a specific reason of why you want to start biopax exports after bulk exports?

I will get rid of that, probably overthinking things.

@jvwong jvwong merged commit ba5fb2d into unstable Jan 26, 2022
@jvwong jvwong deleted the iss1032_export-on-change branch January 26, 2022 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants