-
Notifications
You must be signed in to change notification settings - Fork 8
Open
Milestone
Description
when we read and write files, and pass files from dataSource to deal and ipni strategy, we're passing the entire file around in memory. we need to enable streaming all the way through, or pass filenames and have each place that uses it read it from memory.
This will also be needed prior to moving to bull-mq or other job queue where we need to create data to upload, then upload job picks up, and then retrieval job picks up, and we have to validate it.
We should update/remove the DataFile interface and update all downstream consumers
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
📌 Triage