Skip to content

more streaming #126

@SgtPooki

Description

@SgtPooki

when we read and write files, and pass files from dataSource to deal and ipni strategy, we're passing the entire file around in memory. we need to enable streaming all the way through, or pass filenames and have each place that uses it read it from memory.

This will also be needed prior to moving to bull-mq or other job queue where we need to create data to upload, then upload job picks up, and then retrieval job picks up, and we have to validate it.

We should update/remove the DataFile interface and update all downstream consumers

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    📌 Triage

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions