Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache ingested scratch DB #307

Closed
neilotoole opened this issue Aug 22, 2023 · 1 comment
Closed

Cache ingested scratch DB #307

neilotoole opened this issue Aug 22, 2023 · 1 comment

Comments

@neilotoole
Copy link
Owner

Is your feature request related to a problem? Please describe.

Consider a non-SQL source, such as an Excel file. When sq accesses that source (e.g. sq inspect @excel), it ingests the data into the scratch DB on each command. For large files, this can take some time... seconds, or even minutes. If the file doesn't change between commands, this is wasted work.

Describe the solution you'd like

sq could ingest the data the first time, e.g. into a SQLite scratch DB in /tmp. On the next command, if the source file hasn't changed, sq could skip the ingest step, and use the already populated scratch DB.

neilotoole added a commit that referenced this issue Jan 15, 2024
- Support for ingest cache, download cache, and progress bars.
@neilotoole neilotoole self-assigned this Jan 29, 2024
@neilotoole
Copy link
Owner Author

This is implemented in sq v0.47.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant