Skip to content

Inefficient memory usage for table_azure_storage_blob #854

Open

Description

Describe the bug
When extracting a very large instance of the table_azure_storage_blob table, using steampipe export with jsonl output, it loads the entire table into memory and then streams the result to the output instead of streaming it directly.
See here, it waits for the sub goroutines to complete before streaming:

Steampipe version (steampipe -v)
Not applicable, can use latest I guess.

Plugin version (steampipe plugin list)
v1.1.0

To reproduce
Run: steampipe_export_azure --config $(cat azure.spc) --output jsonl table_azure_storage_blob
Preferably run this connected to an account with a large amount of blobs, to see the memory effect.
Look at the memory usage of the steampipe_export_azure process.

Expected behavior
Stream intermediate results to the output directly as done in AWS for example:
https://github.com/turbot/steampipe-plugin-aws/blob/4089059d34062638fbda70b24986f99e7bfee112/aws/table_aws_accessanalyzer_finding.go#L248

Additional context
Add any other context about the problem here.

Thank you for the help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions