Description
Describe the bug
When extracting a very large instance of the table_azure_storage_blob
table, using steampipe export with jsonl
output, it loads the entire table into memory and then streams the result to the output instead of streaming it directly.
See here, it waits for the sub goroutines to complete before streaming:
Steampipe version (steampipe -v
)
Not applicable, can use latest I guess.
Plugin version (steampipe plugin list
)
v1.1.0
To reproduce
Run: steampipe_export_azure --config $(cat azure.spc) --output jsonl table_azure_storage_blob
Preferably run this connected to an account with a large amount of blobs, to see the memory effect.
Look at the memory usage of the steampipe_export_azure process.
Expected behavior
Stream intermediate results to the output directly as done in AWS for example:
https://github.com/turbot/steampipe-plugin-aws/blob/4089059d34062638fbda70b24986f99e7bfee112/aws/table_aws_accessanalyzer_finding.go#L248
Additional context
Add any other context about the problem here.
Thank you for the help!