Out of memory error when querying large data set with chunked=True

[rmah]

Making large queries for very large result sets results in out of memory errors. This can be traced back to the function _read_chunked_response in client.py which reads the entire response into RAM before returning a ResultSet.

[psy0rz]

I can confirm this. Although chunked mode is supported, its completely useless this way.

They even went to the trouble of returning a generator, but its no use since the whole response is read into ram anyway.

I need this to process millions of measurements in chunked mode, so I expected it to return some kind of generator which i could then iterate while processing the data.

[psy0rz]

First step in solving this is to add stream=True when calling self._session.request().

Now it makes it all the way to _read_chunked_response(), in which case response.iter_lines() will now be a generator.

[psy0rz]

Also found a extra problem when using DataFrameClient.query(): This also uses InfluxDBClient.query(), HOWEVER, for some reason it then calls self._to_dataframe() with the result set, which in turn processes all results in memory instead of returning some kind of generator.

[psy0rz]

ok i think i fixed it, will make a pullrequest.

[goriccardo]

Any update on this?

[pedrofaria09]

Hello, any update on this? It's impossible for me to query 100.000.000 rows.

[psy0rz]

Just add the change from my pull request manually: #538

Someone needs to create a test for it and update the pull request so the devs can accept the change. (i dont have time for it)