Json.Document and Table.GenerateByPage slow performance/alternatives

[xli1000]

We have a Power BI connector that receives JSON data page-by-page. We see that performance in terms of rows of data loaded per second lags behind that of an ODBC driver very significantly. 

In particular, there were a couple of performance bottlenecks that I identified:

• It takes about 1.4 seconds to parse a 30MB JSON object with Json.Document. (I tried to isolate this by running Json.Document on a static JSON file.) By contrast, it only takes a small fraction of a second to do the same in Node.js.

  • Is there any faster way to parse JSON (maybe we're calling it the wrong way?) or can the implementation of Json.Document be potentially improved if it's an inherent limitation?

• Furthermore, we used the Table.GenerateByPage sample code to combine pages of data together. However, this seems to take about 2.4 seconds to add each ~30MB page of data to the combined table, which seems very slow. (I tried to isolate this by running Table.GenerateByPage on dummy data.)

  • Could there be any other way to combine tables together in Power Query that still works for very large datasets? We can also use Table.Combine, but it seems to slow down greatly/fail for 2-3 GB of data.