SQL - utilise JSON aggregations for relationships

[mike12345567]

Description

This PR addresses a number of issues with loading a large number of relationships in Budibase, primarily documented in: #13820.

We also have a linear ticket: https://linear.app/budibase/issue/BUDI-8311/rows-in-a-table-go-missing-when-multiple-relationships-are-attached-to which discusses the change this PR implements, moving to use JSON aggregations to retrieve relationships rather than joining them onto the main table.

The issue with our old method is that as the number of many relationships grows, the size of the response data grows exponentially. For example if a table has 10 many relationships attached to it, each with 1000 rows related to a row in the main table, this is not a response size of 10001 rows (1000 rows for each of the 10 relationships, and 1 row for the main table row) - it is somewhere in the region of a response size 1000 to the power of 10. This is an un-avoidable issue with joining, usually this would be avoided by simply not joining everything, but that is not how Budibase works, we need all relationship data available for the page so that we can calculate formulas (amongst other requirements, like relational filtering etc).

Even if we do not return the full result set we still end up in this scenario, as the sheer amount of data getting joined can cause the query to never respond. We see this when working with SQS, we can cause threads to stop responding with heavily related data.

This fix means that the main table never has any joins on it - all joins are singular and part of sub-queries. This means that the response time will always scale O(n) - with the number of rows that need joined nothing more. The sub-queries can retrieve a large number of rows from the related table (up to our hard limit) and then merge these using JSON aggregation functions into a single column per row, this works well as this is what Budibase would normally do with the data set anyway.

This also fixes some issues we have with pagination - when we limit the main query to 100, we will always get 100 rows from the main table if there is that many rows - this improvement helps implement some other features.

[samwho]

Went through this on a call with Mike just now, approval in principal. Just a few little things to look into that we've left in comments throughout.