More flexibility with syncPageSize on DataStore

[TheMoums]

Is your feature request related to a problem? Please describe.
I'm using DataStore with 30 models. I have several models that contain a lot of information on many records (e.g. 25 variables and 10.000 records). But I also have a lot of models that contain very few information on many records (e.g. 4 variables on 10.000 records).

When syncing to the cloud, I get the following error: "ErrorType: Mapping template", "Message": "Transformation too large".
This indicates me that one of the syncing queries exceeded the 1MB limit provided by appsync.

The actual solution for this, is to reduce the syncPageSize of DataStore so that the syncing queries, fetch less data and thus have less chance to exceed that limit. This means that instead of fetching my records 1000 by 1000, I need to fetch them now 100 by 100.

This works, but it has a major performance drawback. Because of one or two tables making this limit being exceeded. I now have to fetch items 100 by 100 on every table, even on those with very few variables where queries 1000 by 1000 work perfectly fine.

Describe the solution you'd like
A solution would be to be able to configure the syncPageSize model by model instead of one value for all models. This would allow people like me, to set the limit to 1000 for tables with few information and to reduce it to 100 for high information tables.

Describe alternatives you've considered
Other possibilities but much more difficult to implement

• Calculate this syncPageSize dynamically to always return the maximum number of records possible.
• Increase the Appsync hard limit so that this problem almost never occurs again.

[meducati]

I agree to this.

The other way around (while down-syncing), if I cross the 1Mb limit, the Datastore initialization fails silently and my app halts. ("ready" never fires).

In the best of words, Datastore should catch this issue, dynamically lower the syncPageSize so that the initial Datastore sync finishes. At least I would like an "error" Event if "ready" is not an option...

[gorbinphilip]

Per model configuration itself would be nice.

[isaac-rosterlab]

This would be nice. A per model syncPageSize would be beneficial to speeding up our app while preventing failures due to crossing the 1MB app sync limit.

Dynamic would be even better where I don't have to think about it at all, but this would be a good compromise.

[PaulCailly]

+1 for per model syncPageSize. We're experiencing the same performance issues because we have one model that forces us to set syncPageSize to 10.

[adplant]

Agreed. By the way, what is the default syncPageSize if you don't provide a custom one?

[danrivett]

We experienced this today after pushing out a data update. We have one model that is significantly larger than the others due to its denormalized nature and number of records in that table. We only have about 100 records in that table so, tuning down syncPageSize will have an oversized effect as we'll need to tune it down significantly.

Therefore it would be great to tune down syncPageSize just for the model affected, and not all the other models as we have close to 30 total models currently.

[kolodi]

This is very required feature. Once you have a model with nested data that can grow substantially for a particular model it should be possible to treat it differently. I would suggest making syncExpression 9maybe rename them) to return complex object with not only the expression, but the model specific config, including maxRecordsToSync and `syncPageSize'. Then you should be also able to turn off completely the syncing for certain models as mentioned in here: #13264