Seeing spanner 20k mutation limit errors for inserts in production

[tublitzed]

I have a large collection of bookmarks(3078 bookmarks and 546 folders) that all of a sudden is failing to sync in production.

Sync id: 130387415. Via bob, I got the following from the production logs:

A database error occurred: RpcFailure(RpcStatus { status: RpcStatusCode(3), details: Some("The transaction contains too many mutations. Insert and update operations count with the multiplicity of the number of columns they affect. For example, inserting values into one key column and four non-key columns count as five mutations total for the insert. Delete and delete range operations count as one mutation regardless of the number of columns affected. The total mutation count includes any changes to indexes that the transaction generates. Please reduce the number of writes, or use fewer indexes. (Maximum number: 20000)") }), status: 500 }

Here's a link to a sample anonymized collection of this same size.

[tublitzed]

What's interesting here is that I hit the Spanner 20k limit with only ~3500 items...

[pjenvey]

The 20k here translates into 2000 items (1 mutation per each column. bsos table has 8 columns + 2 extra mutations for our secondary indexes).

It looks like the sync client will print the server's limits in its log, can you search your log for "max_total_records"? Let's confirm your client's seeing a value of "2000".

E.g. (from my log, probably non-spanner current sync prod):
1573781473132 Sync.Engine.Tabs TRACE new PostQueue config (after defaults): : {"max_request_bytes":2101248,"max_record_payload_bytes":2097152,"max_post_bytes":2101248,"max_post_records":100,"max_total_bytes":209715200,"max_total_records":10000}

[jrconlin]

Ah, I thought that it was one mutation per row too, not per column. That really changes how I view things. We pretty much want to reduce the number of columns to the bare minimum, but I get the feeling we'd get hit on data transfer then.

…

On Fri, Nov 15, 2019 at 9:30 AM Philip Jenvey ***@***.***> wrote: The 20k here translates into 2000 items (1 mutation per each column. bsos table has 8 columns + 2 extra mutations for our secondary indexes). It looks like the sync client will print the server's limits in its log, can you search your log for "max_total_records"? Let's confirm your client's seeing a value of "2000". E.g. (from my log, probably non-spanner current sync prod) 1573781473132 Sync.Engine.Tabs TRACE new PostQueue config (after defaults): : {"max_request_bytes":2101248,"max_record_payload_bytes":2097152,"max_post_bytes":2101248,"max_post_records":100,"max_total_bytes":209715200,"max_total_records":10000} — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#333?email_source=notifications&email_token=AAAIXKZMOMPKXR2JQNDZBMLQT3MCVA5CNFSM4JNMUJO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEGETYI#issuecomment-554453473>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIXK7TRNTFPEY4CTVVAVLQT3MCVANCNFSM4JNMUJOQ> .

[tublitzed]

@pjenvey confirmed that my max_total_records is getting set correctly to 2k:

{"max_post_bytes":2097152,"max_post_records":100,"max_record_payload_bytes":2097152,"max_request_bytes":2101248,"max_total_bytes":209715200,"max_total_records":2000}

[pjenvey]

Batch commit also deletes the batch upon completion, adding an extra mutation or two. So our max_total_records should probably be 1998.

We need a db_test ensuring this limit, it should create 2 batches: of the max size and max size + 1, ensuring commit returns what we expect.

If it's longer running it could be disabled by default w/ #[ignore] but we have no spanner on CI to ensure it doesn't fail over time :(

[tublitzed]

Would it be worth also making a change to make this value configurable without a deployment (ie, maybe set the default here but have it be able to be overridden with an env var or something like that?

That way if something changes that's outside of our control (ie, since this 20k limit is Spanner internals) we don't need to do a deploy to fix.

[jrconlin]

@tublitzed We made it a config value.
https://github.com/mozilla-services/syncstorage-rs/pull/319/files
I use a config file here because I couldn't figure out a way to specify the sub value as an environment var. I believe that @pjenvey noted a way that it could be done.

We also enforce the hard spanner limit (not as a config, because it's a hard limit)
https://github.com/mozilla-services/syncstorage-rs/pull/324/files