detect unsynchronized contract table and retry

[S11001001]

After the websocket DB query, we check the offset table to ensure that it's consistent with what we just wrote to it in the fetch-and-persist phase. If it isn't, we try again. (We can always use the offset that we just wrote in our final response body, because if it doesn't match, we're going to use the new offset and re-fetch anyway.)

We try 10 times, and if we still don't get consistent results, we fail the WS request. I haven't explored more complex backoff strategies here; the focus of this PR is getting a minimal update that is least likely to livelock (and thus exhaust all retries).

We could also use the fetchAndPersistBracket function for the non-WS queries. I'm not sure that's necessary, though.

Fixes #10334. See that issue for a longer explanation of the possible race conditions this PR addresses.

CHANGELOG_BEGIN
- [JSON API] Under rare conditions, a multi-template query backed by database
  could have an ACS portion that doesn't match its transaction stream, if
  updated concurrently.  This conditions is now checked and accounted for.
  See `issue #10617 <https://github.com/digital-asset/daml/pull/10617>`__.
CHANGELOG_END

• get rid of XTag/XX unless I need it for a different minimumViableOffsets approach

Pull Request Checklist

• Read and understand the contribution guidelines
• Include appropriate tests
• Set a descriptive title and thorough description
• Add a reference to the issue this PR will solve, if appropriate
• Include changelog additions in one or more commit message bodies between the CHANGELOG_BEGIN and CHANGELOG_END tags
• Normal production system change, include purpose of change in description

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

[stefanobaghino-da]

How will this impact performance? Is there some visible effect on existing performance tests?

[akshayshirahatti-da]

what should be the behaviour for an unsyncedParty which is not a queriedParty and has unsyncedOff < expectedOffset ?

[S11001001]

We can safely ignore it. This is the test "not require update for lagging, but unqueried, party".

[S11001001]

I've been thinking of it as an optimization, but in reality it's a must-have. If an unqueried party is lagging, we don't have the token to update it, so if we decided to update based on that, we'd get stuck in the loop and exhaust retries.

[akshayshirahatti-da]

again slightly confused on how maxLocalOff being < expectedOffset is considered as caughtUp

[S11001001]

We have the invariant that every queriedParty is already at least expectedOffset, but it may not be in the map. Therefore, if the condition you mention holds, then maxLocalOff must have come from an unqueried party, which is fine as stated in the prior comment.

[S11001001]

Put another way, we care only that the queriedParty is caught up.

[akshayshirahatti-da]

functionally looks fine to me, thanks for explaining 👍

[akshayshirahatti-da]

IIUC expectedOffset should come from one of the queried parties right ? so is this scenario ever possible ?

[S11001001]

expectedOffset is only "what we just wrote"; it never comes from reading. So this can happen if all these template-IDs/parties got updated from 3 to 5 after our own fetch.

[S11001001]

How will this impact performance? Is there some visible effect on existing performance tests?

@stefanobaghino-da There's no visible effect on AsyncQueryConstantAcs between 9641fd5 and 5da2571. The ContractDao benchmarks don't use ContractsFetch so have no code changes.

A couple things on concurrent updates:

1. If we rerun, we don't rollback; we read forward as if we were just starting the request. So successive retries won't redo any fetching that was already done in previous tries.
2. If you want to follow the common (no concurrent update) case while thinking about the code, it's as follows:
   1. Queries#unsyncedOffsets returns a map whose keys are all the template IDs and values are empty maps; the query it does has no rows.
   2. In minimumViableOffsets,
      1. partyOffs always has only unqueried parties;
      2. maxLocalOff is always expectedOffset;
      3. caughtUp is always true by the first branch;
      4. lagging is always Some, but minimumViableOffsets will return None.
   3. In the Lagginess append,
      1. offA === offB;
      2. nc, ncA, and ncB are empty;
      3. c accumulates all the surrogate template IDs we're querying.