Speed up the creation of worksheets with a worksheet template

[xispa]

Description of the issue/feature this PR addresses

Creating a worksheet from a worksheet template takes an unreasonable amount of time on big instances, specially with templates with a high number of slots. Four things are responsible for that.

1. Every unassigned analysis of the system is woken up. The search for candidates in applyWorksheetTemplate is unbounded, and the object is resolved before checking whether there is a slot left for its sample:

analyses = api.search(query, ANALYSIS_CATALOG)   # all unassigned analyses
for analysis in analyses:
    analysis = api.get_object(analysis)          # <-- always woken up
    ...
    if len(available_slots) == 0:
        continue

With a backlog of a few thousand unassigned analyses matching the services of the template, thousands of objects are loaded from the database to fill a few dozen slots.

2. The worksheet is fully reindexed once per analysis added. Part of the metadata of a worksheet is resolved by walking through all the analyses assigned (getNumberOfQCAnalyses, getNumberOfRegularAnalyses, getNumberOfRegularSamples) and getProgressPercentage runs a catalog search. Reindexing on each addAnalysis makes the whole operation quadratic.

3. The sample is fully reindexed twice per analysis added, once by the after_assign event of the analysis and once explicitly by addAnalysis. A full reindex of a sample recomputes every index and metadata column, including the searchable text index and the columns resolved by walking through all its analyses, while assigned_state is the only value that actually changes.

4. The QC analyses recompute the worksheet metadata one by one. Both add_reference_analysis and add_duplicate_analysis end with self.reindexObject(idxs=["getAnalysesUIDs"]). That call restricts the indexes, but update_metadata defaults to 1 all the way down to Products.ZCatalog, so every metadata column of the worksheet is recomputed on every single control, blank and duplicate created. Same quadratic pattern as (2) on a different path, and templates with 64 positions reserve a good share of them for QC.

Current behavior before PR

Measured on a test instance (tiny catalogs, everything in the ZODB cache, so the reindexing costs here are a fraction of what they are in production). The first scenario assigns 40 analyses, the second one 10:

Backlog / routine slots	Sample reindexes	Worksheet reindexes	ZODB loads	Time
60 samples / 20 slots	80 (full)	41	1225	1.22s
200 samples / 5 slots	20 (full)	11	1499	0.42s

For the QC analyses, the full recomputes of the worksheet metadata grow with the number of controls, blanks and duplicates created: 6 of them for a slot with 3 duplicates.

Desired behavior after PR is merged

Backlog / routine slots	Sample reindexes	Worksheet reindexes	ZODB loads	Time
60 samples / 20 slots	41 (targeted)	1	1176	0.84s
200 samples / 5 slots	11 (targeted)	1	914	0.32s

• the sample of the analysis is resolved from the catalog metadata first, and the object is only woken up when it can actually be assigned. The number of objects loaded no longer grows with the size of the backlog, but with the number of analyses that are actually assigned
• the worksheet is reindexed once, when the template is applied. addAnalysis accepts a reindex parameter for that, and addAnalyses relies on it too, so adding a collection of analyses without a template benefits as well
• addReferenceAnalyses, add_reference_analysis, addDuplicateAnalyses and add_duplicate_analysis accept a reindex parameter as well. The plural ones reindex the worksheet once, when all the analyses are added, and the template application skips it, cause it reindexes the worksheet at the end anyway. The defaults preserve the current behavior, so the views that add blanks, controls and duplicates, as well as the retract event of a reference analysis (that calls add_reference_analysis directly), need no changes
• after_assign restricts the reindex of the sample (and of its ancestors) to assigned_state. The idxs parameter of reindex_request was silently ignored, it is honored now
• getAnalyses is no longer used to keep track of the analyses assigned, neither in addAnalysis nor when adding QC analyses, cause it wakes up all of them on every call. UIDs are enough
• the reference samples that suit each slot are resolved once per reference definition instead of once per slot, and the services of the template are no longer resolved to objects just to get their UIDs

This PR also adds a new function to the API, api.reindex, used by the changes above:

api.reindex(obj)                                     # reindex everything
api.reindex(obj, idxs=["assigned_state"],            # this index and this
            cols=["assigned_state"])                 # column only
api.reindex(obj, idxs=["assigned_state"], cols=[])   # index only

Only the indexes named in idxs are reindexed (all of them when None), and catalogs without any of them are skipped. cols does the same for the metadata: all the columns are recomputed when None, only the named ones otherwise, and none at all when empty. Note this is not what obj.reindexObject(idxs=[...]) does, that restricts the indexes but recomputes all the metadata columns. The uid_catalog is taken into account too, resolving its record with the path convention of the content type (relative to the portal root for AT contents, absolute for DX ones), so no duplicate record is left behind. The function is covered in API.rst.

The WorksheetApplyTemplate doctest asserts now the metadata of the worksheet record after applying a template, and after adding analyses, duplicates and reference analyses in bulk, which was not covered before.

Three notes for reviewers:

• the modification date of a sample is no longer bumped when one of its analyses is assigned to a worksheet, cause the sample object itself is not modified anymore. This also keeps the sample out of the write set of the transaction, reducing the chance of conflict errors
• the re-sort of the slots for new samples was a no-op that, because of a leaked loop variable (samples_slots[sample_uid] instead of the sorted sample_id), could assign the slot of one sample to a different one, ending up with two samples sharing a slot. It has been removed: slots keep the order in which the analyses are resolved, by priority
• duplicate analyses are created inside the worksheet, so Dexterity's reindexOnModify subscriber still reindexes it fully on every duplicate created. Traced with 3 duplicates, the full recomputes of the worksheet metadata go from 6 (one explicit and one implicit per duplicate) down to 4 (one implicit per duplicate plus the single explicit one at the end), so this path is improved, but not linear yet. Batching that event-driven reindex is out of the scope of this PR

--
I confirm I have tested this PR thoroughly and coded it according to PEP8 and Plone's Python styleguide standards.