[skip ci]

…s_in_place function

…symmetry is missing

…rounding

…hon2 syntax fix

…er fix

…want to calculate distance between two sets of atoms and you don't care about labels!

…ct output format. With test. Propagate output_cs parameter.

* Framework for compile-time polymorphism.

In the exascale_api, allow pixel values to be calculation either on
large array (all pixels), or with low-memory on just the whitelist
consisting of shoebox pixels.  This commit only gives the polymorphism
framework; both implementations are currently identical giving the
large-array behavior.

* Debug case for non-working test.

The script tests/tst_memory_policy.py fails with a cuda illegal access.
The intention is to get help from NESAP to get a functional test.

* Minimal changes for working tst_memory_policy.py

Memory savings achieved through code specialization, for the case where
pixel values are simulated on a small whitelist. Specializations are not
yet optimal, as there is still a lot of code duplication.

Changes give ~4.5x reduction in memory footprint, but no success yet in
resizing the array m_accumulate_floatimage.  Attempts so far lead to
cuda memory allocation error.

* Remove debugging output to conserve stdout size.

* Add first verion of `UsageMonitor` to monitor C/GPU memory/use

* Suppress logging information from loggers other than `usage`

* Explicitely point to `target` when starting threads

* Add logging details, period, null loggers

* Correctly calculate total `UsageStats` across cpu, gpu

* Don't pass `self` as arg to `Thread` (oddly, this is correct)

* Add `UsageStatsHistory`, try to fix empty log

* Add test prints for logging

* All ranks need to call `log_current_usage`

* Why does logging usage stop at some point?

* Correctly measure GPU memory use, only sample rank if `single`

* Add some basic usage stats plotting every 10th iteration

* Allow plotting empty, plot only in main thread

* Correctly calculate max memory

* Add timestamp to print statements to correlate with usage log

* Make `rank/node_usage_stats` a method, avoid unnecessary calls

* Only rank 0 should attempt plotting `usage_stats_histories`

* `gpu_usage_and_memory`: return -1 if no GPU

* `log_current_usage`: No longer print that you're logging

* Don't try to sleep backwards :)

* Move logging to `UsageLogManager`, allow plotting from logs

* Implement inheritance structure that allows for easy extension

* Reorganize, heavily document the code

* Reorganize, heavily document the code 2

* Rename `UnitIntervalFloat` to PerCentFloat`, fix gpu mem. range

* Rename `UsageStats` to `ResourceStats` (and associated)

* Improve type annotations

* Add option to monitor node 0 only

* Add `MonitorWorker` for `cctbx.xfel.merge`

* Improve `ResourceStatsArtist` axes description

* Add `DummyResourceProbe` for CPU & GPU in case no `Probe` works

* Fix typo

* Fix typo 2

* Fix typo 3

* Fix typo 4

* Fix missing microseconds in log

* In `ResourceStatsArtist`, change `tight_layout` to `constrained_layout`

* Use only single `.png` path extension

* Fix `ResourceStatsHistory.get_deltas_array` if empty

* Correctly print milliseconds

* Correctly check for empty array

* Revert `ResourceLogManager` as it's used outside

* Correctly pass phil parameters to `MonitorWorker`

* Correctly pass phil parameters to `MonitorWorker`

* In `get_node_resource_stats` communicate within node only

* Comm in `get_node_resource_stats` fails. Try adding barrier.

* Don't attempt to collect data across ranks, mpi fails

* Allow two loggers to work independently by customizing `logger_name`

* Add options and phil pars to disable writing and disable plotting

* Add options and phil pars to disable writing and disable plotting 2

* Update phil

* Update phil 2

* Add `libtbx/resource_monitor.py` to py2 syntax exceptions

* Remove Nick's changes specific to `memory_policy` branch

* Remove outdated line in `py2_syntax_exceptions.txt`

* Remove python3.5+ syntax from `xfel/merging/application/monitor`

* Add clutter because libtbx requires it

* Update `ResourceLogManager.line_regex`

* Assert all `ResourceStats` contain only `PerCentFloat`s, not `str`

* support for python versions < 3.9

* Increase plot font size from 10 to 16

* Add support for PathLike ResourceMonitor.prefix as suggested by @dermen

* Add grid to make trends more visible as suggested by @dermen

* Set libtbx dispatcher name `libtbx.resource_monitor_plot` for plot_logs

* Set font.size 12 (16 is big), and locally to not affect any other plot

* Move resource monitor's `plot_log` thin wrapper to libtbx/command_line

* Make `lilbtbx.resource_monitor_plot`'s prefix optional

---------

Co-authored-by: Nicholas K Sauter <nksauter@lbl.gov>
Co-authored-by: dermen <dermen@lbl.gov>

…x_distance_between_connecting_atoms. Enabling tests.

Makes the tdata globally available for use in the new ucinline worker of
psii_spread.

See dials/dials#2679
dials/dials#2679

- Add filter for datasets in the datasets tab
- Use medians in the energy tab
- Bugfix for d_min in the runstats plot
- Handle UNKWN for ensemble_refinement, which is the state if some jobs haven't finished in a job with multiple submitted job ids
- More precision for the merging stats plot

…y(map_data) is very slow relative to map_data.deep_copy())

Co-authored-by: Aaron Brewster <asbrewster@lbl.gov>

…mmetry_if_necessary in the class which should be used instead.

- add test

[skip ci]

- Set locale and clean up XFEL CI
- Add make to XFEL CI
- Fix macOS cache for XFEL CI

…avings for models without alt locs.

… crashes!)

… atom selection is faster than constructing it from list of lists

…lton_geometry_validation

…k0 (cctbx#999)

…tests. Not enabling just yet.

This reverts commit 555aee4.

…ust pdb_hierarchy; use hierarchy not pdb_input in validation