Merge remote-tracking branch 'origin/main' into 227

Author: Tom-Willemsen

.github/workflows/check_pr_has_label.yml

@@ -0,0 +1,24 @@
+name: Check PR has release labels
+on:
+  pull_request:
+    types:
+      - opened
+      - reopened
+      - synchronize
+      - labeled
+      - unlabeled
+
+jobs:
+  has_label:
+    name: Check PR has release labels
+    runs-on: ubuntu-latest
+    steps:
+      - run: |
+          echo "PR does not have a release label."
+          exit 1
+        if: |
+          !contains(github.event.pull_request.labels.*.name, 'bluesky-bug') &&
+          !contains(github.event.pull_request.labels.*.name, 'bluesky-Semver-Major') &&
+          !contains(github.event.pull_request.labels.*.name, 'bluesky-Semver-Minor') &&
+          !contains(github.event.pull_request.labels.*.name, 'bluesky-documentation') &&
+          !contains(github.event.pull_request.labels.*.name, 'bluesky-ignore-for-release')

doc/architectural_decisions.md

@@ -0,0 +1,9 @@
+# Architectural Decisions
+
+```{toctree}
+:glob:
+:titlesonly:
+:maxdepth: 1
+
+architectural_decisions/*
+```

doc/architectural_decisions/007-output-file-archiving.md

@@ -8,10 +8,10 @@ Accepted
 
 Our bluesky implementation contains bluesky callbacks which produce scientist-facing output files, for example:
 - [Human-readable scan result files](/callbacks/file_writing): {py:obj}`HumanReadableFileCallback <ibex_bluesky_core.callbacks.HumanReadableFileCallback>`
-- [Fitting results](/fitting/livefit_logger): {py:obj}`LiveFitLogger <ibex_bluesky_core.callbacks.LiveFitLogger>`
+- [Fitting results](/callbacks/fitting/livefit_logger): {py:obj}`LiveFitLogger <ibex_bluesky_core.callbacks.LiveFitLogger>`
 - [Plot PNGs](#plot_png_saver): {py:obj}`PlotPNGSaver <ibex_bluesky_core.callbacks.PlotPNGSaver>`
 
-In addition, we have a [developer-facing callback for diagnostics](/callbacks/docs_logging_callback), 
+In addition, we have a {ref}`developer-facing callback for diagnostics <event_doc_cb>`, 
 {py:obj}`DocLoggingCallback <ibex_bluesky_core.callbacks.DocLoggingCallback>`.
 
 The above callbacks produce files on disk in response to a bluesky scan. These files contain valuable data and so we

doc/architectural_decisions/008-centre-of-mass.md

@@ -0,0 +1,23 @@
+# 8. Using our own Centre of Mass Callback
+
+## Status
+
+Current
+
+## Context
+
+A decision needs to be made about whether to make changes to upstream Bluesky so that their {external+bluesky:py:obj}`bluesky.callbacks.fitting.PeakStats` callback works for us, or we make our own.
+
+## Decision
+
+We will be making our own {py:obj}`CentreOfMass <ibex_bluesky_core.callbacks.CentreOfMass>` callback.
+
+## Justification & Consequences
+
+[We attempted to make changes to upstream Bluesky which were rejected](https://github.com/bluesky/bluesky/pull/1878), as it adds limits to the functionality of the callback. We also found other limitations with using their callback, such as not being able to have disordered and non-continuous data sent to it without it skewing the calculated value - we need it to work with disordered and non-continuous data as we need to be able to run continuous scans.
+
+We currently only need to support positive peaks, but in future a toggle could be added to {py:obj}`CentreOfMass <ibex_bluesky_core.callbacks.CentreOfMass>` to allow for negative peaks. 
+
+This will mean that...
+- Our version of the callback will not be supported by Bluesky and may need changes as Bluesky updates.
+- We can have a version of the callback that is made bespokely for our use cases.

doc/callbacks/centre_of_mass.md

@@ -0,0 +1,30 @@
+# Centre of Mass
+
+{py:obj}`ibex_bluesky_core.callbacks.CentreOfMass` is a callback that provides functionality for calculating our definition of Centre of Mass. We calculate centre of mass from the 2D region bounded by min(y), min(x), max(x), and straight-line segments joining (x, y) data points with their nearest neighbours along the x axis.
+
+{py:obj}`ibex_bluesky_core.callbacks.CentreOfMass` has a property, {py:obj}`result <ibex_bluesky_core.callbacks.CentreOfMass.result>`, which stores the centre of mass value once the callback has finished.
+
+## Our CoM Algorithm
+
+Given non-continuous arrays of collected data `x` and `y`, ({py:obj}`ibex_bluesky_core.callbacks.CentreOfMass`) returns the `x` value of the centre of mass.
+
+Our use cases require that our algorithm abides to the following rules:
+- Any background on data does not skew the centre of mass
+- The order in which data is received does not skew the centre of mass
+- Should support non-constant point spacing without skewing the centre of mass
+
+```{note}
+Note that this is designed for only **positive** peaks.
+```
+
+### Step-by-step
+
+1) Sort `x` and `y` arrays in respect of `x` ascending. This is so that data can be received in any order.
+2) From each `y` element, subtract `min(y)`. This means that any constant background over data is ignored. (Does not work for negative peaks)
+3) Calculate weight/widths for each point; based on it's `x` distances from neighbouring points. This ensures non-constant point spacing is accounted for in our calculation.
+4) For each decomposed shape that makes up the total area under the curve, `CoM` is calculated as the following:
+```{math}
+com_x = \frac{\sum_{}^{}x * y * \text{weight}}{\sum_{}^{}y * \text{weight}}
+```
+
+{py:obj}`ibex_bluesky_core.callbacks.CentreOfMass` can be used from our callbacks collection. See {doc}`isiscallbacks`.

doc/callbacks/docs_logging_callback.md

@@ -1,9 +1,11 @@
 # File writing callbacks
+
+{#hr_file_cb}
 ## Human readable files
 
 A callback ([`HumanReadableFileCallback`](ibex_bluesky_core.callbacks.HumanReadableFileCallback))  exists to write all documents to a separate human-readable file which contains the specified fields. 
 
-This callback will add units and honour precision for each field as well as add some metadata ie. the `uid` of each scan as well as the RB number, which is injected using the {doc}`/preprocessors/rbnumberpp`
+This callback will add units and honour precision for each field as well as add some metadata ie. the `uid` of each scan as well as the RB number, which is injected using the {doc}`/dev/rbnumberpp`
 
 ### Example
 An example of using this could be: 
@@ -56,3 +58,22 @@ The data is prepended on the first event with the names and units of each logged
 for each scan separated by a newline. All of this is separated by commas, though the metadata is not.
 
 The file also contains metadata such as the bluesky version, plan type, and rb number.
+
+## Fit outputs
+
+See {ref}`livefit_logger`
+
+## Plot PNGs
+
+See {ref}`plot_png_saver`
+
+{#event_doc_cb}
+## Bluesky Event documents
+
+```{note}
+This callback is added automatically and is not intended to be user-facing - it is primarily for developer diagnostics.
+```
+
+The [`DocLoggingCallback`](ibex_bluesky_core.callbacks.DocLoggingCallback) is a callback that the BlueSky RunEngine subscribes to unconditionally in {py:obj}`RunEngine import level<ibex_bluesky_core.run_engine.get_run_engine>`. After receiving each document, if they share the same start document (in the same run) then it will write them to the same file. These logs are stored under `C:/instrument/var/logs/bluesky/raw_documents` and are handled by the log rotation.
+
+Each document is stored in a JSON format so can be both machine and human readable. It is in the format `{"type": name, "document": document}` whereby `name` is the type of the document, e.g start, stop, event, descriptor and the `document` is the [document from BlueSky in JSON format](https://blueskyproject.io/bluesky/main/documents.html). As these files are produced per BlueSky run, these will be useful for debugging.

doc/callbacks/file_writing.md

@@ -0,0 +1,9 @@
+# Fits
+
+```{toctree}
+:glob:
+:titlesonly:
+:maxdepth: 1
+
+fitting/*
+```

doc/callbacks/fitting.md

@@ -0,0 +1,47 @@
+# Chained Fitting (`ChainedLiveFit`)
+
+[`ChainedLiveFit`](ibex_bluesky_core.callbacks.ChainedLiveFit) is a specialised callback that manages multiple LiveFit instances in a chain, where each fit's results inform the next fit's initial parameters. This is particularly useful when dealing with complex data sets where subsequent fits depend on the parameters obtained from previous fits.
+
+This is useful for when you need to be careful with your curve fitting due to the presence of noisy data. It allows you to fit your widest (full) wavelength band first and then using its fit parameters as the initial guess of the parameters for the next fit
+
+To show how we expect this to be used, we will use the PolarisingDae and wavelength bands to highlight the need for the carry over of fitting parameters. Below shows two wavelength bands, first bigger than the second, we will fit to the data in the first and carry it over to the data in the second to improve its, otherwise worse, fit.
+
+```python
+# Needed for PolarisingDae
+flipper = block_rw(float, "flipper")
+total_flight_path_length = sc.scalar(value=10, unit=sc.units.m)
+
+x_axis = block_rw(float, "x_axis", write_config=BlockWriteConfig(settle_time_s=0.5))
+wavelength_band_0 = sc.array(dims=["tof"], values=[0, 9999999999.0], unit=sc.units.angstrom, dtype="float64")
+wavelength_band_1 = sc.array(dims=["tof"], values=[0.0, 0.07], unit=sc.units.angstrom, dtype="float64")
+
+dae = polarising_dae(det_pixels=[1], frames=50, flipper=flipper, flipper_states=(0.0, 1.0),
+                     intervals=[wavelength_band_0, wavelength_band_1],
+                     total_flight_path_length=total_flight_path_length, monitor=2)
+
+
+def plan() -> Generator[Msg, None, None]:
+  fig, (ax1, ax2) = yield from call_qt_aware(plt.subplots, 2)
+  chained_fit = ChainedLiveFit(method=Linear.fit(), y=[dae.reducer.wavelength_bands[0].calculate_polarisation.name,
+                                                       dae.reducer.wavelength_bands[1].calculate_polarisation.name],
+                               x=bob.name, ax=[ax1, ax2])
+
+  # Subscribe chained_fit to RE and run do a scan for example
+  # chained_fit.get_livefits()[-1].result will give you the fitting results for the last wavelength band
+```
+
+- You are expected to pass in the list of signal names for each independent variable to `y` in order of how you want the subsequent fitting to go.
+- You may also pass in a list of matplotlib axes, which will mean that LiveFitPlots are created per LiveFit, and it will plot the each respective fit to an axis. LiveFitPlots are not created if you do not pass `ax`.
+- Similar to the `y` parameter, you may pass signal names which correspond to uncertainty values for each independent variable.
+
+```{hint}
+The method for fitting is the same across all independent variables.
+```
+
+```{note}
+Parameter uncertainties are not carried over between fits 
+```
+
+```{important}
+If a fit fails to converge, subsequent fits will use their default guess functions
+```

doc/callbacks/fitting/chained_fitting.md

@@ -1,6 +1,6 @@
-# Fitting Callbacks
+# Fitting (`LiveFit`)
 
-Similar to [`LivePlot`](../callbacks/plotting.md), [`ibex_bluesky_core`](ibex_bluesky_core) provides a thin wrapper around Bluesky's [`LiveFit`](ibex_bluesky_core.callbacks.LiveFit) class, enhancing it with additional functionality to better support real-time data fitting. This wrapper not only offers a wide selection of models to fit your data on, but also introduces guess generation for fit parameters. As new data points are acquired, the wrapper refines these guesses dynamically, improving the accuracy of the fit with each additional piece of data, allowing for more efficient and adaptive real-time fitting workflows.
+Similar to [`LivePlot`](/callbacks/plotting.md), [`ibex_bluesky_core`](ibex_bluesky_core) provides a thin wrapper around Bluesky's [`LiveFit`](ibex_bluesky_core.callbacks.LiveFit) class, enhancing it with additional functionality to better support real-time data fitting. This wrapper not only offers a wide selection of models to fit your data on, but also introduces guess generation for fit parameters. As new data points are acquired, the wrapper refines these guesses dynamically, improving the accuracy of the fit with each additional piece of data, allowing for more efficient and adaptive real-time fitting workflows.
 
 In order to use the wrapper, import[`LiveFit`](ibex_bluesky_core.callbacks.LiveFit from [`ibex_bluesky_core`](ibex_bluesky_core) rather than 
 `bluesky` directly:
@@ -37,7 +37,7 @@ fit_plot_callback = LiveFitPlot(fit_callback, ax=ax, color="r")
 
 Using the `yerr` argument allows you to pass uncertainties via a signal to LiveFit, so that the "weight" of each point influences the fit produced. By not providing a signal name you choose not to use uncertainties/weighting in the fitting calculation. Each weight is computed as `1/(standard deviation at point)` and is taken into account to determine how much a point affects the overall fit of the data. Same as the rest of [`LiveFit`](ibex_bluesky_core.callbacks.LiveFit), the fit will be updated after every new point collected now taking into account the weights of each point. Uncertainty data is collected from Bluesky event documents after each new point.
 
-The `plot_callback` and `fit_plot_callback` objects can then be subscribed to the `RunEngine`, using the same methods as described in [`LivePlot`](../callbacks/plotting.md). See the following example using `@subs_decorator`:
+The `plot_callback` and `fit_plot_callback` objects can then be subscribed to the `RunEngine`, using the same methods as described in [`LivePlot`](/callbacks/plotting.md). See the following example using `@subs_decorator`:
 
 ```py
 @subs_decorator(
@@ -213,52 +213,3 @@ fit_method = FitMethod(Polynomial.model(3), different_guess) # If using a custom
 lf = LiveFit(fit_method, ...)
 ```
 See the [standard fits](#models) list above for standard fits which require parameters. It gets more complicated if you want to define your own custom model or guess which you want to pass parameters to. You will have to define a function that takes these parameters and returns the model / guess function with the subsituted values.
-
-## Chained Fitting
-
-[`ChainedLiveFit`](ibex_bluesky_core.callbacks.ChainedLiveFit) is a specialised callback that manages multiple LiveFit instances in a chain, where each fit's results inform the next fit's initial parameters. This is particularly useful when dealing with complex data sets where subsequent fits depend on the parameters obtained from previous fits.
-
-This is useful for when you need to be careful with your curve fitting due to the presence of noisy data. It allows you to fit your widest (full) wavelength band first and then using its fit parameters as the initial guess of the parameters for the next fit
-
-# Usage
-To show how we expect this to be used we will use the PolarisingDae and wavelength bands to highlight the need for the carry over of fitting parameters. Below shows two wavelength bands, first bigger than the second, we will fit to the data in the first and carry it over to the data in the second to improve its, otherwise worse, fit.
-
-```python
-# Needed for PolarisingDae
-flipper = block_rw(float, "flipper")
-total_flight_path_length = sc.scalar(value=10, unit=sc.units.m)
-
-x_axis = block_rw(float, "x_axis", write_config=BlockWriteConfig(settle_time_s=0.5))
-wavelength_band_0 = sc.array(dims=["tof"], values=[0, 9999999999.0], unit=sc.units.angstrom, dtype="float64")
-wavelength_band_1 = sc.array(dims=["tof"], values=[0.0, 0.07], unit=sc.units.angstrom, dtype="float64")
-
-dae = polarising_dae(det_pixels=[1], frames=50, flipper=flipper, flipper_states=(0.0, 1.0),
-                     intervals=[wavelength_band_0, wavelength_band_1],
-                     total_flight_path_length=total_flight_path_length, monitor=2)
-
-
-def plan() -> Generator[Msg, None, None]:
-  fig, (ax1, ax2) = yield from call_qt_aware(plt.subplots, 2)
-  chained_fit = ChainedLiveFit(method=Linear.fit(), y=[dae.reducer.wavelength_bands[0].calculate_polarisation.name,
-                                                       dae.reducer.wavelength_bands[1].calculate_polarisation.name],
-                               x=bob.name, ax=[ax1, ax2])
-
-  # Subscribe chained_fit to RE and run do a scan for example
-  # chained_fit.get_livefits()[-1].result will give you the fitting results for the last wavelength band
-```
-
-- You are expected to pass in the list of signal names for each independent variable to `y` in order of how you want the subsequent fitting to go.
-- You may also pass in a list of matplotlib axes, which will mean that LiveFitPlots are created per LiveFit, and it will plot the each respective fit to an axis. LiveFitPlots are not created if you do not pass `ax`.
-- Similar to the `y` parameter, you may pass signal names which correspond to uncertainty values for each independent variable.
-
-```{hint}
-The method for fitting is the same across all independent variables.
-```
-
-```{note}
-Parameter uncertainties are not carried over between fits 
-```
-
-```{important}
-If a fit fails to converge, subsequent fits will use their default guess functions
-```