Skip to content

Estia #108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 53 commits into from
Mar 18, 2025
Merged

Estia #108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
a0abff8
Create estia-data-reduction.md
jokasimr Nov 27, 2024
0dff310
feat: initial setup for estia workflow
jokasimr Dec 10, 2024
91a69cb
feat: compute angle without divergence angle per pixel
jokasimr Jan 29, 2025
62b654c
fix: dont modify input + dont keep intermediates
jokasimr Feb 5, 2025
c0cd69e
fix: remove operation modifying input
jokasimr Feb 5, 2025
0af30e9
feat: estia coordinate transformations + workflow
jokasimr Feb 5, 2025
29c983f
WIP: estia workflow - changes to the generic part of the workflow
jokasimr Feb 11, 2025
f594a68
WIP: estia workflow
jokasimr Feb 11, 2025
ee1b3d8
fix
jokasimr Feb 17, 2025
b8ef151
wip
jokasimr Feb 18, 2025
09b50a4
feat: add mcstas h5 file loader
jokasimr Feb 18, 2025
0032aa1
fix: allow loading ascii and h5 files
jokasimr Feb 18, 2025
4666e24
Apply automatic formatting
pre-commit-ci-lite[bot] Feb 17, 2025
696668e
fix: reshape to expected dimensions
jokasimr Feb 19, 2025
83406c2
test: add mcstas test files and tests
jokasimr Feb 20, 2025
39422bf
minor
jokasimr Feb 27, 2025
6afc1eb
docs: removing from this pr because documentation needs some rewriting
jokasimr Feb 27, 2025
6b21a1b
refactor: share parts of the workflow between amor and estia
jokasimr Feb 27, 2025
32c6a46
fix: remove comment and add providers to load
jokasimr Feb 28, 2025
98ff25c
docs: typehints
jokasimr Feb 28, 2025
0a2946e
docs: add parameters in docstrings
jokasimr Feb 28, 2025
9c88841
fix: mod t to get event_time_offset + remove unnecessary coordinates
jokasimr Feb 28, 2025
761233e
docs: update licence headers
jokasimr Feb 28, 2025
6c92208
docs: add licence header
jokasimr Feb 28, 2025
5fd4b42
fix: add comment explaining missing value and set better default
jokasimr Mar 3, 2025
c64ea36
docs: add docstring
jokasimr Mar 3, 2025
011c904
docs: add type hints
jokasimr Mar 3, 2025
6eec3a0
fix: rename test file
jokasimr Mar 3, 2025
9d0c3aa
fix: rename test
jokasimr Mar 3, 2025
fde2787
fix: remove center of incident angle concept
jokasimr Mar 4, 2025
6b27e69
refactor: move masking
jokasimr Mar 4, 2025
539b9ae
Update src/ess/estia/conversions.py
jokasimr Mar 4, 2025
f23e91e
doc: add comment clarifying mcstas weight variances
jokasimr Mar 10, 2025
5d8f3d8
refactor: remove 'center of incoming' concept from amor to make consi…
jokasimr Mar 11, 2025
ecc2dac
fix: better guess
jokasimr Mar 11, 2025
ddf7ec3
fix: allows passing a number as QBins
jokasimr Mar 11, 2025
dd02ed5
docs: add variance comment to ascii mcstas loader
jokasimr Mar 11, 2025
e012cd0
refactor: move figures to common reflectometry module
jokasimr Mar 12, 2025
78a87cf
fix: load required data from mcstas hdf5 output stored in attributes
jokasimr Mar 12, 2025
8b72a05
Merge remote-tracking branch 'origin/main' into estia
jokasimr Mar 12, 2025
3dd2295
fix: handle when theta is in outer coords, accept integer bins
jokasimr Mar 13, 2025
8c2097b
docs: add example datasets
jokasimr Mar 13, 2025
e304f2a
docs: mcstas data redcution examples
jokasimr Mar 13, 2025
6ec0b06
fix: keep variances on reference, to make it nicer to inspect, remove…
jokasimr Mar 13, 2025
16c43cf
docs: clean up notebook
jokasimr Mar 13, 2025
d283075
docs: clean up notebook
jokasimr Mar 13, 2025
54bd312
fix
jokasimr Mar 13, 2025
19ab5b9
docs: add estia section to docs
jokasimr Mar 14, 2025
c48ba09
docs: update api reference page
jokasimr Mar 14, 2025
8c0cf0f
docs: move modules in docs that have been moved in the code
jokasimr Mar 14, 2025
1d0f2d4
docs: add ground truth reflectivity comparison and some explanation i…
jokasimr Mar 14, 2025
ae42bb5
fix
jokasimr Mar 14, 2025
413ed1a
fix
jokasimr Mar 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 25 additions & 2 deletions docs/api-reference/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
types
tools
workflow
figures
```

## Amor
Expand All @@ -48,11 +49,33 @@

conversions
data
instrument_view
load
orso
resolution
types
utils
figures
normalization
workflow
```

## Estia

```{eval-rst}
.. currentmodule:: ess.estia

.. autosummary::
:toctree: ../generated/modules
:template: module-template.rst
:recursive:

conversions
maskings
corrections
data
mcstas
load
resolution
types
normalization
workflow
```
6 changes: 3 additions & 3 deletions docs/user-guide/amor/amor-reduction.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,7 @@
"metadata": {},
"outputs": [],
"source": [
"from ess.amor.figures import wavelength_theta_figure\n",
"from ess.reflectometry.figures import wavelength_theta_figure\n",
"\n",
"wavelength_theta_figure(\n",
" [result[ReflectivityOverZW] for result in diagnostics.values()],\n",
Expand Down Expand Up @@ -329,7 +329,7 @@
"metadata": {},
"outputs": [],
"source": [
"from ess.amor.figures import q_theta_figure\n",
"from ess.reflectometry.figures import q_theta_figure\n",
"\n",
"q_theta_figure(\n",
" [res[ReflectivityOverZW] for res in diagnostics.values()],\n",
Expand All @@ -355,7 +355,7 @@
"metadata": {},
"outputs": [],
"source": [
"from ess.amor.figures import wavelength_z_figure\n",
"from ess.reflectometry.figures import wavelength_z_figure\n",
"\n",
"workflow[Filename[SampleRun]] = runs['608'][Filename[SampleRun]]\n",
"workflow[SampleRotation[SampleRun]] = runs['608'][SampleRotation[SampleRun]]\n",
Expand Down
328 changes: 328 additions & 0 deletions docs/user-guide/estia/estia-mcstas-reduction.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,328 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "0",
"metadata": {},
"source": [
"# Reduction of ESTIA McStas data"
]
},
{
"cell_type": "markdown",
"id": "1",
"metadata": {},
"source": [
"This notebook demonstrates how to run the data reduction on the output of McStas simulations of the instrument.\n",
"\n",
"Essentially this looks very similar to how one would do data reduction on real data files from the physical instrument,\n",
"but we replace the default loader with the `load_mcstas_events` provider."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2",
"metadata": {},
"outputs": [],
"source": [
"#%matplotlib widget\n",
"import scipp as sc\n",
"\n",
"from ess.estia.load import load_mcstas_events\n",
"from ess.estia.data import estia_mcstas_example, estia_mcstas_groundtruth\n",
"from ess.estia import EstiaWorkflow\n",
"from ess.reflectometry.types import *\n",
"from ess.reflectometry.figures import wavelength_z_figure, wavelength_theta_figure, q_theta_figure"
]
},
{
"cell_type": "markdown",
"id": "3",
"metadata": {},
"source": [
"The Estia reduction workflow is created and we set parameters such as region of interest, wavelengthbins, and q-bins."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4",
"metadata": {},
"outputs": [],
"source": [
"\n",
"wf = EstiaWorkflow()\n",
"wf.insert(load_mcstas_events)\n",
"wf[Filename[ReferenceRun]] = estia_mcstas_example('reference')\n",
"\n",
"wf[YIndexLimits] = sc.scalar(35), sc.scalar(64)\n",
"wf[ZIndexLimits] = sc.scalar(0), sc.scalar(14 * 32)\n",
"wf[BeamDivergenceLimits] = sc.scalar(-0.75, unit='deg'), sc.scalar(0.75, unit='deg')\n",
"wf[WavelengthBins] = sc.geomspace('wavelength', 3.5, 12, 2001, unit='angstrom')\n",
"wf[QBins] = 1000\n",
"\n",
"# There is no proton current data in the McStas files, here we just add some fake proton current\n",
"# data to make the workflow run.\n",
"wf[ProtonCurrent[SampleRun]] = sc.DataArray(\n",
" sc.array(dims=('time',), values=[]),\n",
" coords={'time': sc.array(dims=('time',), values=[], unit='s')})\n",
"wf[ProtonCurrent[ReferenceRun]] = sc.DataArray(\n",
" sc.array(dims=('time',), values=[]),\n",
" coords={'time': sc.array(dims=('time',), values=[], unit='s')})\n",
"\n",
"\n",
"def compute_reflectivity_curve_for_mcstas_data(wf, results):\n",
" R, ref, da = w.compute((ReflectivityOverQ, Reference, ReducibleData[SampleRun])).values()\n",
" # In the McStas simulation the reference has quite low intensity.\n",
" # To make the reflectivity curve a bit more clean\n",
" # we filter out the Q-points where the reference has too large uncertainties.\n",
" ref = ref.hist(Q=R.coords['Q'])\n",
" too_large_uncertainty_in_reference = sc.stddevs(ref).data > 0.3 * ref.data\n",
" R = R.hist()\n",
" R.data = sc.where(too_large_uncertainty_in_reference, sc.scalar(float('nan'), unit=R.unit), R.data)\n",
" results[f\"{da.coords['sample_rotation'].value} {da.coords['sample_rotation'].unit}\"] = R\n"
]
},
{
"cell_type": "markdown",
"id": "5",
"metadata": {},
"source": [
"## Ni/Ti multilayer sample\n",
"\n",
"Below is a comparison between the reflectivity curve obtained using the reduction workflow and the ground truth reflectivity curve that was used in the McStas simulation.\n",
"The sample was simulated at different sample rotation settings, each settings produces a separate reflectivity curve covering a higher Q-range, and that is the angle in the legend of the figure."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6",
"metadata": {},
"outputs": [],
"source": [
"results = {}\n",
"for path in estia_mcstas_example('Ni/Ti-multilayer'):\n",
" w = wf.copy()\n",
" w[Filename[SampleRun]] = path\n",
" compute_reflectivity_curve_for_mcstas_data(w, results)\n",
"\n",
"ground_truth = estia_mcstas_groundtruth('Ni/Ti-multilayer')\n",
"\n",
"sc.plot({'ground_truth': ground_truth} | results, norm='log', vmin=1e-8)"
]
},
{
"cell_type": "markdown",
"id": "7",
"metadata": {},
"source": [
"Below are a number of figures displaying different projections of the measured intensity distribution."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8",
"metadata": {},
"outputs": [],
"source": [
"results = []\n",
"for path in estia_mcstas_example('Ni/Ti-multilayer'):\n",
" w = wf.copy()\n",
" w[Filename[SampleRun]] = path\n",
" results.append(w.compute(Sample))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9",
"metadata": {},
"outputs": [],
"source": [
"wavelength_z_figure(results[3], wavelength_bins=400)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "10",
"metadata": {},
"outputs": [],
"source": [
"wavelength_theta_figure(results, wavelength_bins=400, theta_bins=200, q_edges_to_display=[sc.scalar(0.016, unit='1/angstrom'), sc.scalar(0.19, unit='1/angstrom')])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "11",
"metadata": {},
"outputs": [],
"source": [
"q_theta_figure(results, q_bins=300, theta_bins=200)"
]
},
{
"cell_type": "markdown",
"id": "12",
"metadata": {},
"source": [
"## Ni on Silicon"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "13",
"metadata": {},
"outputs": [],
"source": [
"results = {}\n",
"for path in estia_mcstas_example('Ni-film on silicon'):\n",
" w = wf.copy()\n",
" w[Filename[SampleRun]] = path\n",
" compute_reflectivity_curve_for_mcstas_data(w, results)\n",
"\n",
"ground_truth = estia_mcstas_groundtruth('Ni-film on silicon')\n",
"sc.plot({'ground_truth': ground_truth} | results, norm='log', vmin=1e-9)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "14",
"metadata": {},
"outputs": [],
"source": [
"results = []\n",
"for path in estia_mcstas_example('Ni-film on silicon'):\n",
" w = wf.copy()\n",
" w[Filename[SampleRun]] = path\n",
" results.append(w.compute(ReducibleData[SampleRun]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "15",
"metadata": {},
"outputs": [],
"source": [
"wavelength_z_figure(results[3], wavelength_bins=400)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "16",
"metadata": {},
"outputs": [],
"source": [
"wavelength_theta_figure(results, wavelength_bins=400, theta_bins=200, q_edges_to_display=[sc.scalar(0.016, unit='1/angstrom'), sc.scalar(0.19, unit='1/angstrom')])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "17",
"metadata": {},
"outputs": [],
"source": [
"q_theta_figure(results, q_bins=300, theta_bins=200)"
]
},
{
"cell_type": "markdown",
"id": "18",
"metadata": {},
"source": [
"## SiO2 on Silicon"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "19",
"metadata": {},
"outputs": [],
"source": [
"results = {}\n",
"for path in estia_mcstas_example('Natural SiO2 on silicon'):\n",
" w = wf.copy()\n",
" w[Filename[SampleRun]] = path\n",
" compute_reflectivity_curve_for_mcstas_data(w, results)\n",
"\n",
"ground_truth = estia_mcstas_groundtruth('Natural SiO2 on silicon')\n",
"sc.plot({'ground_truth': ground_truth} | results, norm='log', vmin=1e-9)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "20",
"metadata": {},
"outputs": [],
"source": [
"results = []\n",
"for path in estia_mcstas_example('Natural SiO2 on silicon'):\n",
" w = wf.copy()\n",
" w[Filename[SampleRun]] = path\n",
" results.append(w.compute(ReducibleData[SampleRun]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21",
"metadata": {},
"outputs": [],
"source": [
"wavelength_z_figure(results[3], wavelength_bins=400)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "22",
"metadata": {},
"outputs": [],
"source": [
"wavelength_theta_figure(results, wavelength_bins=400, theta_bins=200, q_edges_to_display=[sc.scalar(0.016, unit='1/angstrom')])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "23",
"metadata": {},
"outputs": [],
"source": [
"q_theta_figure(results, q_bins=300, theta_bins=200)"
]
}
],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The notebook makes a lot a plots. I think it would really benefit from some short explanations between the figures, explaining what we are showing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you, but I think it's good to not make this PR any longer now. It's a basic notebook that someone experienced in the technique will at least get the gist of. Let's postpone this step to a separate PR for improving the docs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree, I think leaving it til later will increase the chances of us forgetting.

I don't think it's so much work to add small sentences and sections explaining a little what we are doing, especially if this is going to appear in the docs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some explanations already added, maybe take a look at those and see if that's what you wanted to see there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, sorry I had missed those.

"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.14"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Loading