-
Notifications
You must be signed in to change notification settings - Fork 3
Estia #108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Estia #108
Changes from all commits
Commits
Show all changes
53 commits
Select commit
Hold shift + click to select a range
a0abff8
Create estia-data-reduction.md
jokasimr 0dff310
feat: initial setup for estia workflow
jokasimr 91a69cb
feat: compute angle without divergence angle per pixel
jokasimr 62b654c
fix: dont modify input + dont keep intermediates
jokasimr c0cd69e
fix: remove operation modifying input
jokasimr 0af30e9
feat: estia coordinate transformations + workflow
jokasimr 29c983f
WIP: estia workflow - changes to the generic part of the workflow
jokasimr f594a68
WIP: estia workflow
jokasimr ee1b3d8
fix
jokasimr b8ef151
wip
jokasimr 09b50a4
feat: add mcstas h5 file loader
jokasimr 0032aa1
fix: allow loading ascii and h5 files
jokasimr 4666e24
Apply automatic formatting
pre-commit-ci-lite[bot] 696668e
fix: reshape to expected dimensions
jokasimr 83406c2
test: add mcstas test files and tests
jokasimr 39422bf
minor
jokasimr 6afc1eb
docs: removing from this pr because documentation needs some rewriting
jokasimr 6b21a1b
refactor: share parts of the workflow between amor and estia
jokasimr 32c6a46
fix: remove comment and add providers to load
jokasimr 98ff25c
docs: typehints
jokasimr 0a2946e
docs: add parameters in docstrings
jokasimr 9c88841
fix: mod t to get event_time_offset + remove unnecessary coordinates
jokasimr 761233e
docs: update licence headers
jokasimr 6c92208
docs: add licence header
jokasimr 5fd4b42
fix: add comment explaining missing value and set better default
jokasimr c64ea36
docs: add docstring
jokasimr 011c904
docs: add type hints
jokasimr 6eec3a0
fix: rename test file
jokasimr 9d0c3aa
fix: rename test
jokasimr fde2787
fix: remove center of incident angle concept
jokasimr 6b27e69
refactor: move masking
jokasimr 539b9ae
Update src/ess/estia/conversions.py
jokasimr f23e91e
doc: add comment clarifying mcstas weight variances
jokasimr 5d8f3d8
refactor: remove 'center of incoming' concept from amor to make consi…
jokasimr ecc2dac
fix: better guess
jokasimr ddf7ec3
fix: allows passing a number as QBins
jokasimr dd02ed5
docs: add variance comment to ascii mcstas loader
jokasimr e012cd0
refactor: move figures to common reflectometry module
jokasimr 78a87cf
fix: load required data from mcstas hdf5 output stored in attributes
jokasimr 8b72a05
Merge remote-tracking branch 'origin/main' into estia
jokasimr 3dd2295
fix: handle when theta is in outer coords, accept integer bins
jokasimr 8c2097b
docs: add example datasets
jokasimr e304f2a
docs: mcstas data redcution examples
jokasimr 6ec0b06
fix: keep variances on reference, to make it nicer to inspect, remove…
jokasimr 16c43cf
docs: clean up notebook
jokasimr d283075
docs: clean up notebook
jokasimr 54bd312
fix
jokasimr 19ab5b9
docs: add estia section to docs
jokasimr c48ba09
docs: update api reference page
jokasimr 8c0cf0f
docs: move modules in docs that have been moved in the code
jokasimr 1d0f2d4
docs: add ground truth reflectivity comparison and some explanation i…
jokasimr ae42bb5
fix
jokasimr 413ed1a
fix
jokasimr File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,328 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "markdown", | ||
"id": "0", | ||
"metadata": {}, | ||
"source": [ | ||
"# Reduction of ESTIA McStas data" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "1", | ||
"metadata": {}, | ||
"source": [ | ||
"This notebook demonstrates how to run the data reduction on the output of McStas simulations of the instrument.\n", | ||
"\n", | ||
"Essentially this looks very similar to how one would do data reduction on real data files from the physical instrument,\n", | ||
"but we replace the default loader with the `load_mcstas_events` provider." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "2", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"#%matplotlib widget\n", | ||
"import scipp as sc\n", | ||
"\n", | ||
"from ess.estia.load import load_mcstas_events\n", | ||
"from ess.estia.data import estia_mcstas_example, estia_mcstas_groundtruth\n", | ||
"from ess.estia import EstiaWorkflow\n", | ||
"from ess.reflectometry.types import *\n", | ||
"from ess.reflectometry.figures import wavelength_z_figure, wavelength_theta_figure, q_theta_figure" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "3", | ||
"metadata": {}, | ||
"source": [ | ||
"The Estia reduction workflow is created and we set parameters such as region of interest, wavelengthbins, and q-bins." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "4", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"\n", | ||
"wf = EstiaWorkflow()\n", | ||
"wf.insert(load_mcstas_events)\n", | ||
"wf[Filename[ReferenceRun]] = estia_mcstas_example('reference')\n", | ||
"\n", | ||
"wf[YIndexLimits] = sc.scalar(35), sc.scalar(64)\n", | ||
"wf[ZIndexLimits] = sc.scalar(0), sc.scalar(14 * 32)\n", | ||
"wf[BeamDivergenceLimits] = sc.scalar(-0.75, unit='deg'), sc.scalar(0.75, unit='deg')\n", | ||
"wf[WavelengthBins] = sc.geomspace('wavelength', 3.5, 12, 2001, unit='angstrom')\n", | ||
"wf[QBins] = 1000\n", | ||
"\n", | ||
"# There is no proton current data in the McStas files, here we just add some fake proton current\n", | ||
"# data to make the workflow run.\n", | ||
"wf[ProtonCurrent[SampleRun]] = sc.DataArray(\n", | ||
" sc.array(dims=('time',), values=[]),\n", | ||
" coords={'time': sc.array(dims=('time',), values=[], unit='s')})\n", | ||
"wf[ProtonCurrent[ReferenceRun]] = sc.DataArray(\n", | ||
" sc.array(dims=('time',), values=[]),\n", | ||
" coords={'time': sc.array(dims=('time',), values=[], unit='s')})\n", | ||
"\n", | ||
"\n", | ||
"def compute_reflectivity_curve_for_mcstas_data(wf, results):\n", | ||
" R, ref, da = w.compute((ReflectivityOverQ, Reference, ReducibleData[SampleRun])).values()\n", | ||
" # In the McStas simulation the reference has quite low intensity.\n", | ||
" # To make the reflectivity curve a bit more clean\n", | ||
" # we filter out the Q-points where the reference has too large uncertainties.\n", | ||
" ref = ref.hist(Q=R.coords['Q'])\n", | ||
" too_large_uncertainty_in_reference = sc.stddevs(ref).data > 0.3 * ref.data\n", | ||
" R = R.hist()\n", | ||
" R.data = sc.where(too_large_uncertainty_in_reference, sc.scalar(float('nan'), unit=R.unit), R.data)\n", | ||
" results[f\"{da.coords['sample_rotation'].value} {da.coords['sample_rotation'].unit}\"] = R\n" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "5", | ||
"metadata": {}, | ||
"source": [ | ||
"## Ni/Ti multilayer sample\n", | ||
"\n", | ||
"Below is a comparison between the reflectivity curve obtained using the reduction workflow and the ground truth reflectivity curve that was used in the McStas simulation.\n", | ||
"The sample was simulated at different sample rotation settings, each settings produces a separate reflectivity curve covering a higher Q-range, and that is the angle in the legend of the figure." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "6", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"results = {}\n", | ||
"for path in estia_mcstas_example('Ni/Ti-multilayer'):\n", | ||
" w = wf.copy()\n", | ||
" w[Filename[SampleRun]] = path\n", | ||
" compute_reflectivity_curve_for_mcstas_data(w, results)\n", | ||
"\n", | ||
"ground_truth = estia_mcstas_groundtruth('Ni/Ti-multilayer')\n", | ||
"\n", | ||
"sc.plot({'ground_truth': ground_truth} | results, norm='log', vmin=1e-8)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "7", | ||
"metadata": {}, | ||
"source": [ | ||
"Below are a number of figures displaying different projections of the measured intensity distribution." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "8", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"results = []\n", | ||
"for path in estia_mcstas_example('Ni/Ti-multilayer'):\n", | ||
" w = wf.copy()\n", | ||
" w[Filename[SampleRun]] = path\n", | ||
" results.append(w.compute(Sample))" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "9", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"wavelength_z_figure(results[3], wavelength_bins=400)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "10", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"wavelength_theta_figure(results, wavelength_bins=400, theta_bins=200, q_edges_to_display=[sc.scalar(0.016, unit='1/angstrom'), sc.scalar(0.19, unit='1/angstrom')])" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "11", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"q_theta_figure(results, q_bins=300, theta_bins=200)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "12", | ||
"metadata": {}, | ||
"source": [ | ||
"## Ni on Silicon" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "13", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"results = {}\n", | ||
"for path in estia_mcstas_example('Ni-film on silicon'):\n", | ||
" w = wf.copy()\n", | ||
" w[Filename[SampleRun]] = path\n", | ||
" compute_reflectivity_curve_for_mcstas_data(w, results)\n", | ||
"\n", | ||
"ground_truth = estia_mcstas_groundtruth('Ni-film on silicon')\n", | ||
"sc.plot({'ground_truth': ground_truth} | results, norm='log', vmin=1e-9)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "14", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"results = []\n", | ||
"for path in estia_mcstas_example('Ni-film on silicon'):\n", | ||
" w = wf.copy()\n", | ||
" w[Filename[SampleRun]] = path\n", | ||
" results.append(w.compute(ReducibleData[SampleRun]))" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "15", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"wavelength_z_figure(results[3], wavelength_bins=400)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "16", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"wavelength_theta_figure(results, wavelength_bins=400, theta_bins=200, q_edges_to_display=[sc.scalar(0.016, unit='1/angstrom'), sc.scalar(0.19, unit='1/angstrom')])" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "17", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"q_theta_figure(results, q_bins=300, theta_bins=200)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "18", | ||
"metadata": {}, | ||
"source": [ | ||
"## SiO2 on Silicon" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "19", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"results = {}\n", | ||
"for path in estia_mcstas_example('Natural SiO2 on silicon'):\n", | ||
" w = wf.copy()\n", | ||
" w[Filename[SampleRun]] = path\n", | ||
" compute_reflectivity_curve_for_mcstas_data(w, results)\n", | ||
"\n", | ||
"ground_truth = estia_mcstas_groundtruth('Natural SiO2 on silicon')\n", | ||
"sc.plot({'ground_truth': ground_truth} | results, norm='log', vmin=1e-9)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "20", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"results = []\n", | ||
"for path in estia_mcstas_example('Natural SiO2 on silicon'):\n", | ||
" w = wf.copy()\n", | ||
" w[Filename[SampleRun]] = path\n", | ||
" results.append(w.compute(ReducibleData[SampleRun]))" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "21", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"wavelength_z_figure(results[3], wavelength_bins=400)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "22", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"wavelength_theta_figure(results, wavelength_bins=400, theta_bins=200, q_edges_to_display=[sc.scalar(0.016, unit='1/angstrom')])" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "23", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"q_theta_figure(results, q_bins=300, theta_bins=200)" | ||
] | ||
} | ||
], | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "Python 3 (ipykernel)", | ||
"language": "python", | ||
"name": "python3" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"name": "ipython", | ||
"version": 3 | ||
}, | ||
"file_extension": ".py", | ||
"mimetype": "text/x-python", | ||
"name": "python", | ||
"nbconvert_exporter": "python", | ||
"pygments_lexer": "ipython3", | ||
"version": "3.10.14" | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 5 | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The notebook makes a lot a plots. I think it would really benefit from some short explanations between the figures, explaining what we are showing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with you, but I think it's good to not make this PR any longer now. It's a basic notebook that someone experienced in the technique will at least get the gist of. Let's postpone this step to a separate PR for improving the docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I disagree, I think leaving it til later will increase the chances of us forgetting.
I don't think it's so much work to add small sentences and sections explaining a little what we are doing, especially if this is going to appear in the docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some explanations already added, maybe take a look at those and see if that's what you wanted to see there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, sorry I had missed those.