Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Requirement] Masking tools #50

Closed
celinedurniak opened this issue Apr 16, 2024 · 10 comments
Closed

[Requirement] Masking tools #50

celinedurniak opened this issue Apr 16, 2024 · 10 comments

Comments

@celinedurniak
Copy link
Collaborator

celinedurniak commented Apr 16, 2024

Executive summary

Provide masks for different physical quantities: wavelength, 2 theta, intensity and detectors

Context and background knowledge

These masks are first to be used in the Powder Diffraction workflow for DREAM.
Different physical parameters could be used in order to mask:

  • Faulty detectors
  • Contribution from SE or instrument component in the detected signal
  • Debye-Scherrer rings
  • Too-high, too low or uneven distribution of intensity

Follow-up:

  • adapt implementation with DREAM NeXus file
  • use widget to determine masks (optional and for some physical quantities only)

Inputs

  • For masks in 2 theta, wavelength: DataArray containing the quantity to be masked as coordinate

  • For masks of detectors: metadata about detectors should be stored (positions, other characteristics like sumo, segment, sector…)

Methodology

  • For the detectors, the masks could be specified either by the spatial coordinates of the voxels to mask or by sub-elements (i.e., sumo, sector, wire, strip). It could be a set like bank (hr, sans, endcap fwd, endcap bwd, mantle), sector (for SANS and HR) or sumo (for endcaps), module, segment, wire(s) and or strip(s), counter

Outputs

Data array with the same structure as input and with added mask.

Which interfaces are required?

Integrated into reduction workflow, Python module / function

Test cases

  • Mask in two theta, intensity: data from D16 @ ILL link1 or DREAM csv files below
  • Detector mask, wavelength: DREAM csv data (any csv files in link2)

Examples:

  • mask in 2 theta

Figure_2theta

  • mask of pixels (2d detectors for D16 @ ILL)

Figure_mask_pix

Comments

Note: The file from D16 can be used in the online documentation.

@nvaytet
Copy link
Member

nvaytet commented Apr 25, 2024

So masking should be part of a workflow.
Would you expect the masks to be different for each run/sample during an experiment or can you use a mask that was saved to a file?
I'm trying to figure out how this would fit in a pipeline?

Would you have a set of parameters at the top of your notebook like a list of wavelength or two_theta ranges that would defined what to mask, or do you need to have an interactive tool where you can draw masks?
I guess maybe a mixture of the 2?

The former could be something like (API to be discussed):

params[TwothetaMasks] = [sc.array(dims=['two_theta'], values=[53, 55], units='deg),
                         sc.array(dims=['two_theta'], values=[80, 84], units='deg)]

params[WavelengthMasks] = [sc.array(dims=['wavelength'], values=[3.2, 3.7], units='angstrom)]

An interactive tool could be used to save masks to file, and then we could load them like we do with the SANS workflows (we don't have to use xml format, we can use something else).

@nvaytet
Copy link
Member

nvaytet commented Apr 25, 2024

Another question: the lines in the two_theta plots are curved. Will there be some curved lines that we also want to mask or will the unwanted signal not be curved?
If so, is it possible to convert the data into a coordinate (maybe d-spacing?) where the lines are straight? If we can delay the masking until then, it will be much easier to accurately mask features either drawing rectangles or using a range of values as in the example code snippet above.

@celinedurniak
Copy link
Collaborator Author

About the 2nd question: In principle, the ranges of two theta to mask could be linear or curved. For this first implementation, we could focus on linear masking only and split this requirement into 2 parts with the mask in 2theta to be implemented later. Let me know what you think.

About the 1st question, "Would you expect the masks to be different for each run/sample during an experiment or can you use a mask that was saved to a file?"
It is a mixture of both (as you guessed):

  • for example with the polariser in place, masking in wavelength will have to be applied to all runs corresponding to this setting (cycle, several beamtimes...)
  • masking in 2theta could be different from sample to sample (for example, different textures)
  • for the detectors, some masking should be in the NeXus files if issues with some detectors are picked up by NICOS. But additional masking should be implemented during reduction.
  • masking in intensity could be detector- or sample-dependent

@nvaytet
Copy link
Member

nvaytet commented Apr 30, 2024

Looking at the workflow in https://scipp.github.io/essdiffraction/examples/POWGEN_data_reduction.html#Compute-final-result, we are converting the raw data directly to d-spacing.

Since you would like to have masks for detector pixels, wavelength ranges, and 2theta ranges, is it correct that we should add intermediate steps in the workflow that converts the data to the relevant coordinates?

What I mean is we could:

  1. Mask pixels after the data has been normalized to proton charge: have a function that takes in NormalizedByProtonCharge[SampleRun], adds pixel masks, and then returns e.g. PixelMaskedData[SampleRun].

  2. Mask one or more wavelength ranges: convert PixelMaskedData[SampleRun] to WavelengthData[SampleRun]. Then have a function that takes in WavelengthData[SampleRun] and returns WavelengthMaskedData[SampleRun].

  3. Mask in 2theta: convert WavelengthMaskedData[SampleRun] to TwoThetaData[SampleRun]. Then have a function that takes in TwoThetaData[SampleRun] and returns MaskedTwoThetaData[SampleRun].

  4. Finally, change to_dspacing_with_calibration to accept MaskedTwoThetaData[SampleRun] as input instead of NormalizedByProtonCharge[SampleRun].

Does that sounds like a good plan?

Question: would you also need to mask in d-spacing? i.e. add a MaskedDspacingData[SampleRun]? If not immediately required, I will leave it out for now. We can always add it later.

@celinedurniak
Copy link
Collaborator Author

@nvaytet Your plan is good. Here are a few comments

The conversion to $\lambda$ and $2\theta$ should be added to a more complete version of the workflow anyway. Their presence of a specific mask also depends on the type of experiments: for example, powder diffraction with known texture, parasitic scattering by some SEs….
These masks will also be essential tools for cold and hot commissioning of the instrument.

For the workflow of POWGEN

  1. Masking pixels could be also done at an earlier stapge in the reduction. For example, after loading if some counts are too high.
  2. OK
  3. OK
  4. OK for the POWGEN workflow. But for a more general approach, can’t we have both inputs?

A mask in d-spacing would also be required. But it can be left out of this first implementation after having tested the other types of masks.

@nvaytet
Copy link
Member

nvaytet commented May 6, 2024

Thanks for the reply.
Another question: in one of your images, the mask in 2theta is a vertical band that spans all pixels. Could we apply masks on a 1d plot of intensity vs 2theta? (i.e. making a 1d mask)

@jl-wynen
Copy link
Member

jl-wynen commented May 6, 2024

Alternatively, is the 'spectrum' dim in those pictures a stand-in for another dim? E.g., phi (azimuthal angle) or y?

@celinedurniak
Copy link
Collaborator Author

@nvaytet I think the 2D mask should be implemented since users will first look at 2D data and decide to create mask at this stage. Visualising data in 1D comes at a later stage in the pipeline after a bit of clean up.

@jl-wynen the vertical axis has been renamed as 'spectrum' when reshaping the initial NeXus file, so this axis could be associated to other physical quantities.

This was referenced May 21, 2024
@nvaytet nvaytet moved this from Selected to In progress in Development Board May 30, 2024
@nvaytet
Copy link
Member

nvaytet commented Jun 19, 2024

@celinedurniak when you find the time could you look at #55 and scipp/scippneutron#521 to see if those combined satisfy your needs?

Both are currently unreleased so you have to get essdiffraction and scippneutron from main, let me know if you need assistance with that.

@celinedurniak
Copy link
Collaborator Author

celinedurniak commented Jun 23, 2024

@nvaytet I had a look at 2 notebooks masking_tools.ipynb and `` #55 and scipp/scippneutron#521 are fine as first implementations.

Here are a few additional comments / questions:

  • An example showing how to remove one of the masks would be useful because it is likely that masks won't be set correctly at the first try.
  • When setting the masks, are the input values always in the same unit as the displayed parameters (i.e., x, y for the example in the notebook)?
  • I do not have any interactivity with the masking tools of the plot containing the masks (widgets at the top). I can click on the buttons but nothing happens. SOLVED: the issue was caused by the execution of a cell which made the figure static*
  • How can I change the color of the masks individually?

The comments above do not prevent this requirement to be closed

@nvaytet nvaytet closed this as completed Jun 26, 2024
@github-project-automation github-project-automation bot moved this from In progress to Done in Development Board Jun 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

No branches or pull requests

3 participants