picking vertex in brain object to see time course

[agramfort]

I would be great to get a plot of a time course of a vertex on a 3D brain object.

a suggestion from @SophieHerbst is also to have a short cut to pick automatically the vertex with the highest amplitude in the current view.

@GuillaumeFavelier

[larsoner]

I would be great to get a plot of a time course of a vertex on a 3D brain object.

At least in principle there is vtkChart:

• https://vtk.org/doc/nightly/html/classvtkChart.html
• https://vtk.org/Wiki/VTK/Examples/Cxx/Plotting/LinePlot

a suggestion from @SophieHerbst is also to have a short cut to pick automatically the vertex with the highest amplitude in the current view.

I'm actually not sure the extent to which this will be possible. Picking with the mouse is usually done by rendering tricks or casting a ray (I think VTK does the latter). I'm not sure there is a way to ask for "all mesh vertices currently visible). And given a non-zero alpha, this isn't even necessarily well defined. So for a first pass I say we just make it so that clicking on a point in the mesh does mesh vertex picking and does a vtkChart plot of the time course. (If off the mesh, don't change the point currently used. This way clicking always will show the time course, and if you don't want it to change, click in the empty space around the mesh. Either that or use ctrl-click for picking or something.)

I leave it as an exercise to @GuillaumeFavelier to figure out where this plot should live :) Maybe a show_timecourse=True would make a new full-width (regardless of views) plot area at the bottom reserved for a time course chart like our volume viewer:

https://mne.tools/dev/auto_examples/inverse/plot_lcmv_beamformer_volume.html#plot-source-space-activity

[jasmainak]

why not extend the functionality of 4D viewer with cortical ribbon first?

[larsoner]

It's nicer to view surface source estimate data on the surface

[GuillaumeFavelier]

That's a nice idea and thank you so much for the pointers @larsoner, I have somewhere to start now.

[larsoner]

And for mouse click vertex picking in mayavi (so you can translate to PyVista) see

https://github.com/mne-tools/mne-python/blob/master/mne/gui/_fiducials_gui.py#L359

[GuillaumeFavelier]

Vertex picking is available by default in PyVista:

[SophieHerbst]

Would this allow to spit out MNI coordinates at the pick (or the label of a pre-defined atlas)?

[SophieHerbst]

Also, a very small add-on: labels for the 3 clim-scales would be helpful.

[GuillaumeFavelier]

I think any data associated to the point data/vertex id could be retrieved that way in theory.

[GuillaumeFavelier]

I discovered vertex_to_mni() to take care of the conversion.

[GuillaumeFavelier]

Also, a very small add-on: labels for the 3 clim-scales would be helpful.

fmax/fmid/fmin labels were present in previous version but were dropped in favour of a more compact UI.

[GuillaumeFavelier]

ToDo (suggested in #7247 (comment))

• Display the MNI coordinates at the pick (suggested in picking vertex in brain object to see time course #7235 (comment))
• Enable Picking & Plot of time courses with show_traces=True
• Display the picked point in 3d with a dot glyph (sphere)
• Add a time course to the chart by picking on the surface vertex
• Remove an existing time course in the chart by clicking on the dot glyph
• Display the chart with vtkChartXY in split view or embed the MPL figs in a Qt frame
• Show the time course of the vertex with max(abs(t)) by default (suggested in MRG: Enable point picking in _TimeViewer #7247 (comment))
• Use mouse click instead of p to trigger picking (suggested in MRG: Enable point picking in _TimeViewer #7247 (comment))
• Remove the red bounding box appearing during picking
• Update the 3d backend overview table
• Use cell picking instead of point picking (expecting better precision)
• Use a reasonable radius for the glyphs (suggested in MRG: Enable point picking in _TimeViewer #7247 (comment))
• Scale when hemi=both or hemi=split
• Use show_traces=separate to detach the matplotlib canvas (suggested in picking vertex in brain object to see time course #7235 (comment))
• Add support for views (suggested in MRG: Enable point picking in _TimeViewer #7247 (comment))
• Add testing of picking features (suggested in MRG: Enable point picking in _TimeViewer #7247 (review))

[larsoner]

+10 to embedding a mpl figure rather than figuring out vtkChartXY, it gives us more flexibility going forward, e.g., including showing time-frequency images eventually, and will unify the user experience of 2D plots around MPL and its standard interactions (like zooming and view resetting). The "only" tricky parts of this approach will be:

• setting up our own Qt window instead of relying on PyVista and MPL to do it for us, and then also
• eventually making sure that when in a notebook we embed the canvases properly rather than creating a Qt window