WIP: updates for current LiberTEM version #10

sk1p · 2022-07-18T13:13:08Z

I tried to reanimate this plugin for current LiberTEM (and nion) versions. The code runs so far as it doesn't crash at start-up, and it's possible to select files for opening. Sadly, after clicking the "Open" button in the system file dialog, the following error is logged, and clicking the "Load" button doesn't seem to have an effect, beyond closing the load dialog:

Traceback (most recent call last):


  File "/home/alex/miniconda/envs/nion/lib/python3.10/site-packages/nion/ui/PyQtProxy.py", line 271, in __periodic
    self.object.periodic()


  File "/home/alex/miniconda/envs/nion/lib/python3.10/site-packages/nion/ui/QtUserInterface.py", line 1931, in periodic
    self._handle_periodic()


  File "/home/alex/miniconda/envs/nion/lib/python3.10/site-packages/nion/ui/UserInterface.py", line 3506, in _handle_periodic
    self.on_periodic()


  File "/home/alex/miniconda/envs/nion/lib/python3.10/site-packages/nion/ui/Window.py", line 364, in periodic
    self.__event_loop.run_forever()


  File "/home/alex/miniconda/envs/nion/lib/python3.10/asyncio/base_events.py", line 590, in run_forever
    self._check_running()


  File "/home/alex/miniconda/envs/nion/lib/python3.10/asyncio/base_events.py", line 582, in _check_running
    raise RuntimeError('This event loop is already running')


RuntimeError: This event loop is already running

I'm not sure what is going wrong there - I thought maybe this is because of fork safety issues, but setting the multiprocessing start method to spawn didn't seem to fix things. Maybe I also did something wrong when installing nion swift, as I didn't find the right link to the development setup docs and did things from memory.

TODO

Fix the above-mentioned bug
Further updates for I/O parameter handling (i.e. update for current formats and parameters supported by LiberTEM)
Maybe updates regarding to the current nion swift version?

sk1p · 2022-07-18T13:23:22Z

I'm not sure what is going wrong there - I thought maybe this is because of fork safety issues, but setting the multiprocessing start method to spawn didn't seem to fix things. Maybe I also did something wrong when installing nion swift, as I didn't find the right link to the development setup docs and did things from memory.

Current dask (since dask/distributed#3461) uses spawn as default anyways, so that was not it.

Brow71189 · 2022-07-20T06:59:35Z

Hm I get a different error when trying to load a file:

File "C:\Users\Andi\Miniconda3\envs\developer\lib\threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
File "C:\Users\Andi\Miniconda3\envs\developer\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
File "C:\Users\Andi\Miniconda3\envs\developer\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
File "D:\git\nionswift\nion\swift\DocumentController.py", line 2381, in receive_files_on_thread
    traceback.print_stack()
Traceback (most recent call last):
File "D:\git\nionswift\nion\swift\DocumentController.py", line 2375, in receive_files_on_thread
    data_items = ImportExportManager.ImportExportManager().read_data_items(file_path)
File "D:\git\nionswift\nion\swift\model\ImportExportManager.py", line 155, in read_data_items
    return io_handler.read_data_items(extension, path)
File "D:\git\nionswift\nion\swift\model\ImportExportManager.py", line 57, in read_data_items
    data_items.extend(self._read_data_items(extension, file_path))
File "D:\git\nionswift\nion\swift\model\ImportExportManager.py", line 62, in _read_data_items
    data_elements = self.read_data_elements(extension, file_path)
File "D:\git\nionswift\nion\swift\Facade.py", line 3193, in read_data_elements
    data_and_metadata = self.__io_handler_delegate.read_data_and_metadata(extension, str(file_path))
File "D:\git\libertem-hackaton\nionswift_plugin\LiberTEM_IO\__init__.py", line 70, in read_data_and_metadata
    return self.read_data_and_metadata_from_stream(file_path)
File "D:\git\libertem-hackaton\nionswift_plugin\LiberTEM_IO\__init__.py", line 100, in read_data_and_metadata_from_stream
    result = UDFRunner(PickUDF()).run_for_dataset(ds, executor, roi=roi)
File "C:\Users\Andi\Miniconda3\envs\developer\lib\site-packages\libertem\udf\base.py", line 1931, in run_for_dataset
    for res in self.run_for_dataset_sync(
File "C:\Users\Andi\Miniconda3\envs\developer\lib\site-packages\libertem\udf\base.py", line 2005, in run_for_dataset_sync
    for part_results, task in result_iter:
File "C:\Users\Andi\Miniconda3\envs\developer\lib\site-packages\libertem\udf\base.py", line 1954, in results_for_dataset_sync
    tasks, params = self._prepare_run_for_dataset(
File "C:\Users\Andi\Miniconda3\envs\developer\lib\site-packages\libertem\udf\base.py", line 1885, in _prepare_run_for_dataset
    input_dtype=self._get_dtype(dataset.dtype, corrections),
File "C:\Users\Andi\Miniconda3\envs\developer\lib\site-packages\libertem\udf\base.py", line 1636, in _get_dtype
    for udf in self._udfs:
TypeError: 'PickUDF' object is not iterable

Just to make sure we are doing the same thing: I clicked "File -> Import Data..." in Swift, selected an hdf5 file on disk and clicked "Open". That brings up the libertem-plugin dialog for loading files. After filling in the fields in the dialog I clicked "Load" in it, which brings up the above error.

Installation instructions for a developer installation of Nion Swift are here: https://github.com/nion-software/nionswift/wiki/Developer-Installation#installation-notes

I will also try this again with a fresh installation when I get to it.

Just quickly looking into the source code at the place where your error occurs: https://github.com/nion-software/nionui/blob/105cdf44961f99bc8e6561546050117c73e63320/nion/ui/Window.py#L364

Here the event loop gets stopped and then started immediately again. Is is possible that other code also using the event loop can prevent it from being stopped? Or is there a place in libertem that periodically starts the event loop?

sk1p · 2022-07-20T12:12:10Z

Hi Andreas, and thank you for your support!

Hm I get a different error when trying to load a file:

File "C:\Users\Andi\Miniconda3\envs\developer\lib\threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
[...]
File "D:\git\libertem-hackaton\nionswift_plugin\LiberTEM_IO\__init__.py", line 100, in read_data_and_metadata_from_stream
    result = UDFRunner(PickUDF()).run_for_dataset(ds, executor, roi=roi)
[...]
File "C:\Users\Andi\Miniconda3\envs\developer\lib\site-packages\libertem\udf\base.py", line 1636, in _get_dtype
    for udf in self._udfs:
TypeError: 'PickUDF' object is not iterable

Ah, I see the problem - I didn't even get that far. I think my installation was not correct and I'll try this from scratch.

This error comes from using internal APIs that changed in the meantime - the UDFRunner these days accepts a list of UDFs instead of just one, to run multiple UDFs "in parallel" on the data stream. I'll try to reproduce this error and replace the UDFRunner usage our newer stable API.

Just to make sure we are doing the same thing: I clicked "File -> Import Data..." in Swift, selected an hdf5 file on disk and clicked "Open". That brings up the libertem-plugin dialog for loading files. After filling in the fields in the dialog I clicked "Load" in it, which brings up the above error.

Sounds right - I did the same with an MIB dataset.

Installation instructions for a developer installation of Nion Swift are here: https://github.com/nion-software/nionswift/wiki/Developer-Installation#installation-notes

Thanks!

Just quickly looking into the source code at the place where your error occurs: https://github.com/nion-software/nionui/blob/105cdf44961f99bc8e6561546050117c73e63320/nion/ui/Window.py#L364

Here the event loop gets stopped and then started immediately again. Is is possible that other code also using the event loop can prevent it from being stopped? Or is there a place in libertem that periodically starts the event loop?

Hm, no - the only place we are messing with event loops is inside worker threads we start ourselves - outside of that, we just use the existing event loop. I think I really messed up when installing, as a pip install -e . in nionswift pulled some dependencies from pypi (as far as I remember, it was nionui and nionutils, but I could be wrong), which are then of course not the right version. I tried to overwrite them using the current git version, but I guess I missed something.

sk1p · 2022-07-20T12:18:06Z

Btw. I tested on Python 3.10, maybe the Python version also plays a role

sk1p · 2022-07-20T15:47:28Z

I can reproduce the original traceback (""event loop is already running") with the following script:

#!/bin/bash

set -eux

conda remove -y -n nion --all
conda create -y -n nion -c conda-forge python=3.10 scipy pytz h5py imageio pillow tzlocal pip mypy -y

PYTHON=/home/alex/miniconda/envs/nion/bin/python

$PYTHON -m pip install --no-deps --editable nionutils
$PYTHON -m pip install --no-deps --editable niondata
$PYTHON -m pip install --no-deps --editable nionui
$PYTHON -m pip install --no-deps --editable nionswift
$PYTHON -m pip install --no-deps --editable nionswift-io
$PYTHON -m pip install --no-deps --editable nionswift-instrumentation-kit
$PYTHON -m pip install --no-deps --editable nionswift-video-capture
$PYTHON -m pip install --no-deps --editable nionswift-eels-analysis
$PYTHON -m pip install --no-deps --editable nionswift-experimental

pwd
$PYTHON -m pip install --editable libertem-hackaton

$PYTHON -m pip install pyside2

The part where I had to deviate from the wiki was the installation of pyside2, as qt5-default does not install on my system (Debian might be in a transition to a different Qt version?)

When I install PyQt5 instead, I get another error:

Loading project /home/alex/.local/share/Nion/Nion Swift/Work.nsproj
Traceback (most recent call last):

  File "/home/alex/source/nion/nionswift/nion/swift/Application.py", line 331, in __open_default_project

    
document_controller = self.open_project_window(project_reference, update_last_project_reference)


  File "/home/alex/source/nion/nionswift/nion/swift/Application.py", line 366, in open_project_window

    
document_controller = self.create_document_controller(document_model, "library", project_reference=project_reference)


  File "/home/alex/source/nion/nionswift/nion/swift/Application.py", line 510, in create_document_controller

    
document_controller = DocumentController.DocumentController(self.ui, document_model, workspace_id=workspace_id,


  File "/home/alex/source/nion/nionswift/nion/swift/DocumentController.py", line 201, in __init__

    
self.__workspace_controller = Workspace.Workspace(self, workspace_id)


  File "/home/alex/source/nion/nionswift/nion/swift/Workspace.py", line 223, in __init__

    
root_widget = self.ui.create_column_widget(properties={"min-width": 640, "min-height": 480})


  File "/home/alex/source/nion/nionui/nion/ui/QtUserInterface.py", line 2092, in create_column_widget

    
return UserInterface.BoxWidget(QtBoxWidgetBehavior(self.proxy, "column", properties), alignment)


  File "/home/alex/source/nion/nionui/nion/ui/QtUserInterface.py", line 666, in __init__

    
super().__init__(proxy, widget_type, properties)


  File "/home/alex/source/nion/nionui/nion/ui/QtUserInterface.py", line 480, in __init__

    
self.update_properties()


  File "/home/alex/source/nion/nionui/nion/ui/QtUserInterface.py", line 506, in update_properties

    
self.proxy.Widget_setWidgetProperty(self.widget, key, self.proxy.encode_variant(self.properties[key]))


  File "/home/alex/source/nion/nionui/nion/ui/PyQtProxy.py", line 4073, in Widget_setWidgetProperty

    
widget.setMinimumWidth(value * display_scaling)


TypeError
: 
setMinimumWidth(self, int): argument 1 has unexpected type 'float'



During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/home/alex/miniconda/envs/nion/bin/nionswift", line 33, in <module>

    
sys.exit(load_entry_point('nionswift', 'console_scripts', 'nionswift')())


  File "/home/alex/source/nion/nionswift/nion/swift/command.py", line 93, in main

    
app.run()


  File "/home/alex/source/nion/nionui/nion/ui/Application.py", line 107, in run

    
self.ui.run(self)


  File "/home/alex/source/nion/nionui/nion/ui/QtUserInterface.py", line 2065, in run

    
self.proxy.run(app)


  File "/home/alex/source/nion/nionui/nion/ui/PyQtProxy.py", line 2273, in run

    
if application.start():


  File "/home/alex/source/nion/nionswift/nion/swift/Application.py", line 304, in start

    
return self.__open_default_project(profile_dir, is_created)


  File "/home/alex/source/nion/nionswift/nion/swift/Application.py", line 333, in __open_default_project

    
self.show_ok_dialog(_("Error Opening Project"), _("Unable to open default project."), completion_fn=self.show_choose_project_dialog)


  File "/home/alex/source/nion/nionui/nion/ui/Application.py", line 209, in show_ok_dialog

    
Declarative.WindowHandler(completion_fn=completion_fn).run(window, app=self)


  File "/home/alex/source/nion/nionui/nion/ui/Declarative.py", line 1134, in run

    
self.window = run_window(d, self, app=app, parent_window=parent_window, window_style=window_style, persistent_id=persistent_id)


  File "/home/alex/source/nion/nionui/nion/ui/Declarative.py", line 1178, in run_window

    
inner_content = construct(ui, window, content, handler, finishes)


  File "/home/alex/source/nion/nionui/nion/ui/Declarative.py", line 1353, in construct

    
return construct_box(ui, window, ui.create_column_widget(properties=properties), d, handler, finishes)


  File "/home/alex/source/nion/nionui/nion/ui/QtUserInterface.py", line 2092, in create_column_widget

    
return UserInterface.BoxWidget(QtBoxWidgetBehavior(self.proxy, "column", properties), alignment)


  File "/home/alex/source/nion/nionui/nion/ui/QtUserInterface.py", line 666, in __init__

    
super().__init__(proxy, widget_type, properties)


  File "/home/alex/source/nion/nionui/nion/ui/QtUserInterface.py", line 480, in __init__

    
self.update_properties()


  File "/home/alex/source/nion/nionui/nion/ui/QtUserInterface.py", line 506, in update_properties

    
self.proxy.Widget_setWidgetProperty(self.widget, key, self.proxy.encode_variant(self.properties[key]))


  File "/home/alex/source/nion/nionui/nion/ui/PyQtProxy.py", line 4089, in Widget_setWidgetProperty

    
widget.setMinimumWidth(value * display_scaling)


TypeError
: 
setMinimumWidth(self, int): argument 1 has unexpected type 'float'

I also tried to use nionswift-tool to start, but that's broken in a different interesting way... the launch script in miniconda/envs/nion/bin/nionswift-tool has an interesting shebang: #!/usr/share/miniconda/envs/test/conda-bld/nionswift-tool_1651765688596/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/bin/python

This somehow causes the file to be executed by the shell, and import is run as a shell command, causing unrelated error messages...

In the end, this doesn't change the error though - if I patch nionswift-tool with a reasonable shebang, I still get the event loop error...

WIP: updates for current LiberTEM version

9eec48b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: updates for current LiberTEM version #10

WIP: updates for current LiberTEM version #10

Uh oh!

sk1p commented Jul 18, 2022

Uh oh!

sk1p commented Jul 18, 2022

Uh oh!

Brow71189 commented Jul 20, 2022

Uh oh!

sk1p commented Jul 20, 2022

Uh oh!

sk1p commented Jul 20, 2022

Uh oh!

sk1p commented Jul 20, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

WIP: updates for current LiberTEM version #10

Are you sure you want to change the base?

WIP: updates for current LiberTEM version #10

Uh oh!

Conversation

sk1p commented Jul 18, 2022

TODO

Uh oh!

sk1p commented Jul 18, 2022

Uh oh!

Brow71189 commented Jul 20, 2022

Uh oh!

sk1p commented Jul 20, 2022

Uh oh!

sk1p commented Jul 20, 2022

Uh oh!

sk1p commented Jul 20, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants