Save virtual fit results in scene and sessions p#179

[ebrahimebrahim]

This updates the fake virtual fit placeholder to save virtual fit results into the slicer scene (as transform nodes with special attribues tagged on them) and to make it so that when you go to save a Session the virtual fit results are converted into openlifu format and written into the session.

This PR comes in parallel to OpenwaterHealth/OpenLIFU-python#212 which it depends on. This PR is needed before virtual fitting is fully flesh out in order to maintain compatibility of SlicerOpenLIFU with OpenLIFU-python

This is connected to #145 but we won't close that out until we bring in the actual virtual fitting algorithm.

TODO:

• When fake VF is complete, create a VF result node with appropriate attributes.
• Add convenience functions to retrieve, filter, and sort VF result nodes.
• Make the pre-planning module VF approval button act as a toggle and operate on VF result nodes rather than on the session directly.
• Make session saving convert and write the virtual fit results
• Make session loading read in the virtual fit results into the scene
• Once Incorporate virtual fit results into Session (#179) OpenLIFU-python#212 and Fix handling of null transducer models (#218) OpenLIFU-python#219 are merged, update the python-requirements hash.
• Oops I messed up the DVC merge... first merge Fix DVC merge mistake OpenLIFU-python#221 and then update the python hash here again

[ebrahimebrahim]

@sadhana-r This is ready for review!

To test this you need to set the python-requirements OpenLIFU-python commit hash to 70e52c60f801f3be27e5488a348e1396ec9f4313, which is this branch (it's the changes coming for P#179 plus a cherry pick of the bugfix OpenwaterHealth/OpenLIFU-python#219 which is needed to work with the DVC example data)

• The virtual fit algorithm is still a placeholder, but what this PR does is to represent virtual fit results in the slicer scene. Virtual fit results are represented in the scene as just transform nodes, but they are tagged with attributes that link them to a target and possibly to a session. They are also "ranked", so if you have several virtual fit results ordered from best to worst, then their place in the ranking is represented by a node attribute (rank "1" for best, "2" for the second best, etc.).
• The way of interfacing with in-scene virtual fit results is totally encapsulated in OpenLIFULib.virtual_fit_results
• A big change from previous logic is that everything has a representation in the scene, even approval. Virtual fit approval is now an attribute on a virtual fit result transform node. Previously virtual fit approval was only represented directly in the openlifu session. So now there is a need to sync virtual fit information between the "underlying openlifu session" and the slicer scene, just like we have to for everything else in the scene.
• Things should work with or without a loaded session. When doing things without a loaded session, you end up creating "sessionless" virtual fit results. Everything works the same.
• OpenLIFULib.transform_convestion was created as part of a refactor of existing code to make it easier to go back and forth between openlifu style transducer transforms and slicer transform nodes.

Here is a snippet I was using to test the individual things in OpenLIFULib.virtual_fit_results and OpenLIFULib.transform_convestion which may or may not be helpful:

import slicer
from OpenLIFULib.virtual_fit_results import *
from OpenLIFULib.transform_conversion import *
import numpy as np
from scipy.linalg import expm
from openlifu import Transducer
from openlifu.db.session import ArrayTransform

slicer.util.selectModule("OpenLIFUData")
dw = slicer.modules.OpenLIFUDataWidget
dw.onLoadDatabaseClicked(True) # or dw.ui.databaseLoadButton.click()
dl = dw.logic

def load_session():
    dl.load_session("example_subject", "example_session")

def load_session2():
    dl.load_session("mannequin", "demo_4_simulated")

def clear():
    dl.clear_session(clean_up_scene=True)
    slicer.mrmlScene.Clear()

def make_random_matrix() -> np.ndarray:
    rng = np.random.default_rng()
    affine = np.eye(4)
    affine[:3,:3] = expm((lambda A: (A - A.T)/2)(rng.normal(size=(3,3)))) # generate a random orthogonal matrix
    affine[:3,3] = rng.random(3) # generate a random origin
    return affine

def make_random_transform_node(node_name:str) -> vtkMRMLTransformNode:
    affine = make_random_matrix()
    node = slicer.mrmlScene.AddNewNodeByClass("vtkMRMLTransformNode")
    node.SetName(node_name)
    node.SetMatrixTransformToParent(numpy_to_vtk_4x4(affine))
    return node
    

def test_virtual_fit_result_module():
    clear()
    test_transform_node = make_random_transform_node("Test Node to be turned into VF result")
    test_transform_node2 = make_random_transform_node("Test Node to be turned into VF result 2")
    test_transform_node3 = make_random_transform_node("Test Node to be cloned into VF result 3")
    test_transform_node4 = make_random_transform_node("Test Node to be turned into VF result 4")
    vf1 = add_virtual_fit_result(
        transform_node = test_transform_node,
        target_id = "target123",
        session_id = "session123",
        approval_status = True,
        clone_node = False,
    )
    vf2 = add_virtual_fit_result(
        transform_node = test_transform_node2,
        target_id = "target123",
        session_id = "session123",
        approval_status = False,
        rank = 2,
        clone_node = False,
    )
    vf3 = add_virtual_fit_result(
        transform_node = test_transform_node3,
        target_id = "a_different_target",
        session_id = "session123",
        clone_node = True,
    )
    vf4 = add_virtual_fit_result(
        transform_node = test_transform_node4,
        target_id = "target123",
        session_id = "a_different_session",
        approval_status = False,
        clone_node = False,
    )
    assert get_approval_from_virtual_fit_result_node(vf1)
    assert not get_approval_from_virtual_fit_result_node(vf3)
    assert not get_approval_from_virtual_fit_result_node(vf4)
    vfs = list(get_virtual_fit_result_nodes(target_id="target123"))
    assert len(vfs)==3
    assert vf3 not in vfs
    vfs = list(get_virtual_fit_result_nodes(session_id="session123"))
    assert len(vfs)==3
    assert vf4 not in vfs
    vfs = list(get_virtual_fit_result_nodes(target_id="target123", best_only=True))
    assert len(vfs)==2
    vf = get_best_virtual_fit_result_node(target_id="target123", session_id="session123")
    assert vf is vf1
    vf_results_openlifu = get_virtual_fit_results_in_openlifu_session_format(session_id="session123", units="m")
    assert len(vf_results_openlifu)==2
    assert len(vf_results_openlifu["a_different_target"][1])==1
    assert len(vf_results_openlifu["target123"][1])==2
    assert not vf_results_openlifu["a_different_target"][0]
    assert vf_results_openlifu["target123"][0]
    assert len(get_virtual_fit_result_nodes())==4
    clear_virtual_fit_results(target_id="target123", session_id="a_different_session")
    assert len(get_virtual_fit_result_nodes())==3
    dpn = dl.getParameterNode()
    transducer_openlifu = Transducer(units="m")
    clear()
    add_virtual_fit_results_from_openlifu_session_format(vf_results_openlifu,session_id="session1234",transducer=transducer_openlifu)
    assert len(get_virtual_fit_result_nodes())==3
    assert get_approved_target_ids(session_id="session1234") == ['target123']
    vf5 = add_virtual_fit_result(transform_node=make_random_transform_node("hello"),target_id="target123", approval_status=True)
    set_virtual_fit_approval_for_target(False, target_id="target123", session_id=None)
    assert get_virtual_fit_approval_for_target(session_id="session1234", target_id="target123")
    set_virtual_fit_approval_for_target(False, target_id="target123", session_id="session1234")
    assert not get_virtual_fit_approval_for_target(session_id="session1234", target_id="target123")

def test_transform_conversion_tools():
    clear()
    test_transform_node = make_random_transform_node("test transform node")
    array_transform = transform_node_to_openlifu(test_transform_node, "cm")
    reconstructed_transform_node = transform_node_from_openlifu(
        openlifu_transform_matrix = array_transform.matrix,
        transducer = Transducer(units="cm"),
        transform_units = array_transform.units,
    )
    reconstructed_transform_node.SetName("reconstruction")
    assert np.allclose(
        slicer.util.arrayFromTransformMatrix(reconstructed_transform_node),
        slicer.util.arrayFromTransformMatrix(test_transform_node)
    )

    clear()
    transducer_that_uses_cm = Transducer(units="cm")
    array_transform = ArrayTransform(matrix=make_random_matrix(), units="km")
    transform_node = transform_node_from_openlifu(array_transform.matrix, transducer_that_uses_cm, array_transform.units)
    reconstructed_array_transform = transform_node_to_openlifu(transform_node, "cm")
    original_array_transform_matrix_in_transducer_native_units = transducer_that_uses_cm.convert_transform(
        array_transform.matrix,
        array_transform.units,
    )
    assert reconstructed_array_transform.units == "cm" # Not necessary but happens to be the way it works; ensures the next comparison is valid
    assert np.allclose(
        original_array_transform_matrix_in_transducer_native_units,
        reconstructed_array_transform.matrix
    )

test_virtual_fit_result_module()
test_transform_conversion_tools()

What this doesn't test is all the integration into the general workflow. So besides this you'd want to check that you can use the "virtual fit" button to add a virtual fit result, that it can be saved to and loaded from a session, that approval can be toggled with the button, and that approval is revoked if the affiliated target or transform get modified.

[sadhana-r]

This is a lot of work! Great job - it is well structured/organized. It all seems to work well except for a few small things. With regards to saving the Slicer scene, I think I prefered having the scene directly update the database, or atleast on approval. Otherwise, its easy to lose track of what is and isn't saved when switching between modules. If we want to keep the slicer scene as its own state, we could prompt the user upon approval/revoking approval to save the session, or add some text to the virtual fit status message to indicate which results are saved/not saved?

A few other thoughts:

• While VF is running, it may be a good idea to disable the target combo box. This is probably only relevant now for the manual place holder, but its easy to accidently switch the target when doing manual fitting if there are multiple, and then accidently save the virtual fit result to a different target than what was intended.
• In the pre-planning module, it may be nice to add a button to apply a virtual fit result for a specific target if it already exists in the scene.

[sadhana-r]

I might be getting confused, but is calling get_virtual_fit_approvals_in_session different from calling get_approved_target_ids and providing it the session id? This separate function may not be required?

[ebrahimebrahim]

The difference is that the one in OpenLIFUDataLogic grabs the info from the underlying session, while the other one which is in the preplanning module grabs what is in the scene. Hopefully it's the same info. I do think they belong where they are, since the data module handles talking with the session and the preplanning module handles in-scene virtual fit related stuff. I felt that in the sonication planning module we should look to the session for the source of truth about approval, even though the information should hopefully match what's in the scene.

[sadhana-r]

I don't think I fully understand this. I think it would be helpful to add an example situations where clone_node would be True vs False. What does cloning do different to just taking the transform node? Is cloning a temporary functionality till the VF algorithm is in place?

[ebrahimebrahim]

It's not temporary functionality, it's just to make the function more useful. The idea is to set clone_node to False if you no longer need the original transform_node, and to set it to True if you want to preserve the integrity of the original transform_node. Setting it to False is almost equivalent to setting it to True and then deleting the original transform_node.

thanks for asking, I added a bit to clarify the docstring

[sadhana-r]

I don't seem to be getting this error? When I re-run virtual fitting when there is already a virtual fit result, I expected to see an error since replace has a default value of False but it just overwrites the result.

[ebrahimebrahim]

Confused me too for a while haha. It does work but the reason it's not called is that there is a call to clear_virtual_fit_results before add_virtual_fit_result gets called when you re-run virtual fitting. So that's deliberate: when you click "virtual fit" it's making sure to clear out any other virtual fit stuff that may already be present.

(test that the error does work)

from OpenLIFULib.virtual_fit_results import add_virtual_fit_result

test_transform_node = slicer.mrmlScene.AddNewNodeByClass("vtkMRMLTransformNode")
test_transform_node2 = slicer.mrmlScene.AddNewNodeByClass("vtkMRMLTransformNode")
vf1 = add_virtual_fit_result(
    transform_node = test_transform_node,
    target_id = "target123",
    session_id = "session123",
    approval_status = False,
    clone_node = True,
)
vf2 = add_virtual_fit_result(
    transform_node = test_transform_node2,
    target_id = "target123",
    session_id = "session123",
    approval_status = False,
    clone_node = False,
) # This should raise a RuntimeError unless you set replace=True

[ebrahimebrahim]

In the pre-planning module, it may be nice to add a button to apply a virtual fit result for a specific target if it already exists in the scene.

Agreed! Let me punt this improvement to #139, which is coming soon after the algorithm is implemented.

[ebrahimebrahim]

(Will rebase after it's approved, just added a few commits on top to make the re-review easier)

[ebrahimebrahim]

@sadhana-r, after you approved I added one non-trivial commit 00014a0 to update things to work with the new OpenLIFU-python. It's because in OpenwaterHealth/OpenLIFU-python@611ff30 there was a small change to the way transducer absolute path info is represented. In your re-review you can just check this one commit. I already tested that the mannequin and example_subject examples can be loaded.

[sadhana-r]

looks good to me!