Ultrasound Simulator: ToolTimeStampedUpdate failed for tool...

[olivares471]

Hi everyone,

Basically I'm currently working on a way to reproduce the interactions between Slicer3D and the PLUS Toolkit's Ultrasound simulator, but using Unity instead of Slicer in order to easily create a 3D environment supporting VR interactions. My project revolves around simulating, in real time, an ultrasound image of the forearm of a patient (3D mesh).

I tried to understands how Slicer3D communicates with PLUS via IGT and to reproduce it in Unity/C#. The Unity system I'm working on works by creating a TCP Listener to which the PLUS US Sim will connect to in order to send it various tool's transform matrices. Simultaneously, I create a TCP client that connects to PLUS and that will receive the simulated ultrasound image data.

My problem is that my solution works very briefly before the PlusServer.exe stops responding and shows me a few errors. I was wondering if some of you would have an idea of what it could mean.

Right after sending the starting transforms and receiving the first US image, PLUS seems to stop working and sends multiple error logs. This next screenshot show the errors after the first data exchange. The error seems to be Failed to get oldest timestamp: no overlap between tracking and video data[...]:

After a few logs of this error, I get multiple Failed to get video buffer item by timestamp [X ...]. And then I get a ToolTimeStampedUpdate failed for tool: message until I close the PlusServer.exe. This next screenshot show all of these errors.

I'm pretty confident that at least a few data messages are exchanged between my solution and PLUS before it stops working since my mesh objects (tools) are move to the right position/rotation in the PLUS 3D space and I receive at least one image from PLUS that I can display (as seen on the screenshot bellow)

It might be important to note that this behavior is from the latest version of Unity 2022 LTS. I have the same code running on Unity 2021 LTS and this issue takes a bit more time to arise. I'm able to exchange data with PLUS for a few minutes before the same issue happens.

From what I can tell, maybe there is a timestamp issue between the data messages I send and PLUS, but I'm not sure how that would be possible since from what I understand PLUS is not using the timestamp (always 0) I send via the data message containing the transform matrix of X tool. Would anyone know what these error messages could mean?

Thanks!

[olivares471]

Just in case someone has this same error in the future, I will post my solution.

To communicate with PLUS server I created a TCP listener and a TCP client. The TCP listener would allow the PLUS application to connect to it in order for Unity to send my transform data via a coroutine.

The process to receive PLUS' image is similar but inverted. This time my TCP client connects to the PLUS application server to receive the images sent by PLUS. My listenToPLUSToolkitData function is also executed in a coroutine.

My workaround to this issue was to simply add a yield return new WaitForSeconds(0.02f) to both my coroutines. I think the problem comes from the fact that I was sending transform data way too fast to PLUS and that's what created the issues. I'm still working on it and trying to figure out what is the actual problem (it's probably that I'm sending data too fast either to PLUS or to my TCP Listener). I'm also still trying to figure out what is the best wait time. For now 0.02 seconds seems to be good enough.