Cannot Complete Service Call with ros2 cli.

[Carter12s]

Advertising a service server via roslibrust -> ros-z.

Making a call to service server with ros2 service call

I can see from Zenoh logs transaction is mostly completing, however take_request() and take_request_async() never actually yield a value, and ros2 cli hangs indefinitely waiting for request to complete.

I've debugged reasonably deeply, but cannot spot how the Query is failing to be triggered.

Logs from server show queryable being declared:

[2025-09-18T19:27:52Z DEBUG zenoh::net::routing::dispatcher::queries] Face{0, 56a64d2b280d8663a1fa44c87cb6c240} Declare queryable 5 (0/set_bool/std_srvs::srv::dds_::SetBool_/RIHS01_abe9e4bb6b41b40e6789712c00ec8871923e089af3f667a79992a428cff2da0a)
[2025-09-18T19:27:52Z DEBUG zenoh::net::routing::dispatcher::token] Face{0, 56a64d2b280d8663a1fa44c87cb6c240} Declare token 6 (@ros2_lv/0/56a64d2b280d8663a1fa44c87cb6c240/0/1/SS/%/%/test_service_server_callable_node/%set_bool/std_srvs::srv::dds_::SetBool_/RIHS01_abe9e4bb6b41b40e6789712c00ec8871923e089af3f667a79992a428cff2da0a/1:2:1,10:,:,:,,)
[2025-09-18T19:27:52Z DEBUG zenoh::net::routing::dispatcher::resource] Register resource @ros2_lv/0/56a64d2b280d8663a1fa44c87cb6c240/0/1/SS/%/%/test_service_server_callable_node/%set_bool/std_srvs::srv::dds_::SetBool_/RIHS01_abe9e4bb6b41b40e6789712c00ec8871923e089af3f667a79992a428cff2da0a/1:2:1,10:,:,:,,

Logs from ros2 cli show query being executed and that a "final reply" was received:

requester: making request: std_srvs.srv.SetBool_Request(data=True)

2025-09-18T19:28:41.479056Z DEBUG ThreadId(02) zenoh::net::routing::dispatcher::queries: Face{1, 5c14e3490ccee35fcfdeab919696a098}:0 Route query for res 0/set_bool/std_srvs::srv::dds_::SetBool_/RIHS01_abe9e4bb6b41b40e6789712c00ec8871923e089af3f667a79992a428cff2da0a
2025-09-18T19:28:41.479158Z DEBUG  rx-1 ThreadId(09) zenoh::net::routing::dispatcher::queries: Face{2, 30a60dd41571db2fd4db54ff5a704eda}:1 Received final reply for query Face{1, 5c14e3490ccee35fcfdeab919696a098}:0
2025-09-18T19:28:41.479166Z DEBUG  rx-1 ThreadId(09) zenoh::net::routing::dispatcher::queries: Face{1, 5c14e3490ccee35fcfdeab919696a098}:0 Propagate final reply

[YuanYuYuan]

It was a bug, and I believe it has been fixed in #11.

[Carter12s]

Interestingly this issue is still occurring for me on the main branch as of today, at this point it may very well be an environment issue, but I did want to at least re-flag it for your review.

I'm testing in a docker image built on top of ros-kilted-core

Exact image is here

In this docker image if I execute cargo test --features interop-tests in the ros-z-tests directory with the kilted environment sourced. I get:

cargo test --features interop-tests
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.14s
     Running tests/pubsub_interop.rs (/ros-z/target/debug/deps/pubsub_interop-8bf85a3d7a7087fc)

running 2 tests
test test_ros2_pub_to_ros_z_sub ... ok
test test_ros_z_pub_to_ros2_sub ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.87s

     Running tests/service_interop.rs (/ros-z/target/debug/deps/service_interop-0359a05fd81e63c4)

running 3 tests
Package 'demo_nodes_cpp' not found
test test_ros_z_server_ros_z_client ... ok
test test_ros2_server_ros_z_client ... ok
test test_ros_z_server_ros2_client ... FAILED

failures:

---- test_ros_z_server_ros2_client stdout ----

=== Test: ros-z server <-> ROS2 client ===
Server ready for ROS2 client...
Received request from ROS2: 15 + 9
Sending response: 24
ROS2 output: waiting for service to become available...


thread 'test_ros_z_server_ros2_client' panicked at ros-z-tests/tests/service_interop.rs:138:5:
Expected sum: 17, got: waiting for service to become available...

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    test_ros_z_server_ros2_client

test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.42s

error: test failed, to rerun pass `--test service_interop

All tests other than ros_z_server_ros2_client pass. This was the previous behavior I was seeing with roslibrust before these tests existed. I'll poke around and try to figure out why that particular test has issues in this environment, but it does seem like there may be a compatibility issue somewhere.

[Carter12s]

@YuanYuYuan I think this should be re-opened, or I'm really missing something.

I've recreated above but based on rolling instead of kilted, so I am seeing the same results on "latest ROS".

I've so far been unable to replicate your tests passing in any of my environments. Well... All tests pass except for test_ros_z_server_ros2_client

[YuanYuYuan]

Hi @Carter12s! Thank you for testing it. The error log you pasted said "demo_node_cpp" is missing. You need to install it in the container.

ros-z/.github/workflows/test.yml

Line 24 in a8c732c

ros-jazzy-demo-nodes-cpp

Since we don't use package.xml to manage dependencies like ROS, we will frequently encounter this error. IMO, the best way is to use Nix as a solution, like I demonstrated in the CI. But it's definitely too new for most people. Anyways, let's reopen the issue and then fix it by explicitly printing an error.

[Carter12s]

@YuanYuYuan

Thanks for flagging the demo_node_cpp issue, and appreciate the updated error message for it! I've heard good things about nix, but have yet to adopt it into my workflows. Seeing your setups here, this may be a good time give it a spin.

Unfortunately, I'm still having issues here, but they have gotten much much stranger. I do suspect there is some underlying bug.

Sorry for long log here, but this issue does reproduce consistently:

root@calculon-2:/ros-z/ros-z-tests# cargo test --features interop-tests test_ros_z_server_ros2
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.13s
warning: the following packages contain code that will be rejected by a future version of Rust: num-bigint-dig v0.8.5
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
     Running tests/pubsub_interop.rs (/ros-z/target/debug/deps/pubsub_interop-b6d65e20fe20dd50)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

     Running tests/service_interop.rs (/ros-z/target/debug/deps/service_interop-9bb0dfce8786df17)

running 1 test
test test_ros_z_server_ros2_client ... FAILED

failures:

---- test_ros_z_server_ros2_client stdout ----
🚀 Starting shared rmw_zenohd daemon...
✅ rmw_zenohd daemon ready on tcp/127.0.0.1:7447
Started process: rmw_zenohd

=== Test: ros-z server <-> ROS2 client ===
Server ready for ROS2 client...
ROS2 output: waiting for service to become available...


thread 'test_ros_z_server_ros2_client' (32127) panicked at ros-z-tests/tests/service_interop.rs:138:5:
Expected sum: 17, got: waiting for service to become available...

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    test_ros_z_server_ros2_client

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 2 filtered out; finished in 17.66s

error: test failed, to rerun pass `--test service_interop`
root@calculon-2:/ros-z/ros-z-tests# cargo test --features interop-tests 
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.13s
warning: the following packages contain code that will be rejected by a future version of Rust: num-bigint-dig v0.8.5
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
     Running tests/pubsub_interop.rs (/ros-z/target/debug/deps/pubsub_interop-b6d65e20fe20dd50)

running 2 tests
test test_ros2_pub_to_ros_z_sub ... ok
test test_ros_z_pub_to_ros2_sub ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 11.94s

     Running tests/service_interop.rs (/ros-z/target/debug/deps/service_interop-9bb0dfce8786df17)

running 3 tests
[INFO] [1763569859.606700561] [add_two_ints_server]: Incoming request
a: 5 b: 3
test test_ros_z_server_ros_z_client ... ok
[INFO] [1763569862.610048752] [add_two_ints_server]: Incoming request
a: 15 b: 9
test test_ros2_server_ros_z_client ... ok
[INFO] [1763569867.809005966] [add_two_ints_server]: Incoming request
a: 10 b: 7
test test_ros_z_server_ros2_client ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 12.44s

root@calculon-2:/ros-z/ros-z-tests# cargo test --features interop-tests test_ros_z_server_ros2
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.13s
warning: the following packages contain code that will be rejected by a future version of Rust: num-bigint-dig v0.8.5
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
     Running tests/pubsub_interop.rs (/ros-z/target/debug/deps/pubsub_interop-b6d65e20fe20dd50)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

     Running tests/service_interop.rs (/ros-z/target/debug/deps/service_interop-9bb0dfce8786df17)

running 1 test
[INFO] [1763569886.637130484] [add_two_ints_server]: Incoming request
a: 10 b: 7
test test_ros_z_server_ros2_client ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 12.67s

root@calculon-2:/ros-z/ros-z-tests# 

What is happening:

• test_ros_z_server_ros2_client can never pass if run by itself in a clean environment
• test_ros_z_server_ros2_client will pass if the the test test_ros2_server_ros_z_client is run before it
• The more times I run test_ros2_server_ros_z_client the times I'll see "Incoming request" logged when running test_ros_z_server_ros2_client

Checking the output of ps -aux | grep add_two, I can see that test_ros2_server_ros_z_client is leaving copies of add_two_ints_server running, and the only reason that test_ros_z_server_ros2_client is passing is because it is finding the old servers from the previous test.