Inconsistency in running multiple missions

[Turokhunter]

Hi, I'm a researcher working on drone data, and I'm trying to make the process of collecting simulation data easier. In that pursuit, I made a python script that executes a series of missions. The architecture that I use is similar to PX4 testing, where the PX4 instance is launched, followed by a mission. However, I noticed that there are times when all 15 missions work fine, and there are times when one of the missions gets hanged upon a connect. I was wondering if there is a way to detect when this happens so that I can either try to reconnect to the vehicle. I'm using PX4, Gazebo, and MAVSDK-Python.

[julianoes]

Which version are you using? I recently found the bug leading to the connection issue: mavlink/MAVSDK#1739

[Turokhunter]

mavsdk 0.23.0
at least that is what the pip3 list claims.

[julianoes]

That would explain it. Try with the latest and greatest :).

[Turokhunter]

I have uninstalled and reinstalled mavsdk 1.3.0.
I ran my scripts six times, four times it runs all 15 tests. Once it got hung up on the connecting vehicle and once on starting mission (so at some point during the mission).


)
Is there a way to detect these occurrences? So I can stop the run and restart in code? Maybe a timeout feature for connecting to a vehicle.

I can provide the code if that helps.

[julianoes]

You can always do workarounds like you said but I would be more interested to fix the actual problems.

The first thing I would do is to run mavsdk_server separately, like described here:
mavsdk-python-docs.s3-website.eu-central-1.amazonaws.com/#debug-connection-issues

Then you can share the output of when it happens, or you could even run it with gdb --args path/to/mavsdk_server ..., and when it stalls, you hit Ctrl+C and in gdb you do: thread apply all backtrace and copy the output to here.

[Turokhunter]

I will try to get you an output in the next two days.

[Turokhunter]

Hey, sorry for the delay been sick with C. I ran what you asked me to run.
First, there is the mission not starting issue.

This seems connected to the script running too quickly before everything is set up in px4. I found that adding a 10-sec delay gave enough time for missions to avoid this problem. It would be great if MavSDK could detect that everything isn't ready and only start when it is.

As for the connect error, I had recorded images of when it occurred.


There doesn't seem to be any indication on the left. Do you want me to run a different debugging measure?

[julianoes]

It would be great if MavSDK could detect that everything isn't ready and only start when it is.

This should be improved with mavlink/MAVSDK#1794.

For the connection issue, I would have to see in gdb what is going on as mentioned above:

Then you can share the output of when it happens, or you could even run it with gdb --args path/to/mavsdk_server ..., and when it stalls, you hit Ctrl+C and in gdb you do: thread apply all backtrace and copy the output to here.

Alternatively, you can provide all steps/scripts to me so that I can reproduce it.

[julianoes]

I managed to reproduce something similar where it doesn't connect. I'm investigating. If you could provide details how to reproduce your setup that would still be helpful though.

[Turokhunter]

I couldn't get the gdb to work. Currently, I'm working on documenting and making my git public. It contains all the code I have been working on. I will post again once I'm done.

[Turokhunter]

Here is the GitHub with all my setup: https://github.com/VisDunneRight/Px4-Batch-Simulation

[julianoes]

Thanks I will check it out, hopefully next week.

[julianoes]

I was able to run this and for me it fails to arm, mostly in the second test:

--> Test case 2 of 15: 'Mission 2: Square flight plan' running ...
Connecting to vehicle
Drone discovered!
home location
	>lat:47.3977508
	>lon:8.5456074
-- Uploading mission
Waiting for drone to have a global position estimate...
-- Global position estimate OK
-- Arming
Traceback (most recent call last):
  File "/home/julianoes/src/upstream/Px4-Batch-Simulation/missions/sdk_square_mission.py", line 114, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/julianoes/src/upstream/Px4-Batch-Simulation/missions/sdk_square_mission.py", line 103, in main
    await mission.arm()
  File "/home/julianoes/src/upstream/Px4-Batch-Simulation/missions/missionHelperSDK.py", line 48, in arm
    await self.vehicle.action.arm()
  File "/home/julianoes/.local/lib/python3.9/site-packages/mavsdk/action.py", line 320, in arm
    raise ActionError(result, "arm()")
mavsdk.action.ActionError: COMMAND_DENIED: 'Command Denied'; origin: arm(); params: ()

[10:05:23|Debug] Discovered 1 component(s) (system_impl.cpp:499)
[10:05:23|Warn ] command denied (512). (mavlink_command_sender.cpp:193)
[10:05:24|Debug] Falling back to gimbal protocol v1 (mission_impl.cpp:124)
[10:05:24|Debug] MAVLink: info: [logger] ./log/2022-06-20/08_05_21.ulg	 (system_impl.cpp:248)
[10:05:28|Debug] MAVLink: critical: Arming denied! manual control lost (system_impl.cpp:248)

Have you seen that too?

[julianoes]

@Turokhunter you still there?