Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mavros dies and kills jetson TX1 also #645

Closed
skysingh opened this issue Jan 12, 2017 · 6 comments
Closed

Mavros dies and kills jetson TX1 also #645

skysingh opened this issue Jan 12, 2017 · 6 comments
Labels

Comments

@skysingh
Copy link

skysingh commented Jan 12, 2017

This is only bug and feature tracker, please use it
to report bugs or request features.


Issue details

I have been running mavros on ubuntu laptop 14.04 indigo fine but I am moving to jetson TX1 running 16.04 kinetic. I am connected over UART1 (ttyS0)
Please describe the problem, or desired feature
I can see traffic on ttyS0 using cat /dev/ttyS0 so I know the hardware is working. when I launch px4.launch I get the output shown below but within a few seconds the system dies and the TX1 is completety unresponsive. Need to disconnect the pixhawk and then restart TX1 !

MAVROS version and platform

Mavros: ?0.18.4?
ROS: Kinetic - installed using procdedure by Jetson hacks. ROS works ok
Ubuntu: ?16.04?

kinetic 16.06 tegra

Autopilot type and version

[ ] ArduPilot
[X] PX4

Version: ?3.7.1?

Node logs

NODES
  /
    mavros (mavros/mavros_node)
    mavros_node (rosservice/rosservice)

auto-starting new master
process[master]: started with pid [3146]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to fb84298a-d905-11e6-8ee2-00044b65c8de
process[rosout-1]: started with pid [3160]
started core service [/rosout]
process[mavros-2]: started with pid [3178]
process[mavros_node-3]: started with pid [3179]
[ INFO] [1484253041.516252133]: FCU URL: /dev/ttyS0:921600
[ WARN] [1484253041.517881599]: init: message from ASLUAV, MSG-ID 201 ignored! Table has different entry.
[ WARN] [1484253041.518318092]: init: message from autoquad, MSG-ID 150 ignored! Table has different entry.
[ WARN] [1484253041.518368923]: init: message from autoquad, MSG-ID 152 ignored! Table has different entry.
[ WARN] [1484253041.518770886]: init: message from matrixpilot, MSG-ID 150 ignored! Table has different entry.
[ WARN] [1484253041.518815260]: init: message from matrixpilot, MSG-ID 151 ignored! Table has different entry.
[ WARN] [1484253041.518853852]: init: message from matrixpilot, MSG-ID 152 ignored! Table has different entry.
[ WARN] [1484253041.518900933]: init: message from matrixpilot, MSG-ID 153 ignored! Table has different entry.
[ WARN] [1484253041.518935776]: init: message from matrixpilot, MSG-ID 155 ignored! Table has different entry.
[ WARN] [1484253041.518968899]: init: message from matrixpilot, MSG-ID 156 ignored! Table has different entry.
[ WARN] [1484253041.519009471]: init: message from matrixpilot, MSG-ID 157 ignored! Table has different entry.
[ WARN] [1484253041.519085301]: init: message from matrixpilot, MSG-ID 158 ignored! Table has different entry.
[ WARN] [1484253041.519127695]: init: message from matrixpilot, MSG-ID 170 ignored! Table has different entry.
[ WARN] [1484253041.519189203]: init: message from matrixpilot, MSG-ID 171 ignored! Table has different entry.
[ WARN] [1484253041.519222639]: init: message from matrixpilot, MSG-ID 172 ignored! Table has different entry.
[ WARN] [1484253041.519252638]: init: message from matrixpilot, MSG-ID 173 ignored! Table has different entry.
[ WARN] [1484253041.519282897]: init: message from matrixpilot, MSG-ID 174 ignored! Table has different entry.
[ WARN] [1484253041.519313104]: init: message from matrixpilot, MSG-ID 175 ignored! Table has different entry.
[ WARN] [1484253041.519342843]: init: message from matrixpilot, MSG-ID 176 ignored! Table has different entry.
[ WARN] [1484253041.519373675]: init: message from matrixpilot, MSG-ID 177 ignored! Table has different entry.
[ WARN] [1484253041.519402997]: init: message from matrixpilot, MSG-ID 178 ignored! Table has different entry.
[ WARN] [1484253041.519467734]: init: message from matrixpilot, MSG-ID 179 ignored! Table has different entry.
[ WARN] [1484253041.519505024]: init: message from matrixpilot, MSG-ID 180 ignored! Table has different entry.
[ WARN] [1484253041.519534866]: init: message from matrixpilot, MSG-ID 181 ignored! Table has different entry.
[ WARN] [1484253041.519564605]: init: message from matrixpilot, MSG-ID 182 ignored! Table has different entry.
[ WARN] [1484253041.519592833]: init: message from matrixpilot, MSG-ID 183 ignored! Table has different entry.
[ WARN] [1484253041.519621425]: init: message from matrixpilot, MSG-ID 184 ignored! Table has different entry.
[ WARN] [1484253041.519649862]: init: message from matrixpilot, MSG-ID 185 ignored! Table has different entry.
[ WARN] [1484253041.519678246]: init: message from matrixpilot, MSG-ID 186 ignored! Table has different entry.
[ WARN] [1484253041.578291466]: init: message from paparazzi, MSG-ID 180 ignored! Table has different entry.
[ WARN] [1484253041.578389223]: init: message from paparazzi, MSG-ID 181 ignored! Table has different entry.
[ WARN] [1484253041.578564789]: init: message from paparazzi, MSG-ID 182 ignored! Table has different entry.
[ WARN] [1484253041.578629109]: init: message from paparazzi, MSG-ID 183 ignored! Table has different entry.
[ WARN] [1484253041.578735407]: init: message from paparazzi, MSG-ID 184 ignored! Table has different entry.
[ WARN] [1484253041.579210596]: init: message from slugs, MSG-ID 170 ignored! Table has different entry.
[ WARN] [1484253041.579275750]: init: message from slugs, MSG-ID 172 ignored! Table has different entry.
[ WARN] [1484253041.579329445]: init: message from slugs, MSG-ID 173 ignored! Table has different entry.
[ WARN] [1484253041.579782813]: init: message from slugs, MSG-ID 176 ignored! Table has different entry.
[ WARN] [1484253041.579852029]: init: message from slugs, MSG-ID 177 ignored! Table has different entry.
[ WARN] [1484253041.579954264]: init: message from slugs, MSG-ID 179 ignored! Table has different entry.
[ WARN] [1484253041.580062437]: init: message from slugs, MSG-ID 180 ignored! Table has different entry.
[ WARN] [1484253041.580164620]: init: message from slugs, MSG-ID 181 ignored! Table has different entry.
[ WARN] [1484253041.580272064]: init: message from slugs, MSG-ID 184 ignored! Table has different entry.
[ WARN] [1484253041.580399246]: init: message from slugs, MSG-ID 185 ignored! Table has different entry.
[ WARN] [1484253041.580499555]: init: message from slugs, MSG-ID 186 ignored! Table has different entry.
[ WARN] [1484253041.580605280]: init: message from slugs, MSG-ID 188 ignored! Table has different entry.
[ WARN] [1484253041.580897716]: init: message from slugs, MSG-ID 191 ignored! Table has different entry.
[ WARN] [1484253041.580957140]: init: message from slugs, MSG-ID 192 ignored! Table has different entry.
[ WARN] [1484253041.581127863]: init: message from slugs, MSG-ID 193 ignored! Table has different entry.
[ WARN] [1484253041.581181142]: init: message from slugs, MSG-ID 194 ignored! Table has different entry.
[ INFO] [1484253041.582359688]: serial0: device: /dev/ttyS0 @ 921600 bps
[ INFO] [1484253041.582879303]: GCS bridge disabled
[ INFO] [1484253041.681673879]: Plugin 3dr_radio loaded
[ INFO] [1484253041.687380577]: Plugin 3dr_radio initialized
[ INFO] [1484253041.687601766]: Plugin actuator_control loaded
[ INFO] [1484253041.696113300]: Plugin actuator_control initialized
[ INFO] [1484253041.696323239]: Plugin altitude loaded
[ INFO] [1484253041.699726540]: Plugin altitude initialized
[ INFO] [1484253041.699948771]: Plugin command loaded
[ INFO] [1484253041.714858797]: Plugin command initialized
[ INFO] [1484253041.715113058]: Plugin ftp loaded
[ INFO] [1484253041.735580049]: Plugin ftp initialized
[ INFO] [1484253041.735809519]: Plugin global_position loaded
[ INFO] [1484253041.759760797]: Plugin global_position initialized
[ INFO] [1484253041.759979330]: Plugin hil_actuator_controls loaded
[ INFO] [1484253041.762463137]: Plugin hil_actuator_controls initialized
[ INFO] [1484253041.762670889]: Plugin hil_controls loaded
[ INFO] [1484253041.764702995]: Plugin hil_controls initialized
[ INFO] [1484253041.764898820]: Plugin imu_pub loaded
[ INFO] [1484253041.788059870]: Plugin imu_pub initialized
[ INFO] [1484253041.788277569]: Plugin local_position loaded
[ INFO] [1484253041.807672832]: Plugin local_position initialized
[ INFO] [1484253041.807888084]: Plugin manual_control loaded
[ INFO] [1484253041.810224553]: Plugin manual_control initialized
[ INFO] [1484253041.810439336]: Plugin param loaded
[ INFO] [1484253041.818698848]: Plugin param initialized
[ INFO] [1484253041.818927276]: Plugin rc_io loaded
[ INFO] [1484253041.831573331]: Plugin rc_io initialized
[ INFO] [1484253041.831657234]: Plugin safety_area blacklisted
[ INFO] [1484253041.831881756]: Plugin setpoint_accel loaded
[ INFO] [1484253041.844050018]: Plugin setpoint_accel initialized
[ INFO] [1484253041.844282561]: Plugin setpoint_attitude loaded
[ INFO] [1484253041.883539475]: Plugin setpoint_attitude initialized
[ INFO] [1484253041.883777955]: Plugin setpoint_position loaded
[ INFO] [1484253041.902617355]: Plugin setpoint_position initialized
[ INFO] [1484253041.902854220]: Plugin setpoint_raw loaded
[ INFO] [1484253041.934850540]: Plugin setpoint_raw initialized
[ INFO] [1484253041.935085322]: Plugin setpoint_velocity loaded
[ INFO] [1484253041.945261737]: Plugin setpoint_velocity initialized
[ INFO] [1484253041.945635576]: Plugin sys_status loaded
[ INFO] [1484253041.964608565]: Plugin sys_status initialized
[ INFO] [1484253041.964852409]: Plugin sys_time loaded
[ INFO] [1484253041.975734056]: Plugin sys_time initialized
[ INFO] [1484253041.975956911]: Plugin vfr_hud loaded
[ INFO] [1484253041.979996853]: Plugin vfr_hud initialized
[ INFO] [1484253041.980227886]: Plugin waypoint loaded
[ INFO] [1484253041.991972987]: Plugin waypoint initialized
[ INFO] [1484253041.992075275]: Autostarting mavlink via USB on PX4
[ INFO] [1484253041.992150115]: Built-in SIMD instructions: ARM NEON
[ INFO] [1484253041.992200582]: Built-in MAVLink package version: 2016.12.12
[ INFO] [1484253041.992259694]: Known MAVLink dialects: common ardupilotmega ASLUAV autoquad matrixpilot paparazzi slugs standard uAvionix ualberta
[ INFO] [1484253041.992307505]: MAVROS started. MY ID 1.240, TARGET ID 1.1
[mavros_node-3] process has finished cleanly
log file: /home/ubuntu/.ros/log/fb84298a-d905-11e6-8ee2-00044b65c8de/mavros_node-3*.log
[master] process has died [pid 3146, exit code -9, cmd rosmaster --core -p 11311 -w 3 __log:=/home/ubuntu/.ros/log/fb84298a-d905-11e6-8ee2-00044b65c8de/master.log].
log file: /home/ubuntu/.ros/log/fb84298a-d905-11e6-8ee2-00044b65c8de/master*.log
================================================================================REQUIRED process [mavros-2] has died!
process has died [pid 3178, exit code -9, cmd /opt/ros/kinetic/lib/mavros/mavros_node __name:=mavros __log:=/home/ubuntu/.ros/log/fb84298a-d905-11e6-8ee2-00044b65c8de/mavros-2.log].
log file: /home/ubuntu/.ros/log/fb84298a-d905-11e6-8ee2-00044b65c8de/mavros-2*.log
Initiating shutdown!
================================================================================
[mavros-2] killing on exit
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

Diagnostics

place here result of:
rostopic echo -n1 /diagnostics

cannot run this is uart connected to pixhawk !

Check ID

rosrun mavros checkid

gives no response. need to ctrl-C

@skysingh
Copy link
Author

Following up on my own question. I changed the TX1 to use UART1 instead of UART0. This shows up in 16.04 L4T as /dev/ttyTHS2. I can see traffic from the pixhawk using cat. Now if run px4.launch with ttyTHS2 as the _fcu_url mavros starts up and communicates with the pixhawk. So some progress. Maybe the TX1 only wants console on UART0. HOWEVER, I cannot keep mavros running - gives all kinds of timeouts:
[mavros_node-3] process has finished cleanly
log file: /home/ubuntu/.ros/log/ef0866ae-d90e-11e6-9afb-00044b65c8de/mavros_node-3*.log
[ INFO] [1484256982.599069436]: CON: Got HEARTBEAT, connected. FCU: ArduPilotMega / ArduCopter
[ INFO] [1484256982.605230523]: FCU: Initialising APM
[ INFO] [1484256982.607989299]: FCU: Calibrating barometer
[ WARN] [1484256984.641218281]: VER: broadcast request timeout, retries left 4
[ WARN] [1484256985.621655956]: VER: broadcast request timeout, retries left 3
[ WARN] [1484256991.628238323]: VER: unicast request timeout, retries left 2
[ WARN] [1484256992.714774777]: CON: Lost connection, HEARTBEAT timed out.
[ WARN] [1484256993.605895284]: PR: request list timeout, retries left 2
[ WARN] [1484256994.613373326]: PR: request list timeout, retries left 1
[ WARN] [1484256995.617775147]: PR: request list timeout, retries left 0

If i try ubuntu@tegra-ubuntu:~$ rostopic echo mavros/state

I get just this one response then hangs...

header:
seq: 3
stamp:
secs: 1484256996
nsecs: 675819790
frame_id: ''
connected: False
armed: False
guided: False
mode: ''

---- diagnostics -->

ubuntu@tegra-ubuntu:~$ rostopic echo -n1 /diagnostics
header:
seq: 262
stamp:
secs: 1484257236
nsecs: 494919849
frame_id: ''
status:

level: 1
name: mavros: FCU connection
message: not connected
hardware_id: /dev/ttyTHS2:921600
values: 
  - 
    key: Received packets:
    value: 10
  - 
    key: Dropped packets:
    value: 0
  - 
    key: Buffer overruns:
    value: 0
  - 
    key: Parse errors:
    value: 0
  - 
    key: Rx sequence number:
    value: 8
  - 
    key: Tx sequence number:
    value: 119
  - 
    key: Rx total bytes:
    value: 635
  - 
    key: Tx total bytes:
    value: 97358
  - 
    key: Rx speed:
    value: 0.000000
  - 
    key: Tx speed:
    value: 277.000000
  • level: 2
    name: mavros: GPS
    message: No satellites
    hardware_id: /dev/ttyTHS2:921600
    values:

    key: Satellites visible
    value: 0
    
    • key: Fix type
      value: 0
    • key: EPH (m)
      value: Unknown
    • key: EPV (m)
      value: Unknown
  • level: 2
    name: mavros: Heartbeat
    message: No events recorded.
    hardware_id: /dev/ttyTHS2:921600
    values:

    key: Heartbeats since startup
    value: 2
    
    • key: Frequency (Hz)
      value: 0.000000
    • key: Vehicle type
      value: Ground rover
    • key: Autopilot type
      value: ArduPilotMega / ArduCopter
    • key: Mode
      value: INITIALISING
    • key: System status
      value: Calibrating
  • level: 2
    name: mavros: System
    message: Sensor helth
    hardware_id: /dev/ttyTHS2:921600
    values:

    key: Sensor present
    value: 0x0020DC07
    
    • key: Sensor enabled
      value: 0x00208007
    • key: Sensor helth
      value: 0x0020DC04
    • key: 3D gyro
      value: Fail
    • key: 3D accelerometer
      value: Fail
    • key: 3D magnetometer
      value: Ok
    • key: motor outputs / control
      value: Ok
    • key: AHRS subsystem health
      value: Ok
    • key: CPU Load (%)
      value: 0.0
    • key: Drop rate (%)
      value: 0.0
    • key: Errors comm
      value: 0
    • key: Errors count mavconn library. #1
      value: 0
    • key: Errors count Pub/sub for Mavlink.msg #2
      value: 0
    • key: Errors count Plugin loading functionality. #3
      value: 0
    • key: Errors count Diagnostic mesages. #4
      value: 0
  • level: 1
    name: mavros: Battery
    message: Low voltage
    hardware_id: /dev/ttyTHS2:921600
    values:

    key: Voltage
    value: 0.00
    
    • key: Current
      value: -0.0
    • key: Remaining
      value: -1.0
  • level: 2
    name: mavros: Time Sync
    message: No events recorded.
    hardware_id: /dev/ttyTHS2:921600
    values:

    key: Timesyncs since startup
    value: 0
    
    • key: Frequency (Hz)
      value: 0.000000
    • key: Last dt (ms)
      value: 0.000000
    • key: Mean dt (ms)
      value: 0.000000
    • key: Last system time (s)
      value: 0.000000000
    • key: Time offset (s)
      value: 0.000000000

ubuntu@tegra-ubuntu:~$

@mhkabir
Copy link
Member

mhkabir commented Jan 13, 2017

It's is a bug in the Tegra serial driver. I have been able to reproduce this forever.

@mhkabir
Copy link
Member

mhkabir commented Jan 13, 2017

But as far as Nvidia told me, the 16.04 release shouldn't have it.

Did you disable the system serial console on UART0? Because it will cause the same issues as you're seeing unless the console is completely disabled. I don't remember the exact procedure, but you need to edit a few files to disable console output on UART0.

I would suggest using UART1 however.

@vooon
Copy link
Member

vooon commented Jan 13, 2017

Also maybe better to use USB-UART since nvidia does crap again...

Look at dmesg, try to run under strace, try lower baud rates (e.g. 57600).

To run under strace you must start roscore, then you may load configuration via rosparam load /mavros ~/ws/src/mavros/mavros/launch/px4_config.yaml
Then you been able to run strace -fy ~/ws/devel/lib/mavros/mavros_node _fcu_url:=/dev/ttyTHS1

@vooon vooon added the question label Jan 13, 2017
@skysingh
Copy link
Author

Good news. mavros works with JetsonTX1 running ubuntu 16.04 L4T! Problem with earlier test using uart1 was a bad regulator so pixhawk was resetting on brownout. BTW I am using the connect-tech orbitty carrier board on the TX1. It is compact and excellent.

So for any one connecting pixhawk via mavros to Jetson TX1:

  1. use UART1. Connect tele2 to UART1 rx and TX. Make sure baud rate is set in pixhawk. Check that you get pixhawk traffic using cat /dev/ttyTHS2. Under ubuntu L4T uart1 shows up as THS2
  2. use the standard px4.launch file. set fcu url = /dev/ttyTHS2:921600. Then roslaunch px4.launch
  3. start rosservice (can put inside launch file):
4. node should start up with good messages 5. in another terminal try rostopic echo mavros/state. You should get good messages. Then you are all set.

Hope this helps someone. Thanks to mhkabir and vooon

@vooon vooon closed this as completed Jan 13, 2017
@sunnye26
Copy link

How did you guys connect the Jetson TX1 to the pixhawk? Do you have a picture of the wiring done?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants