Configuring receiver -> then UBXReader stops reading after several succesful readings (F9P)

[djdan7]

Firstly I want to thank you for the effort you put into creating this library - it's AWESOME and you are GREAT! :D

My config:

u-blox C099-F9P // FWVER = HPG 1.13 // PROTVER = 27.12
navigation frequency: 10Hz
connection via USB cable (UART1)
windows 10 x64
python 3.7.9
pyubx2 ver. 1.0.2

My problem:

I want to start GNSS (via CFG-RST message), then read UBX messages for some time (in my case these are MON-RF, NAV-DOP, NAV-EELL, NAV-HPPOSLLH, NAV-PVT, NAV-SIG, NAV-STATUS) and stop GNSS (again via CFG-RST). The reason why I want to stop and start GNSS is that I want to save battery while keeping GNSS parameters in memory (so then I may perform hotstart).

How my code below works:

#receiver is stopped
-Starting GNSS is succesfull
-Reading several first message is also succesful, but then it hangs up and wait for message from serial (but the receiver is still working, and the serial send messages)
-I terminate the script
#start again (now the receiver keeps working - we are not stopped it)
-"Starting" GNSS is succesful - it just sends the message, but of course receiver is already wroking
-Reading all messages is succesfull
-Stopping GNSS is succesful
#again receiver is stopped
-Starting GNSS is succesfull
-Reading several first message is also succesful, but then it hangs up...
...

I tried to set validate flag to True - it raise

UBXStreamError(f"Unknown data header {byte1}. {nmeawarn}")
pyubx2.exceptions.UBXStreamError: Unknown data header 

What I noticed - after several OK readings it starts to divide binary message to single bytes and this is why this error occurs.

To prevent this I need to close port after writing, sleep for 1 second (shorter values don't help), open serial port again and flush it. This way everything works OK. But the problem is that in the future I would like to configure receiver on the run and then 1 second break looks ugly :/

Code:

import time
import serial
from pyubx2 import UBXReader, UBXMessage, SET

print("hello")

ser = serial.Serial(port="COM9", baudrate=921600, timeout=1)
print("Port is open? -> ", ser.is_open)

if ser.is_open:
    # START GNSS
    msg = UBXMessage(6, 4, SET, resetMode=9)
    print(msg)
    ser.write(msg.serialize())
    print("STARTED!")

    """
    # CLOSE, SLEEP and OPEN AGAIN
    ser.close()
    time.sleep(1)
    ser = serial.Serial(port="COM9", baudrate=921600, timeout=1)

    # PORT FLUSH
    print("dummy read", ser.read())
    ser.flush()
    ser.flushInput()
    ser.flushOutput()
    """

    # READING UBX
    i = 0
    while i < 300:
        print("reading ", i, ":")
        ubr = UBXReader(ser)
        (raw_data, parsed_data) = ubr.read()
        print(raw_data)
        print(parsed_data)
        if parsed_data.identity == 'NAV-HPPOSLLH':
            print("LON = ", (parsed_data.lon + parsed_data.lonHp/100)*10**(-7))
        print("\n")
        i += 1

    # STOP GNSS
    msg1 = UBXMessage(6, 4, SET, resetMode=8)
    print(msg1)
    msg1 = msg1.serialize()
    ser.write(msg1)
    print("STOPPED!")

    ser.close()

else:
    print("Problem with serial port")

[semuadmin]

Hi djdan7. Glad you're finding the library useful, but sorry to hear you're having problems.

I notice in your code that you're instantiating the UBXReader object repeatedly within the read loop (and thereby creating 300 separate instances). This only needs to be done once at the head of the loop, i.e.

# READING UBX
    i = 0
    ubr = UBXReader(ser)
    while i < 300:
        print("reading ", i, ":")
        (raw_data, parsed_data) = ubr.read()
        print(raw_data)
        print(parsed_data)
        if parsed_data.identity == 'NAV-HPPOSLLH':
            print("LON = ", (parsed_data.lon + parsed_data.lonHp/100)*10**(-7))
        print("\n")
        i += 1

Suggest you try this change first and see if it resolves your problem.

You could if you prefer also use the UBXReader iterator with a millisecond timer, rather than a fixed number of iterations. e.g. something like this (pseudocode):

duration = 1 minute
start = now()
stream = Serial(port, baud, timeout=5)
ubr = UBXReader(stream)
for (raw, parsed) in ubr:
   if rawformat:
      print(raw)
   else:
      print(parsed)
   if now() - start > duration
      break

Hope this helps.

[djdan7]

Thank you for pointing out that I've wrong instantiated the UBXReader! It didn't solve the problem, but the code is now better :D

I will also answer to your previous suggestions:

1. I've tried to set lower baud rate - unfortunately I'm limited to 406800, because at lower values port can't keep up. But even when I configure receiver to send only one message (NAV-HPPOSLLH) and set baudrate to 38400 - problem still occurs.
2. When I just read messages from port - everything works excellent even on 921600 baud rate.
3. On MON-COMMS and MON-TXBUF all pendings = 0.
4. On u-blox forum they suggested me to turn on/off receiver that way. In u-center there is panel described as "Control GNSS" and in ZED-F9P interface decription it is stated as "Controlled GNSS stop/start"
   

CFG-RST:

BUT! :D

What I've noticed:

1. After I send message to start GNSS it replies with some "NMEA" like message started with $G, that the receiver is started. And I think that this message causes problem, which carries over to the rest of the messages. I only don't know how and why UBXReader is able to read several messages OK until it hangs up. Maybe here is some possibility that UBXReader would filter out such messages?

2. When I configure receiver to send also NMEA message (i.e. GNGGA) it solves problem! :D

Tommorow, or on Monday I will also try to send RTCM messages (to get RTK, centimeter level position) to the receiver every 1 second. I will let you know if there will be any problems then.

[semuadmin]

The UBXReader.read() method shouldn't have a problem with non-UBX (e.g. NMEA) data - provided the validate flag is set to the default 'False', it does effectively 'filter them out'.

I don't have an F9P device to hand but I've run your modified code through a standard-precision device (using NAV-POSLLH rather than NAV-HPPOSLLH) at 10Hz and 921600 baud (nominal) and it seems to run fine the first time, but glitches after a few seconds on any subsequent run. If, however, I explicitly set the navBbrMask in CFG-RST to b'\x0000' (rather than 'b\x00\x00', which is what UBXMessage.read() defaults it to) the problem goes away and I can run the code repeatedly without difficulty.

Suggest trying the following in the relevant sections in your code and see if it improves matters:
START GNSS - msg = UBXMessage(6, 4, SET, navBbrMask=b'\x0000', resetMode=9)
STOP GNSS - msg = UBXMessage(6, 4, SET, navBbrMask=b'\x0000', resetMode=8)

If this works consistently, I may need to make a minor tweak to ubxmessage.py.

This is the modified code that seems to work OK for me:

import time
import serial
from pyubx2 import UBXReader, UBXMessage, SET

PORT = 'COM13'
BAUD = 921600
TIMEOUT = 1
ITERATIONS = 10


print("Hello")

ser = serial.Serial(port=PORT, baudrate=BAUD, timeout=TIMEOUT)
print(f"Serial Port is open? -> {ser.is_open}")

if ser.is_open:
    x = 0
    while x < ITERATIONS:

        print(f"\n\nITERATION {x}\n\n")
        # START GNSS
        print("Sending CFG-RST GNSS Start!")
        msg = UBXMessage(6, 4, SET, navBbrMask=b'\x0000', resetMode=9)
        print(msg)
        ser.write(msg.serialize())
        print("NSS Started!")
    
        # READING UBX
        i = 0
        ubr = UBXReader(ser)
        while i < 300:
            print(f"Reading {i} :")
            (raw_data, parsed_data) = ubr.read()
            print(raw_data)
            print(parsed_data)
            if parsed_data.identity == 'NAV-POSLLH':
                print(f"\n\nLAT = {parsed_data.lat/10**7}, LON = {parsed_data.lon/10**7}")
            print("\n")
            i += 1
    
        # STOP GNSS
        print("Sending CFG-RST GNSS Stop!")
        msg1 = UBXMessage(6, 4, SET, navBbrMask=b'\x0000', resetMode=8)
        print(msg1)
        msg1 = msg1.serialize()
        ser.write(msg1)
        print("GNSS Stopped!")


        x += 1

    ser.close()
    print("Serial Port Closed!")

else:
    print("Problem with serial port")

[djdan7]

If I set navBbrMask it does not start nor stop receiver :/

PS. For what you do u-blox should send you a C099-F9P for free. Have you tried to contact them?

[semuadmin]

Well navBbrMask has to be set in code one way or another - it's a mandatory payload field. It's either defaulting to b'\x00\x00' (which is what UBXMessage defaults unspecified 2-byte bitmasks to), or you set it explicitly. Not sure why setting it explicitly would prevent the stop/start from working? According to the F9P data sheet, 'b\x0000' is the appropriate setting for 'hot start':

And, yes, I did try blagging some free devkit from u-blox but they weren't having it :-(

[djdan7]

I've set navBbrMask to b'\x0000' - it didn't work, so then I've copied all code, which you provided (and changed com port) - it also didn't work. What I mean saying didn't work:

1. If receiver was running (i.e. I started receiver via u-center) and then I run script - it read messages ok, but at the end it didn't stoped it
2. If receiver was stopped - it didn't start it, read 2 MON-RF messages (receiver send them no matter if it is stopped or not) and throws error:

if parsed_data.identity == 'NAV-POSLLH':
AttributeError: 'NoneType' object has no attribute 'identity'

I think that 'Controlled GNSS stop or start' =/= Hot start. When I stop GNSS for couple of minutes - it performs hot start, but when I stop it for i.e. 1 hour - it performs warm start, etc. So it not always performs hot start

What a shame for the u-blox. A decent reading library for ubx (like yours) in Python makes people from non-IT fields (like me) interested in their product. I think Python is the only simple language for people outside the IT industry. I ordered it about 3 months ago and the package arrived after a week.

[semuadmin]

OK leave it with me for now - I'll need to run some comparative tests against u-center to see exactly how u-center is populating the navBbrMask field in the CFG-RST GNSS stop/start message.

[djdan7]

Ok, thank you so much for your effort!

[semuadmin]

OK so I've just run a comparative test against u-center and this is the binary output from u-center showing a CFG-RST stop (highlighted in yellow) followed by a CFG-RST start (highlighted in green), followed by a resumption in NAV messages:

In both cases, the navBbrMask field (bytes 6 & 7 of the message; bytes 1 & 2 of the payload) is set to hex 00 00 (which is exactly what pyubx2 should be setting it to by default). In both cases, the UBX CFG-RST message is 'acknowledged' by one or more NMEA TXT messages (a confirmation and - after a stop - an antenna status), as you've observed.

So I need to understand why we appear to be getting different results from the device when sending apparently identical messages from u-Center and pyubx2....leave it with me.

[semuadmin]

OK I believe this is sorted now with a fix to UBXReader which I've just uploaded in v1.0.3. Basically, you called it in your original issue report but I managed to lead us down a wild goose chase with CFG-RST message formats - doh! The CFG-RST message handling was fine - the problem was in the parsing of the subsequent NMEA TXT responses. Surprised this hasn't cropped up before.

I've rerun my modified version of your test code (with navBbrMask left at the default b'\x00\x00') and it seems to be working fine now. GNSS is successfully stopped at the end and I'm seeing no further UBX messages until I explicitly start it again. Have a go yourself and let me know if it works for you.

[djdan7]

Thank you!
After updating to v1.0.3 everything works GREAT!!! :D

If I can spam a little more here, I'd have a question. What is the appropriate method to also read the NMEA message once every 10 seconds? I need to read a GNGGA message, so then I can send it to the NTRIP client and download the appropriate RTCM corrections. At this point, I wrote something like this, but I predict that it could be done a little better:

import datetime

import pyubx2.exceptions as ube
import serial
from pyubx2 import UBXReader, UBXMessage, SET


# START GNSS
def start_gnss(ser):
    if ser.is_open:
        msg = UBXMessage(6, 4, SET, resetMode=9)
        ser.write(msg.serialize())
        print(msg)
        print(datetime.datetime.now())
        print("STARTED!")
    else:
        print("PORT CLOSED")


# READING UBX
def read_ubx(ser, ubr):
    if ser.is_open:
        i = 0
        while i < 1:
            try:
                (raw_data, parsed_data) = ubr.read()
                print(raw_data)
                print(parsed_data)
                print(datetime.datetime.now())
                if parsed_data.identity == 'NAV-HPPOSLLH':
                    print("LON = ", (parsed_data.lon + parsed_data.lonHp / 100) * 10 ** (-7))
                    print("LAT = ", (parsed_data.lat + parsed_data.latHp / 100) * 10 ** (-7))
                print("\n")
                i += 1
            except ube.UBXStreamError:
                try:
                    print("EXCEPT NMEA")
                    gga = "$" + str((ser.readline()).decode())
                    print(gga)
                except:
                    print("EXCEPT NMEA EXCEPT")
                    ubr = UBXReader(ser, True)
                    ser.flushInput()
            except:
                print("EXCEPT")
                ubr = UBXReader(ser, True)
                ser.flushInput()
    else:
        print("PORT CLOSED")


# STOP GNSS
def stop_gnss(ser):
    if ser.is_open:
        msg = UBXMessage(6, 4, SET, resetMode=8)
        ser.write(msg.serialize())
        print(msg)
        print(datetime.datetime.now())
        print("STOPPED!")
    else:
        print("PORT CLOSED")


print("Hello")

ser = serial.Serial(port="COM9", baudrate=921600, timeout=1)
print("Port is open? -> ", ser.is_open)

start_gnss(ser)

ubr = UBXReader(ser, True)
for i in range(300):
    print(i)
    read_ubx(ser, ubr)

stop_gnss(ser)

ser.close()

It works fine (I have everything I need), but it read NMEA messages at 10Hz unnecessarily. Would it be better to send a message every 10 seconds to configure the receiver to send GNGGA, then read this message and send the configuration message again so that it stops sending GNGGA? Do I need to use "UBX-CFG-VALSET"? I'm not 100% sure how to do it exactly. Thank you in advance! :D