STM32L4 - using TICKLESS and serial (UART - AT cellular) losing characters

[mfatiga]

Description

I seem to be experiencing loss of characters sent/received through UART when communicating with the BG96 modem and when I have TICKLESS mode enabled. I have noticed this by viewing traces when connecting to a network. When I disable TICKLESS mode, the modem connects and works perfectly.

I am using a custom target based on STM32L476VG and I'm using the GCC_ARM toolchain. I have copied the DISCO_L476VG system_clock.c and am using mostly the same peripheral pins and pin names files. I am using the QUECTEL BG96 modem and I'm using the integrated mbed driver. I have connected the modem to uart pins and I have also defined and connected flow control pins. The baud rate is set to default at 115200. Modem pins:

MDMRTS = PA_12,
MDMCTS = PA_11,
MDMTXD = PA_9, 
MDMRXD = PA_10,

custom_targets.json

{
    "MY_TARGET": {
        "inherits": ["FAMILY_STM32"],
        "core": "Cortex-M4F",
        "extra_labels_add": ["STM32L4", "STM32L476xG", "STM32L476VG"],
        "config": {
            "clock_source": {
                "help": "Mask value : USE_PLL_HSE_EXTC (need HW patch) | USE_PLL_HSE_XTAL | USE_PLL_HSI | USE_PLL_MSI",
                "value": "USE_PLL_MSI",
                "macro_name": "CLOCK_SOURCE"
            },
            "lpticker_lptim": {
                "help": "This target supports LPTIM. Set value 1 to use LPTIM for LPTICKER, or 0 to use RTC wakeup timer",
                "value": 1
            }
        },
        "OUTPUT_EXT": "hex",
        "detect_code": ["0820"],
        "macros_add": [
            "USBHOST_OTHER", 
            "TWO_RAM_REGIONS",
            "MBED_TICKLESS"
        ],
        "device_has_add": [
            "ANALOGOUT", 
            "CAN", 
            "LPTICKER",
            "SERIAL_FC",
            "SERIAL_ASYNCH",
            "TRNG",
            "FLASH"
        ],
        "release_versions": ["2", "5"],
        "device_name": "STM32L476VG",
        "bootloader_supported": true
    }
}

mbed_app.json

{
    "target_overrides": {
        "*": {
            "events.use-lowpower-timer-ticker": 1,
            "platform.default-serial-baud-rate": 115200,
            
            "platform.stdio-baud-rate": 115200,
            "platform.stdio-convert-newlines": true,
            "platform.stdio-buffered-serial": true,
            "drivers.uart-serial-rxbuf-size": 256,
            "drivers.uart-serial-txbuf-size": 256,

            "rtos.idle-thread-stack-size": 1024,

            "platform.cpu-stats-enabled": 1,
            "platform.thread-stats-enabled": 1,
            "platform.error-filename-capture-enabled": true,
            
            "cellular.use-apn-lookup": 0,
            "ppp-cell-iface.apn-lookup": 0,

            "mbed-trace.enable": true
        }
    },
    "macros": [
        "DEFAULT_APN=\"internet\"",
        "CELLULAR_DEVICE=QUECTEL_BG96"
    ],
    "config": {
        "trace-level": {
            "help": "Options are TRACE_LEVEL_ERROR,TRACE_LEVEL_WARN,TRACE_LEVEL_INFO,TRACE_LEVEL_DEBUG",
            "macro_name": "MBED_TRACE_MAX_LEVEL",
            "value": "TRACE_LEVEL_INFO"
        }
    }
}

I have tried:

• changing clock sources
• setting lpticker_lptim to 0
• overriding lpticker_delay_ticks from and testing with values from 0 to 4
• changing uart rx/tx buffer sizes
• disabling trace
• testing with different versions of mbed-os with every combination of the above

I even tried setting a custom idle callback function, nothing changed. Nothing seems to work except disabling TICKLESS mode, which is not an option for my project. This is a blocker for my project because it requires low-power mode and is rarely turning on the modem and sending data to the backend.

Issue request type

[ ] Question
[ ] Enhancement
[x] Bug

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-161

[jeromecoutant]

Hi
Maybe you can try to add a sleep_manager_lock_deep_sleep() call
somewhere in the cellular init procedure in order to avoid going into deepsleep ?

[mfatiga]

Hi!
Did that already, no difference. Also tried to set a different idle hook (empty) at boot, also no difference.

[kjbracey]

I would not expect deep sleep to be entered at all - both the console and the modem are using UARTSerial - ie buffered serial with an IRQ handler attached for reception. Both of those should be holding a deep sleep lock.

Deep sleep would only be possible if the modem is closed (possible) and the console input was somehow deactivated (not currently possible).

[kjbracey]

For test purposes if you're not using console input, you can of course set "platform.stdio-buffered-serial" to false, which would get rid of that lock. Downside is no output buffering.

We do need some mechanism to tell UARTSerial "output only" so it doesn't attach the Rx IRQ handler - then it would only hold the deep sleep lock while the TX buffer was occupied.

[mfatiga]

I don't have any problems with UARTSerial locking deep sleep, the modem is turned on only when sending data, and after that I disconnect, delete the instance of the NetworkInterface object, and turn off the modem power.
My problem is that the modem does not want to connect (UART is losing data) when TICKLESS mode is enabled.

[kjbracey]

I think the console will be disabling the deep sleep, and that sounds like it will ultimately be an issue for you.

But your character loss must be happening when deep sleep is disabled for the modem regardless - I was just bringing it up to note that Jerome's "try locking deep sleep" shouldn't affect anything.

The AT handler stuff in the modems doesn't have proper wake-up mechanisms anyway - they basically handle blocking by entering a polling loop for readability with 1 millisecond sleeps. (See mbed_poll.cpp)

As such they're possible worst-case for "tickless" latency behaviour - they're going to be asking for a wakeup every millisecond, thus every idle call will be trying to set up a 1 millisecond timer manually - harder work than just having a continuous millisecond ticker.

If there was an IRQ latency issue with this, you could get character loss.

Should be easy to reproduce though - same basic behaviour is used by any blocking serial read on console input with "platform.stdio-buffered-serial" true (UARTSerial::read), so you would see it on any console test reading stdin.

Now, just noticed you actually replaced the idle hook (with what - null function?). If that's the case - then I'm not sure what's left. We won't be making sleep calls at all to the HAL. So a flaw in the SysTimer / OS_Tick implementation, or the lpticker it uses?

[mfatiga]

@kjbracey-arm yeah, I'm at a loss too here.
I have tried to replace the idle function with:

• an empty function
• a function that just calls Thread::yield()
• a function that calls wait(1.0f)

I also tried setting lpticker_delay_ticks to 0 to eliminate the lpticker wrapper, but it didn't make any difference.

I also suspect that the problem is somewhere in the ticker(s), but I don't have enough knowledge or experience to find out what and where exactly is the problem.

[kjbracey]

You also have CTS/RTS. If that's functional (has a command been issued to the modem to enable its flow control?) then IRQ latency issues should basically be ruled out too. It shouldn't matter if we're sluggish to wait up, the modem should be waiting.

[mfatiga]

I have tried with and without configuring RTS and CTS pins, and it's the same. I don't see flow control is being enabled on the modem, I don't see that the integrated mbed driver is configuring it on the modem. I think that should be updated, so the driver automatically configures it when RTS and CTS pins are configured on a target.
But the problem is even detecting when the modem is turned on to be able to issue any commands to it because the received data is missing characters, so the received data is not reliable.

[kjbracey]

I would expect a wait in the idle function to kill the system - I believe RTX requires that there always be a "ready" thread, so idle has to always be "ready" - it can never wait for something.

Empty and yield would be fine. Just __DSB(); __WFE(); would be a viable minimal "pause the CPU" implementation. Any of those should have the same basic effect - CPU active or frozen waiting for the next millisecond tick, and no real delay in processing it when it happens. When not stopped via the idle hook, the "tickless" OS does still have regular millisecond ticks. And the blocking serial reception is checked on those ticks via polling, not the serial Rx IRQ.

It smells as if the ThisThread::sleep_for(1) calls in the serial/poll blocks are taking significantly longer than 1 millisecond. Maybe use a (micro-second resolution) Timer to check how long they're actually sleeping for.

[mfatiga]

@kjbracey-arm I will try and measure how long the UARTSerial::wait_ms function is actually sleeping for.