Multiple tickers stop working

[crespum]

I'm experiencing problems with multiple tickers in the nRF51822, actually it's something similar to what is described in #832.

I have reproduced the error with a mod of the Nordic UART example. I've added to the sample code three tickers:

1. Each 20 ms to increase a counter
2. Each 10 ms to increase another counter
3. Each second to send the counters values through the BLE UART

After some seconds I stop receiving data and sometimes the connection is lost after a while but most of the times it is not lost, so I can disconnect and the firmware will respond properly. I've also detected with a different code that after this hang, if I wait long enough, the tickers start working again.

I have the libraries updated so this problem should be solved. Is anyone else having it?

#include <string.h>
#include "mbed.h"
#include "BLE.h"

#include "UARTService.h"

#define NEED_CONSOLE_OUTPUT 1 /* Set this if you need debug messages on the console;
                               * it will have an impact on code-size and power consumption. */

#if NEED_CONSOLE_OUTPUT
#define DEBUG(STR) { if (uart) uart->write(STR, strlen(STR)); }
#else
#define DEBUG(...) /* nothing */
#endif /* #if NEED_CONSOLE_OUTPUT */

BLEDevice  ble;
DigitalOut led1(LED1);
UARTService *uart;
bool cb3 = false;
int cb1_counter, cb2_counter;

void disconnectionCallback(const Gap::DisconnectionCallbackParams_t *params)
{
    DEBUG("Disconnected!\n\r");
    DEBUG("Restarting the advertising process\n\r");
    ble.startAdvertising();
}

void periodicCallback1(void)
{
    cb1_counter++;
}

void periodicCallback2(void)
{
    cb2_counter++;    
}

void periodicCallback3(void)
{
    cb3 = true;    
}

void maintask() {
    if(cb3) {
        cb3 = false;

        char str1[15];
        sprintf(str1, "CB1 %i\r\n", cb1_counter);
        DEBUG(str1);

        char str2[15];
        sprintf(str2, "CB2 %i\r\n", cb2_counter);
        DEBUG(str2);
    }
}

int main(void)
{
    led1 = 1;
    Ticker ticker1, ticker2, ticker3;
    ticker1.attach_us(periodicCallback1, 20000);
    ticker2.attach_us(periodicCallback2, 10000);
    ticker3.attach_us(periodicCallback3, 1000000);

    DEBUG("Initialising the nRF51822\n\r");
    ble.init();
    ble.onDisconnection(disconnectionCallback);

    uart = new UARTService(ble);

    /* setup advertising */
    ble.accumulateAdvertisingPayload(GapAdvertisingData::BREDR_NOT_SUPPORTED);
    ble.setAdvertisingType(GapAdvertisingParams::ADV_CONNECTABLE_UNDIRECTED);
    ble.accumulateAdvertisingPayload(GapAdvertisingData::SHORTENED_LOCAL_NAME,
                                     (const uint8_t *)"BLE UART", sizeof("BLE UART") - 1);
    ble.accumulateAdvertisingPayload(GapAdvertisingData::COMPLETE_LIST_128BIT_SERVICE_IDS,
                                     (const uint8_t *)UARTServiceUUID_reversed, sizeof(UARTServiceUUID_reversed));

    ble.setAdvertisingInterval(160); /* 100ms; in multiples of 0.625ms. */
    ble.startAdvertising();

    while (true) {
        ble.waitForEvent();
        maintask();
    }
}

[0xc0170]

cc @rgrover @pan-

[seajayshore]

Bump!

Any updates on this? It seems I am having the exact same issues described.

Code using multiple tickers is stalling after several minutes (exact timeout varies each time) and, usually, leaving the soft-device alive and "connected". I think it also manages to "come alive" again after stalling, but this seems very rare & I can't reproduce it effectively.

Unlike cresprum here I'm using the ticker.attach mechanism (not the attach_us) variant.

[pan-]

I can reproduce it, my guess is that it is related to the way the different counter in us_ticker are wrapped, unwrapped.

Testing a fix now.

[crespum]

We were struggling with this issue for a while but we are pretty confident the problem has nothing to do with mbed but with the Android app. We developed our own app and never found issues with the tickers anymore.

I was not in charge of the app, but what I understood from the developers is that the reference app uses polling to read the values from the virtual UART. Using interrupts instead seems to solve the problem.

[seajayshore]

@crespum Huh, interesting...

We are experiencing the same issue with both 3rd party BLE apps (for iOS) and our own custom app for iOS/Android.

I've also (last hour) managed to make the normally rock-solid BLE HeartRate example do the same thing by adding 10 tickers with different timings.
It takes a fair bit longer to stall but has done so twice now with the BLE remaining connected under the application. I figure this is because the HeartRate code does far less than our custom mbed code which actually needs to poll several things quite fast (all currently triggered by flags set via Tickers).

@pan- Thanks for dropping in on this. I'll be interested in any progress you make. It seems particularly difficult to debug from my point of view as failure occurs inconsistently and can take several minutes or longer.

[seajayshore]

Having re-written my code to use one "master" ticker and using simple timers for all sensor polling/scheduling there is now a world of difference in stability!

95% of code is the same, at-least all the heavy-lifting stuff is identical. Only difference is as I said, Timers not Tickers.

Connected for 15mins+ no drop out. More encouragingly (and concerning for Ticker-nRF51822 compatibility) is this code now runs fine on my nRF51-DK (nRF51422) AND my production nRF51822 modules with 16kB flash:
Before I had to remove big chunky BLE objects just to get the thing to start on this hardware, I thought I was running out of memory! But now, no fiddling whatsoever, the same full-fat code works perfectly on both platforms.

:-)

I'd love to know & help find-out what on earth is going wrong with this Ticker business. I'll keep updated anything I find. I reckon the nRF's hardware timers are not getting shared properly or the BLE softdevice interrupts & tickers collide in some way...

[seajayshore]

So this Ticker bug is still around in my code... It's a lot rarer now I've replaced multiple tickers with one "master" ticker and multiple timers.

Seems to be a present in all apps that use tickers and Softdevices... (For reference my app is also using InterruptIn, SPI & I2C uses).

I understand the Ticker uses the RTC1 hardware of the nRF51822. Anyone more familiar with the code know if this is also used by other mbed APIs? Or if theres a greater chance of interrupts form RTC1 being missed if I use other peripherals?

[seajayshore]

For what it's worth, I noticed today that when the fail/crash occurs the signal on my I2C device (an ADC) goes from this:

To this:

I think this is another symptom (not a cause) of ticker/RTC problem... But if anyone thinks otherwise then let me know.

[seajayshore]

I2C clearly get's ruined by this fault:
img_2757

[ciarmcom]

ARM Internal Ref: IOTMORF-219