GPIO UART - Missing characters when sending longer strings

[endlesscoil]

After upgrading from the 4.4.x kernel branch to 4.9.x using rpi-update I started noticing missing data on the GPIO UART when transmitting longer strings that are being echoed, character by character, back to the sender.

The initial tests were performed with our circuit board that uses the UART for communication. Normal communication seemed to work fine since in most cases sends to our device are single characters, but one case when we sent a larger configuration string caused missing data to become apparent. This device uses 115200 baud and has had no issues with previous versions of the Pi boards or firmware. See Test #1 below.

In order to eliminate our device firmware as causing the issue I replaced it with a TTL serial adapter connected to my workstation with a simple python script that read and echoed each character it received. See Test #2 below.

The tests were performed simply by pasting a long string into 'screen' that had the UART open. After a certain point the communication breaks in a specific pattern. See comparisons below. It should be noted that none of the missing characters were received by our device or the echo script. I was able to replicate the issue with 9600, 19200, 115200, and 330400 baud. It seems like the lower the baudrate the more consistent the issue. At 9600 baud it happens nearly every time, while 115200 might take a few attempts.

Device: Raspberry Pi 3 Model B
Base image: 2017-01-11-raspbian-jessie-lite.img
Initial kernel version: Linux raspberrypi 4.4.34-v7+ #930 SMP Wed Nov 23 15:20:41 GMT 2016 armv7l GNU/Linux
Upgraded kernel version: Linux raspberrypi 4.9.11-v7+ #971 SMP Mon Feb 20 20:44:55 GMT 2017 armv7l GNU/Linux

Steps performed:

1. Flashed 2017-01-11-raspbian-jessie-lite.img to SD card
2. Enable SSH on boot partition
3. Boot
4. Removed console=ttyAMA0,115200 from /boot/cmdline.txt
5. sudo systemctl disable serial-getty@ttyAMA0.service
6. Added enable_uart=1 to /boot/config.txt
7. Reboot
8. Verified device works correctly.
9. sudo apt-get install rpi-update
10. sudo rpi-update
11. Reboot

Test Results #1 (with our device):

Sent: ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890
Received: ABCDEFGHIJKQY7EMU3

Comparison:

ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890
ABCDEFGHIJK     Q       Y       7       E       M       U       3    

Test Results #2 (with echo script):

Sent: ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890
Received: ABCDEFGHIJKLMNOPQRSY7EMU3

Comparison:

ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890
ABCDEFGHIJKLMNOPQRS     Y       7       E       M       U       3       

[JamesH65]

We now use 4.9 as default, and this report seems to be the only one we have had in., so something is awry. Please update to the latest kernel release kernel and retest.

[endlesscoil]

So I'm not seeing the issue with my echo script anymore, however I am still seeing it with our device itself. I'll do some more testing and report back.

[endlesscoil]

Okay, so I think I've made a little progress on this. It appears to be an issue with the mini-uart in the 4.9.x kernel that was not present in the 4.4.x series.

Latest OK Kernel: Linux raspberrypi 4.4.50-v7+ # 970 SMP Mon Feb 20 19:18:29 GMT 2017 armv7l GNU/Linux
Latest Failing Kernel: Linux raspberrypi 4.9.32-v7+ # 1008 SMP Thu Jun 15 17:44:57 BST 2017 armv7l GNU/Linux

Product Test
First, I eliminated our hardware as the source of the issue by testing all three versions that we sell (USB, DB9 Serial, GPIO) using our latest firmware. No issues were seen with the USB or Serial versions.

As for the GPIO version, the error case isn't reproducible by just, for example, holding down a key inside of screen. I've only seen it while sending a "large" amount of data at it. My current real-world test string is 92 characters, however the breaking point seems much smaller than that: anything over 11 characters starts the pattern of missing characters. Maybe the transmit buffer is filling up too fast, at which point the slower clock can no longer keep up with it?

Test String and Echoed Response

Sent: MODE=A&CONFIGBITS=ff04&ADDRESS=18&LRR=N&COM=N&EXP=NNNNN&REL=NNNN&MASK=ffffffff&DEDUPLICATE=N
Received: MODE=A&CONFSD8CPR&fET

Comparison:

MODE=A&CONFIGBITS=ff04&ADDRESS=18&LRR=N&COM=N&EXP=NNNNN&REL=NNNN&MASK=ffffffff&DEDUPLICATE=N
MODE=A&CONF     S       D       8       C       P       R       &       f       E       T

Workaround
I was able to workaround the issue by forcing it back over to the hardware UART by swapping enable_uart=1 for dtoverlay=pi3-miniuart-bt. The issue with missing characters is no longer present at that point. I think this is an acceptable workaround for us, however it still seems like something is wrong with the mini-uart if it previously worked, and it would be nice to allow our users to use the built-in Bluetooth.

Please let me know if you need anything else.

[pelwell]

Thanks for the update - this is on my queue to look at.

[pelwell]

The underlying cause is likely to be behind #2052 as well.

[pelwell]

I have a one line patch that fixes this, but I need a bit longer to understand why it is needed and whether it is the right solution.

[DaveXanatos]

Thanks, looking forward to testing when available.

[pelwell]

The regression is caused by a bug/difference in the UART hardware exposed by switching to the dedicated upstream AUX UART driver instead of using a generic 16550 driver with a few additional settings.

The 8250/16550 driver supports many different variations on the original 8250 design, with parameters and flags to modify the behaviour. It also supports a number of pre-canned configurations, selected using Device Tree compatible strings.

Linux 4.9 introduced a bcm2835-aux-uart configuration. Because the driver is configured through its own probe function (as opposed to the generic 8250 probe function) its defaults are configured slightly differently. In particular the tx_loadsz value defaults to the fifosize value before the underlying 16550 default value is applied, which for the AUX UART means that the value changes from 1 to 8.

tx_loadsz controls how many bytes are written to the TX FIFO on each attempt - larger values should reduce interrupts and increase line utilisation. Unfortunately the MINI UART has a THRE (TX FIFO "empty") bit that is really means "not full", so writing more than one byte after checking THRE can lead to data loss. Luckily we already have a CAP_MINI flag indicating that we are dealing with a MINI UART, so it is easy to add an extra test within the transmit loop that the FIFO hasn't just gone full.

A patch adding this extra THRE checking is now in rpi-4.9.y and will be in the next firmware release.