[Request] Improve serial latency

[dwillmore]

There has been a number of users on the IRC channel on freenode #octoprint who have expressed the desire to have decreased latency in the serial connection between the octopi server and a serially (via USB<>Serial adapter) connected printer. I have looked into this and found a couple of things which may help.

The first of them is to use the command "setserial /dev/ttyUSB0 low_latency". Unfortunately, the package containing setserial is not installed by default. It may be easier to see what setserial is doing when running this command and implement that in octoprint.

I have tested this change and found an improvement in command submission rate. To test this, I composed a file of 32K lines of alternating G20/G21 commands. These were recommended by the Marlin firmware people on their IRC as as close to a NoOp as they have. My tests showed that an unmodified Octopi 0.13.0 setup took 2m56s. After the setserial command it took 1m49s.

Setup is Octopi 0.13.0/octoprint 1.3.8 running on a rpi 2B connected via USB to a Wanhao Duplicator i3 V2 with stock firmware.

The next thing I have no yet tried is to change the HZ value in the kernel to 1000. This has been reported to improve latency on other systems and I expect it will help here as well. I have built the kernel, but have not had time to install it on the server and test it.

I have not tested printing actual prints with this change in place. I will be doing that in the future as well.

[foosel]

Interesting. One thing to note is that /dev/ttyUSB0 isn't necessarily the only serial port that a printer might register under. I've also seen things like /dev/ttyUSB? with other values than 0 and /dev/ttyACM? in the field, plus of course self specified device names (udev rules...).

So if something like this were to be bundled, it would have to default to sane defaults, or somehow be hooked into OctoPrint (plugin?) to be applied automatically on connect. Which brings me to a question - when does this command need to be issued? After connect? Prior? Does it have to be reissued on reconnects?

Setup is Octopi 0.13.0/octoprint 1.3.8

Would also be interesting to repeat the test on the current versions ;) We are at OctoPi 0.15.1 and OctoPrint 1.3.10.

[dwillmore]

Yes, it would need to tie into octoprint to get the printer device name, that makes the "tear seterial apart and see what it's doing" option sound better. Just incorportate that into octoprint.

The sane defaults concern could be addressed by a config option that said "use Low_Latency setting for serial connection" and default to off. I believe the command needs to be issued on every connect for USB serial adapters as the device state is destroyed when the adapter is disconnected. There is no persistant state as such, IIRC.

I will try with newer versions once I have time. I'm currently using my existing working setup--and I'd like not to break it as I rely on it working daily. I may break out different rpi boards to see how they are impacted by these changes as well. I had noticable issues on a single core rpi B+ which caused me to move to a 2B.

[sbts]

@foosel @dwillmore I wouldn't bother messing with what setserial actually does.
Instead simply test for it's availability and call it before attempting to connect.

In the case of port autodetection, once connected should probably work.
However it would be better to first identify the correct port, then disconnect, setserial, reconnect.

IIRC if we were only concerned about OctoPi as distinct from OctoPrint, then it would be possible to set lowlatency via a udev rule.
That of course doesn't help the various other ways Octoprint can be run.

@foosel this link Notes on FTDI Latency with Arduino has a good overview of what's going on etc

[guysoft]

Hey,
Looking in the source code of setserial, its setting flag 0x2000.
Source: pyserial/pyserial#290 (comment)

This can also be done in pyserial which is shipped in OctoPrint and OctoPi.
pyserial/pyserial#287

@foosel I think you can probably add this, example usage:
pyserial/pyserial#290 (comment)

[sbts]

@guysoft Yes, it is possible to do that.
However, depending on the platform that is being used (eg: Linux, MacOS, Windows, Android) it is MUCH safer to call the system binary as the exact implementation (eg: which flag is set and how) can and likely will change.

Even worse is the fact that different USB => Serial devices may require this to be done in different ways depending on the platform.

[guysoft]

@sbts I am not sure if setserial exists in windows or mac, any input would help here because from what I can tell this is a linux thing only.

Also I can't really handle other OSs in the scope of OctoPi.
Would also value @foosel 's input here.

[dwillmore]

At this point, it looks like this is a Linux specific issue (the setserial ioctl). So, OctoPi could just implement it in udev or as a default plugin. That wouldn't allow people who use OctoPrint on Linux machines not by way of OctoPi. I don't know how many people that is.

Would a plugin be capable of performing this action? If so, that might be the best solution. Then OctoPi could enable it by default and OctoPrint could enable it at their option and add 'setserial' to the list of suggested/required packages on Linux install.

The kernel HZ setting is a Linux issue as we don't get to change that value in MacOS and Windows, I imagine. @guysoft, do you build the Linux kernel from the Raspbian sources when building OctoPi or do you use one of the foundations premade kernels? If you don't build the kernel, would you be willing to start doing so if this change shows a reasonable benefit? Thank you.

[sbts]

@guysoft I guess what I'm saying is that calling a system binary to handle system specific config makes more sense than including system specific code within the OctoPrint application.

That then allows trivial support for non linux platforms.
I'd expect that most *nix systems will have a setserial binary (including Mac) while for windows that may need to be provided by custom binary or simply ignored as the binary doesn't exist.

Further to all of this, it's probable that elevated privileges will be required to run the setserial equivalent code.
For setserial that's handled under most linux platforms fairly well by either correct permissions on the device or having setserial SUID root.
However, under windows and likely OSX that's probably a different story.
If the code was part of OctoPrint it's self, then foosel would need to consider the implications of gaining those elevated privileges.

From the setserial manpage....

For the most part, superuser privilege is required to set the configuration parameters of a serial port. A few serial port parameters can be set by normal users, however, and these will be noted as exceptions in this manual page.

NOTE: low_latency is not one of the parameters that a user can change

[sbts]

@foosel ,
It's worth noting that low_latency is a setting that's, in various forms, available on all platforms, but different platforms have different ways of setting it and different defaults.
It's also worth noting that it "only" affects receive latency.

There are further latency issues to be dealt with in the various drivers for USB devices, in particular in the TX path.

For example, in the FTDI drivers it's well known that there is a 16ms TX delay for incomplete (size) packets.

There are ways of mitigating that by setting options for the driver, most of which require elevated privileges and are required to be done for every "instance" of a device (ie a physical disconnect/reconnect would require them to be set again)

I agree that we should find ways of leveraging the improvements that are possible, however, we likely need to do much of it with UDEV rules on linux platforms and perhaps helper scripts or binaries on other platforms like Windows.

[sbts]

@foosel regarding TX latency improvements on all platforms, It's possible a "tuneable" padding parameter could help here.

From memory, for at least the FTDI driver, a serial "packet" is considered full at 64 bytes.
In theory this means that any payload that is not an even multiple of 64 bytes will have....

• int(count/64) bytes sent essentially immediately
• then mod(count,64) bytes delayed by 16ms

Padding out that last packet to 64 bytes should cause it to be sent immediately.

As not all drivers will have the exact same behaviour making that parameter tuneable (and padding disable-able) would make a lot of sense.

Further, the effect of Baudrate and firmware processing speeds/behaviours needs to be considered as well.
If it will take more than 16ms to transmit and for the firmware to process the "padding" characters then you are better off waiting for the 16ms timeout.

If the padding characters are implemented as gcode comments and the firmware is well written then the firmware processing should have little to no impact as the comment will be directly ignored.
But the TX time could easily be a problem if there is a large level of padding.

Consider the following.

• 64 bytes in 16ms = 4000 bytes per second
• 40000 bytes per second = 40000 baud (40k baud @ 10 bits per byte)

The above means that at baud rates below 57600 padding will only be effective based on the number of bytes required in the padding.
However, at higher baudrates padding should easily speed things up if it's correctly tuned.

---

NOTE: The above becomes irrelevant if "ascii" gcode is being sent (instead of a binary protocol such as GPX uses) IF the driver triggers an automatic send when a <CR> is detected (as long as lines are terminated with <CR><LF> of course
Unfortunately in most cases the drivers require such a setting to be manually enabled, and the means of doing that is driver specific and once again requires elevated privileges.

[ntoff]

From that article

The problem stems from the Arduino’s “Serial to USB converter” chip, the FTDI FT232R.

What if your board doesn't use an FTDI chip? One of mine uses a CH340/341 chip (in fact I highly suspect a large number of the cheap printers are using this chip) and the other uses an atmega16u2 with usb-serial firmware on it for the usb connectivity.

https://reprap.org/wiki/Taurino#Bootloaders

ATmega16U2 (with enhanced firmware) for high speed USB serial connection (up to 2MBit)

[dwillmore]

@ntoff I assume that different USB<>serial chips have different buffer and/or transfer sizes. It would probably make the most sense to have a simple value for this padding and maybe a tooltip saying what common chips prefer. The default would be 0 rendering the feature disabled unless the user specifically turns it on.

I have access to a variety of these serial adapter chips, so I could do some testing on them.

The other two options--setserial and kernel HZ value--would be the kind of thing you could just leave on unless they show some kind of performance regression. I've not seen one, but my testing is anything but extensive.

[ntoff]

Pity I just took apart my old printer that used the taurino board, I used 1mbit baud on that thing, might have been a decent test bed vs my "good" printer that's only set to 115200.