Add support for g-code compression/packing over serial connection (MeatPack)

[scottmudge]

UPDATED - Cleaned up whitespace changes and updated to 3.12.2 base, for easier change review.

MeatPack - Easy, fast, effective g-code compression/packing

I've devised and implemented a g-code packing method which is very simple and has very little processing overhead on the MCU, but manages to reduce the size of g-code by a ratio of ~0.61. This is on-par with binary encoding, but it does not require any modifications to the parsing code, g-code files, or anything else. The MeatPack code/module is entirely self-contained, and it only required a few lines to be changed in cmdqueue.c to be integrated.

This effectively increases the baud rate from 115,200 to 188,850, improving print quality for parts with very dense g-code.

I've also written a functional and tested OctoPrint plugin to work with this modification here:

https://github.com/scottmudge/OctoPrint-MeatPack/

The plugin repository better explains how the packing algorithm works. It's fairly simple and requires very little processing power, but leverages the limited nature of the character set used by most g-code.

The implementation with the firmware leaves the firmware functionally identical to how it already exists. The data packing mode is only enabled if a particular set of command bytes (well outside the ASCII range used by standard g-code) is sent to the MeatPack module by the associated OctoPrint plugin. Otherwise the firmware behaves exactly the same as it always does.

The plugin handles everything automatically. G-code files can remain completely unchanged. The plugin extends the standard PySerial Serial() object with additional functionality, packing whatever g-code data produced by OctoPrint right before it is sent to the firmware.

It also manages and synchronizes the MeatPack "state" of the firmware based on the active configuration of the plugin. If the user enables or disables g-code compression/packing, the plugin module will automatically send associated command bytes to the firmware, where the packing mode is dynamically enabled or disabled. The firmware also sends back confirmation of the enable/disable state. Overall the synchronization is very stable, and I have encountered no issues.

If the plugin is not installed or the user has disabled g-code compression in the plugin settings, the firmware will simply behave as it normally does.

How well has it been tested?

I've fully tested both firmware modifications and OctoPrint plugin. I've printed about 100-200 MB of various g-code (produced by both PrusaSlicer and Cura), which was effectively packed to a size of ~60 - 120 MB.

No errors or issues were encountered. And very dense g-code files which normally produced quality issues (zits, stuttering) when being printed at a high feed-rate over the serial connection no longer encountered the same quality issues when the data was packed with this module.

More Information:

Plugin Repository: https://github.com/scottmudge/OctoPrint-MeatPack

[wavexx]

Looks clean! Did you check if we get rid of "plain" strtod at link time, or we still have extra references to it?

[scottmudge]

I don't see it linked anywhere in the ELF binary, so I think it skipped over it:

But that was a concern when I implemented the fixed version.

[leptun]

I rebased the PR onto the latest MK3 branch and fixed the merge conflicts and made it compile with the new cmake build system. The PR uses 538B of flash and 24B of RAM.
I haven't pushed these changes yet since rebasing made me the author of the commit, which I am not.
@scottmudge can you please recommit the changes from https://github.com/leptun/Prusa-Firmware/tree/MK3_3.12-meatpack-pr and force push to this branch so that you are the author of the commits? You also need to change the target branch of the PR to MK3 instead of MK3_3.12. If you can't do that, please let me know.

[scottmudge]

@leptun Sure no problem -- thanks for resolving those merge conflicts to MK3 branch! I should be able to get it done today.

[scottmudge]

@leptun - Alright, should be done, let me know if anything doesn't look right.

[arekm]

"This effectively increases the baud rate from 115,200 to 188,850" - I wonder about that since I was never been able to get my (stock fw) MK3S+ to process g-codes (over serial) faster than 3KB/s which is far less than ~11KB/s of serial port there.

The test was like sending the same simple single g-code over and over. Don't remember which one but I used some g-code that did basically nothing, so firmware was only parsing g-code line and checking checksum repeatedly.

I guess I'll have to check that again.

[scottmudge]

I mean I suppose it can only process so many commands per second, depending on the particular command(s) being processed. If they are very short commands (only a few characters) that still require X number of MCU cycles to complete, the bandwidth required for that command throughput would be relatively small. But when you're sending a bunch of coordinates for small line segments, like with a curve, it might be a different story. What kind of command were you sending, and how long does it take to execute?

[leptun]

@scottmudge It looks perfect now. Thanks for rebasing 👍. I'm hoping I can get this PR into the 3.13.0-RC1 release.

Not sure why the PR has no automated checks running... Seems to build just fine for me, so it's fine.

[scottmudge]

@leptun - No problem, and that sounds excellent!

[leptun]

Rebased again since the previous state of the MK3 branch was quite buggy. Now many UI bugs should be fixed, so testing this PR should go smoother.

[3d-gussner]

On MK404

• Can't upload to SD but that is probably the MK404
• For the rest it works with OctoPrint + Meatpack plugin ✔️

On real MK3S

• Uploading a 103kb file from OCtoPrint to SD
• Meatpack disabled ~ 17.5 sec ✔️
• Meatpack enabled ~ 14 sec ✔️
• Both printable from SD card ✔️
  Printing from PS->OctoPrint works as well. ✔️

MP_CP.gcode 103.3kB is the PrusaSlicer generated one while MP_CP.GCO 73.7kB is the via OctoPrint+meatpack uploaded file to SD card.
MP_CP.GCO.txt
MP_CP.gcode.txt

Even with the meatpack compression it will take some time to get large files to SD uplo
aded. I have seen compression of 0.5 ish but with 115200 baud it will still take for ever.

Uses Flash +540 RAM +24

[3d-gussner]

@scottmudge Thanks for the PR and sorry it took so long. After some magic we got some resources free and now it should fit.

Please consider also to update https://github.com/prusa3d/Prusa-Firmware/blob/MK3/Community_made.md

[scottmudge]

@3d-gussner Awesome! No worries, I know space is tight, but glad you could fit it in.

Should I add a new section to Community_made.md for MeatPack/gcode compression? I'd be able to cross-link with the OctoPrint plugin as well.

[scottmudge]

You probably already thoroughly tested it on your end, but just wanted to confirm MeatPack is working perfect on stock 3.13.0, on my MK3S+. Just as an extra data point. So nice not to need to create separate builds! Also really like the updated build pipeline w/ VS Code on windows. Much faster.

Thanks again for the review/merge.