Flash/RAM scalability: strategies for splitting up INAV firmware

[sensei-hacker]

INAV's total feature set no longer fits in flash on many boards, yet no single pilot needs most of it: everyone needs a different ~350KB slice. I'd like to compare notes with others working on this. Below are the structural options we brainstormed (separate from the tactical effort of shrinking individual consumers). Research was AI-assisted; the sizes quoted are measured from map/nm output, but the rest is unvalidated.

Ideas so far

1. Tiered build variants: coarse feature tiers (cheap, modest payoff).
2. On-demand custom build service: cloud builds like Betaflight's (closest to a full solution; highest cost and maintenance).
3. Pre-built protocol-variant matrix: pre-built binaries per receiver/telemetry combination.
4. Hook-point / driver registration refactor: an enabler for the others.
5. External aux-chip offload: move loosely coupled features to a second chip (e.g. the Bluetooth ESP32 on some boards).
6. Fixed-address protocol slot + pre-flash hex patching: client-side splice of the Intel HEX before flashing.
7. Union the mutually exclusive protocols' static RAM: ~3.5KB measured, RAM only.
8. General bytecode interpreter: generalizing the Programming Framework (logic conditions).
9. Protocol-specialized bytecode interpreter: for simple serial protocol parsers.
10. Shared plain-C helpers for the simple serial protocols.
11. Runtime-loaded relocatable native modules: e.g. loaded from SD card, with a trampoline/jump table; native speed with no reflash to switch protocols (relocation mechanics are the open risk).

Also floated but not yet written up: interpreter bytecode that doubles as native RISC opcodes, and SD-card-loaded modules via a trampoline. An MC-only vs FW-only build split was estimated separately.

Attached files

All in this gist: https://gist.github.com/sensei-hacker/afd66a91e793c80d2b7ae31ef2509d8d

• comparison-document.md: full comparison of all 11 proposals (prior art, MCU feasibility, cost, payoff, measured footprint of the receiver/telemetry cluster)
• mc-fw-split-report.md: flash savings estimate for an MC-only vs FW/rover-only split
• poc-relocatable-protocol-modules-summary.md: planned spike for Flight - Using sticks in POSHOLD + GPS_CRUISE mode result in failure to hold position #11 (not started)
• poc-hexpatch-protocol-slot-summary.md: planned spike for Flight - NAV_POSHOLD messes up with yaw control #6 (not started)
• ram-flash-strategies.md: running pattern library of RAM/flash reduction techniques
• optimize-firmware-code-size-findings.md: function-level size analysis of MATEKF722
• investigate-onboard-bluetooth-chip-capabilities-summary.md: research plan behind Config - Add a flag to indicate that MAG calibration was done #5

Which of these are you pursuing, and what have you learned? I'd like to avoid duplicating work.

[error414]

personaly I would say this should be the most important iniciative now.

[eminesrefakbay]

Server size and load are the main concerns here. To minimize server load and complexity, a pre-compiled build matrix could be easily adopted.I think a matrix based on core features like:

Fixed Wing / MultirotorGPS / No GPS (which also clears hold-type control code)
Optical Flow / No Optical Flow
PWM Servo / No PWM Servo
Waypoint Missions / No Waypoint Missions
LED Strip / No LED Strip

This yields a theoretical matrix of $2^6 = 64$ builds per target. However, the actual number will be much lower if the Configurator handles feature dependencies (e.g., GPS required for Waypoint missions). This should keep it very manageable.I will compile a custom build for a flash-constrained F722 target with unused features disabled and report back with the memory savings.

[eminesrefakbay]

00_baseline flash 472291 ( +0) ram 103224 ( +0) itcm 10944 ( +0)
01_led_strip flash 465395 ( -6896) ram 88544 ( -14680) itcm 11272 ( +328)
02_optical_flow flash 470027 ( -2264) ram 102912 ( -312) itcm 10832 ( -112)
05_wp_max_15_instead_of_120 flash 472339 ( +48) ram 96928 ( -6296) itcm 10944 ( +0)
06_servo_sbus flash 471107 ( -1184) ram 103128 ( -96) itcm 10944 ( +0)
07_fixed_wing_autoland flash 468911 ( -3380) ram 102568 ( -656) itcm 10936 ( -8)
08_pitot_airspeed flash 467555 ( -4736) ram 102888 ( -336) itcm 10944 ( +0)
11_adsb flash 470195 ( -2096) ram 102768 ( -456) itcm 10944 ( +0)
12_programming_framework flash 463225 ( -9066) ram 102720 ( -504) itcm 10680 ( -264)
13_telemetry_protocols_extra flash 446583 ( -25708) ram 100956 ( -2268) itcm 10784 ( -160)
15_multirotor_only_bundle flash 458475 ( -13816) ram 101364 ( -1860) itcm 10832 ( -112)

Extra Telemetry Protocols (13) & Logic Framework (12): Offer the largest Flash savings (~25.7 KB and ~9 KB).

LED Strip (01) & Waypoint Reduction (05): Offer the largest RAM savings (~14.6 KB and ~6.3 KB). Reducing max waypoints from 120 to 15 frees up over 6 KB of RAM with zero Flash impact.

Multirotor-only bundle (15): Instantly saves ~13.8 KB of Flash by excluding fixed-wing features.

Combining just a few of these high-impact toggles can easily reclaim 35 KB+ Flash and 20 KB+ RAM, showing that a modular build matrix managed via Configurator rules is a viable path for constrained targets like F722. Target is Dakefpvf722x8.

[error414]

I read your document many times. for dynamic modules, or loading byte code we would probably would have to rewrite half of Inav code. Separate modules and make strict ABi beetween them. It's seems amount of work like rewite monolit app to microservices.

Paresonaly I would choose same way as betaflight or ardupilot do, precompile FW with USE_XXX. Precompile FW is robust and behave exactly same everytime.

The main cons is need for own server, paid :(, and needs to maintain many combinations.

For choosing what options could be in configurator, I would not focus to how many memory we save, but how users would like to set build.
For example from my perspective I would like to choose target, then HW because how many I change HW on plane, then features.

[sensei-hacker]

for loading dynamic modules or byte code we would probably would have to rewrite half of Inav code. Separate modules and make strict ABi between them.

Yeah for loading from the SD card or a reserved slot using a trampoline, the ABI would have to be defined for the swappable class. Such as rcReceive() would need the same interface for SRXL2, CRSF, Sbus, etc. They would all need to return an array of channels. Only for the hot-swappble functions, like different receiver protocols.

A barometer driver module would have to return the baro value as an int32 every time - so the return value is the same type regardless of which of 12 different baros you use. I think that's mostly already done. I think it pretty much already has a stable API for most hardware drivers.

The documents aren't as clear as I'd like on another variation, flash-time linking. Yes, features that we want to have optional would be refactored into separate module files. So loadTerrain() would be a function, in the terrain file. Not mixed in with the PID loop.

Then that function declaration is the ABI. Same as any other function we have today.
If you write this:

uint8_t[255] loadTerrain(int32_t);

Then that's your ABI -- it takes a 32-bit int as an argument and returns an array of 8-bit.
So there's nothing special there. What changes for optional features is that the feature should be implemented in a file, or set of files, not blended together with unrelated code.

Gcc always compiles each file separately.
Then the linker rewrites the addresses of those compiled so modules (files).

The interesting bit would be that the linker runs in Configurator instead of running on my laptop. Linking a custom hex, rather than me looking them all before a release.

[error414]

ok, I see your main goal is to avoid server compiling because it would need maintanence and mainly money, What I fully understand.

Receiver is simple example, but what about telemetry. Telemetry needs lot of informations from core, so ABI would have to be a huge. Or telemetry manager would have to compose struct with all information and provide to specific telemetry driver via ABI.

Each type of module, like Baro, MAG, Pitot must have reserved space, the size must be by the biggest module. Some HW is on FC, like baro so it's not needed to load it as module, because for 99.9% cases user will not connect diferent baro to FC.

Be honest I have never heard about "flash-time linking" but sounds interesting, it does not need ABI, it has access to any function in core, and it's created static hex without any dynamic loading in runtime. But still there is disadvantage that that for in ld file must be reserved space for modules (I hope I'm right).

Just to be sure, you want to move to module things which are already somehow separated in INAV, like HW drivers (baro, pitot, rangefinder, ......), for I2C devices just keep in mind that there must be async calling which is more complicated, it's similar like state machine in terrain feature, and features like adsb, terrain, ...... (have no idea how other features can be separated).

Are you going to create some concepts to verify that it can work? I really would like to see it, because imagine all in head is not easy
:D.

[eminesrefakbay]

Best approach is and also cheapest is wsl local compiling but more automated version . As all of expert users like us use manually . Users clone git repo than compile locally if they want.

[sensei-hacker]

Yes, and there is simpler, more automated version that doesn't require every user to install WSL and a whole build chain.

The actual compiling doesn't have to be different per user, it's the LINKING step that decides which modules go into each hex.

[sensei-hacker]

ok, I see your main goal is to avoid server compiling because it would need maintanence and mainly money

Yeah and it would need a guarantee of how that money is going to be paid two in years from now. It would be a permanent required ongoing expense.

But still there is disadvantage that that for in ld file must be reserved space for modules

There's no space reserved for modules that aren't actually being used.

A typical build goes like this:

1. The preprocessor expands the macros
2. The compiler compiles each .c file to make a .o object file (module)
3. The linker joins the compiled .o files to make a single .hex
4. The hex files are uploaded to GitHub
5. Configurator copies the hex file to the FC

The change would be:

1. The preprocessor expands the macros
2. The compiler compiles each .c file to make a .o file
3. The .o files are uploaded to GitHub
4. The linker in Configurator joins the .o files to make a single .hex
5. Configurator copies the hex file to the FC

Swapping steps 3 and 4.

[error414]

Ah, I understand, it sounds promissing. So I would avoid aproach with dynamic module loading and try linking object files on client.

At first I thought that dedicated build server could cost aprox 200 - 300$ per year. It was naive, I rafly calculated it and I got number 2000 - 3000$ per year.