Improved compile speed by running multi-threaded library discovery.

[cmaglie]

Please check if the PR fulfills these requirements

See how to contribute

• The PR has no duplicates (please search among the Pull Requests
  before creating one)
• The PR follows
  our contributing guidelines
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• UPGRADING.md has been updated with a migration guide (for breaking changes)
• configuration.schema.json updated if new parameters are added.

What kind of change does this PR introduce?

This is a tentative to improve compile speed by multi-threading the library discovery phase.

How library discovery works

The library discovery is a sequential process, and could be summarized as follows:

• The sketch source files (compilation units) are added to a queue
• An empty includes path list is initialized
• For each file in the queue:
  • A C++ macro preprocessing is run on it (a.k.a. gcc -E ...)
  • If the operation returns a Missing include file "xxx.h" error then:
    • A library providing xxx.h is added to the includes path list
    • The source files (compilation units) of the library are added to the queue
    • The current source file is preprocessed again to see if there are other missing includes

The above loop continues until one of the following happens:

• All the files in the queue are processed successfully
• A missing include can not be provided by any of the installed libraries
• There is a syntax error different from missing include

Could this be multi-threaded?

The main issue here is that each run of gcc -E ... requires the includes path list determined by the previous iterations. At first sight, it seems a strictly sequential process.

BTW, there are some strategies that we may use:

1. When a library is added to the queue, we may speculatively start a multithreaded compilation of the next files in the queue assuming no missing include errors will be produced. Most of the time this assumption turns out to be true and the time saved could be dramatic.

2. We may leverage the "library detection cache" to spawn a speculative multithreaded compilation of all the files (predicting all the libraries added in the process), assuming that the missing include errors would be the same as the previous successful compilation as saved in the "library detection cache". In this case, we reproduce the same compiles as the previous build. The basic assumption is that the libraries detected will not change between compilations.

If an unforeseen missing include error is detected, the speculative multithreaded compilation must be canceled and restarted (basically wasting the work done ahead of time). In a multi-core system, since the work "ahead" is done in parallel, the worst-case scenario should take nearly the same time as the "classic" non-multithreaded compile.

Due to the complexity of these changes, I'll start this as a draft. I want to add integration tests to trigger edge cases before merging.

What is the current behavior?

What is the new behavior?

Compiles should in general be faster than before.

Does this PR introduce a breaking change, and is titled accordingly?

No

Other information

[codecov]

Codecov Report

Attention: Patch coverage is 85.04673% with 48 lines in your changes missing coverage. Please review.

Project coverage is 68.22%. Comparing base (ae68728) to head (7241f25).
Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
...rnal/arduino/builder/internal/detector/detector.go	87.95%	5 Missing and 5 partials ⚠️
internal/arduino/builder/sizer.go	12.50%	6 Missing and 1 partial ⚠️
internal/arduino/builder/builder.go	81.81%	4 Missing and 2 partials ⚠️
...lder/internal/preprocessor/arduino_preprocessor.go	0.00%	6 Missing ⚠️
internal/arduino/builder/internal/runner/task.go	86.04%	4 Missing and 2 partials ⚠️
internal/arduino/builder/compilation.go	86.84%	3 Missing and 2 partials ⚠️
commands/service_compile.go	72.72%	2 Missing and 1 partial ⚠️
internal/arduino/builder/internal/runner/runner.go	96.55%	2 Missing ⚠️
commands/service_monitor.go	0.00%	1 Missing ⚠️
internal/arduino/builder/core.go	0.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2625      +/-   ##
==========================================
+ Coverage   68.01%   68.22%   +0.21%     
==========================================
  Files         239      241       +2     
  Lines       22589    22696     +107     
==========================================
+ Hits        15363    15484     +121     
+ Misses       6023     6012      -11     
+ Partials     1203     1200       -3     

Flag	Coverage Δ
unit	68.22% <85.04%> (+0.21%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[manchoz]

LGTM

[alessio-perugini]

From my POV, is okay. I tested it on some sketches and I can see the improvement.
However, let's wait for some other users to gather feedback. (Just in case some edge cases arise).

[frankcohen]

Hi, I'm using Arduino IDE 2.x everyday on my open-source ReflectionsOS project. It's an ESP32-S3 board with a bunch of sensors. About 18,000 lines of code so far. Compiles on my MacBook Air M2 2022 8 Cores takes about 3-4 minutes. I'd be glad to spend time and money testing the PR. Would someone please give me instructions on how to get the PR code and compile the IDE for Mac? -Frank

[cmaglie]

Hi @frankcohen,

the instructions to test the PR are here:
https://forum.arduino.cc/t/call-for-beta-testers-arduino-cli-sketch-compilation-optimization/1291664

use the following link to download the latest version of the patched arduino-cli:
https://github.com/arduino/arduino-cli/actions/runs/13310273692

[kovachwt]

Hi @cmaglie

I would like to test this as well, as ResolveLibrary calls are taking >30s out of my 45 second compilation time.

However, the artifacts have expired from that run, anyway to re-run the action and get new executables?

[alessio-perugini]

@kovachwt

Using gh

gh repo clone arduino/arduino-cli
cd arduino-cli
gh pr checkout 2625
go build .

Using git

git clone https://github.com/cmaglie/arduino-cli.git
git checkout libs_detector_improvements
go build .

Then you will find the arduino-cli` binary in the project's root dir, and you can use that to test.

You'll need to have git and go installed in your system

[kovachwt]

Thanks Alessio, I was able to build and test this branch.

Works great for me, I compiled all the projects I have with it, no issues.
Tested with ESP32, AVR and SAMD boards.

Compile times have reduced substantially, from 45 seconds in one esp32 project down to 15 seconds.
Great work everyone!