Uploader prefs miss {build.core.path}, {build.system.path} and {build.variant.path}

[JAndrassy]

The keys {build.core.path}, {build.system.path} and {build.variant.path} are not evaluated for platform.txt tools recipes (tools.xy.upload.pattern).

Then the tools recipes must use runtime.platform.path. But then there are problems to use the platform as referenced core, because runtime.platform.path is the referencing board platform path.

For example in the SAMD package the upload.pattern contains {runtime.platform.path}/variants/{build.variant}/{build.openocdscript} instead of {build.variant.path}/{build.openocdscript}. A board in referencing board platform can use in board.txt an existing variant of the referenced core platform in xy.build.variant for compilation, but the upload tool will not find the script for the variant.

I try to make the esp8266 package to be used as as referenced core. Now it uses {runtime.platform.path}/tools all over platform.txt and {runtime.platform.path} will evaluate to referencing board platform, not to referenced core platform. They could use {build.core.path}/../../tools or rename the folder to system and use {build.system.path} variable. It works for building, but it doesn't evaluate in uploader.pattern.

I debugged the Arduino IDE 1 and found out that compiler recipes use prefs evaluated by arduino-builder, while Uploaders use the preferences map of the IDE. So it is not a simple fix. I think, the best way for fixing this would be to pass the prefs evaluated by Compiler over SketchController from build() to upload(). Same in class processing.app.Base.

[per1234]

I did a test with the latest nightly of Arduino CLI before moving this and those properties were not expanded in the upload recipe. So I don't think your feature request is resolved. If I'm wrong about that, and it is now supported, then it should be closed as resolved.

It's the Arduino CLI codebase where this feature request would be implemented, which will then eventually become available to all the tools that use Arduino CLI, including the IDE. So it now makes more sense for this issue to be in the arduino/arduino-cli repo than the IDE repo.

[JAndrassy]

the issue is not resolved in IDE 2.0

[JAndrassy]

after esp8266 and esp32, the pico core is next platform which stockpiles all boards because it can't be referenced because of this problem. and custom boards definitions in sketchbook's hardware folder have to copy and maintain files used in upload recipes.

the specification as written now makes this a feature, not a bug, but this problem practically stops use of referencing a core.

Note that, unlike core references, referencing a tool recipe does not result in any other resources being inherited from the referenced platform.

Is it really not possible to make it work in CLI and IDE 2?

[JAndrassy]

the issue is not resolved in IDE 2.0

[JAndrassy]

after esp8266 and esp32, the pico core is next platform which stockpiles all boards because it can't be referenced because of this problem. and custom boards definitions in sketchbook's hardware folder have to copy and maintain files used in upload recipes.

the specification as written now makes this a feature, not a bug, but this problem practically stops use of referencing a core.

Note that, unlike core references, referencing a tool recipe does not result in any other resources being inherited from the referenced platform.

Is it really not possible to make it work in CLI and IDE 2?

[mcbp223]

While waiting for an implementation, can we symlink to the tools directory in the referenced package and expect the same behavior once a solution is adopted?

[JAndrassy]

While waiting for an implementation, can we symlink to the tools directory in the referenced package and expect the same behavior once a solution is adopted?

if you meant the tools folder of the esp8266 arduino platform, then yes the simlink is a good workaround.
you will still have to change to runtime.tools.eboot to {build.core.path}/../../bootloaders/eboot/eboot.elf

[mcbp223]

While waiting for an implementation, can we symlink to the tools directory in the referenced package and expect the same behavior once a solution is adopted?

if you meant the tools folder of the esp8266 arduino platform, then yes the simlink is a good workaround. you will still have to change to runtime.tools.eboot to {build.core.path}/../../bootloaders/eboot/eboot.elf

Thanks for the hint on {build.core.path}, I haven't tried referencing the esp8266 platform yet but it's in the pipeline. I'm currently using the method to reference esp32, with a platform.txt file containing only the name of the collection, boards.txt with esp32: prefix additions to each board for {build.core}, {bootloader.tool}..., {upload.tool}..., and variant folder copied to the referencing package, to allow for customisations and library specific include files. There was too much to copy for the tools, a symlink worked for me better.
I tried referencing Earle's RP2040 core but no luck yet with tools location, I'll try what you suggested using a relative path. I've heard other people had success with stm32, and of course avr, planning on trying them. Looking to do more referencing and less copying, and eventually not having to make symlinks that need to be changed with each platform upgrade.

It's important to get this method to work, and be somewhat standardized and relatively easy to use, since it answers to years of requests from both users and library maintainers to allow libraries depend on custom user boards, does best to model reality (a combined board + shields has different specs than the ones shipped with the platform for just the board), and at the same time allows for a unidirectional dependency graph:
dev board -> dev board + shields -> libraries -> sketch

[JAndrassy]

@mcbp223 I don't want to pollute the issue here with a discussion so just two links for you
esp8266/Arduino#5769
https://github.com/JAndrassy/my_boards
If you want to discuss it more I am on arduino.forum.cc or open a discussion in my my_boards repository