[question] Using custom cross-compilation toolchains with requirements and CMakeDeps

[rollebolle]

What is your question?

Updating to conan 2.3.2

I'm building packages via a Yocto based x-compilation toolchain, and the SDK generates this toolchain for us. It is vital to use it as it sets CMAKE_FIND_ROOT_PATH among other things, to find packages that are in the SDK and does not come from Conan. Therefore I override the generated toolchain with the tools.cmake.cmaketoolchain:toolchain_file option.

However this poses a problem when we also have requirements statements for our Conan requirements. Conan generates the correct <dep>-config.cmake files for each requirement, but they end up under .../build/Release/generators. As we are using a custom toolchain, we have no way of knowing this path in beforehand, and therefore we dont know how to set the CMAKE_MODULE_PATH.

If I instead do NOT use the custom toolchain file, but rely on the generated one from Conan with CMakeToolchain, I find the requirement packages as the module path is calculated and added to the toolchain file. But then I get problems for packages in our cross-compilation SDK that we cannot find.

The solution we had in Conan 1.63 was to use the old syntax without toolchain etc, and using cmake_find_package. This worked, even with a custom toolchain file, because the Find<dep>.cmake files ended up in the build folder. As cmake_find_package is deprecated, I want to move away from this.

What is the best practice here?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @rollebolle

Thanks for your question.

The self.generators_folder is known in the recipe, and it will point to the folder that you want.
What is the mechanism that you need to have this folder communicated to your CMake? You can add any arbitrary variables to the toolchain and presets like:

def generate(self):
     tc = CMakeToolchain(self)
     gen_folder = self.generators_folder.replace("\\", "/")  # Windows-CMake not happy
     tc.variables["MYPATH_GEN"] = gen_folder   # IT will be a variable in conan_toolchain.cmake
     tc.cache_variables["MYPATH_GEN"] = gen_folder  # It will be a cache variables in CMakePresets (and passed in CLI arg)
     tc.generate()

Does this make sense for your use case?

[rollebolle]

Thanks. This does work and I I'll use it. However it means my recipe now has adapations that is only needed in the special case of supplying a toolchain from outside. Ideally I would like the recipe to be agnostic to this detail. As I wrote above, this was not needed with the old generators as the FindXXX files happened to all end up in the build folder. Let me know if I misunderstood.

[memsharded]

Quick question:

it sets CMAKE_FIND_ROOT_PATH among other things, to find packages that are in the SDK and does not come from Conan. Therefore I override the generated toolchain with the tools.cmake.cmaketoolchain:toolchain_file option.

Why don't you use the generated conan_toolchain.cmake and also your own one via the tools.cmake.cmaketoolchain:user_toolchain? That appends your own toolchain, but also uses the Conan one, so that should work?

[rollebolle]

I tried that but get some nasty conflicts - I think in many advanced cases you want to have complete control of the toolchain file, as in our case.

[memsharded]

I have been thinking a bit about this problem, and honestly I don't know yet what would be the best approach.

I think it might be possible to try to find the generated files from your toolchain. This might be done for example by looking for the location of the conandeps_legacy.cmake that will be in the vast majority of cases in a subfolder of the current "build" folder, containing all the other generated files. It might be a bit fragile, because the conandeps_legacy.cmake might be removed in the future, as its name indicates, but it might work. The CMakePresets.json is also generated inside the generators_folder with the other files, and if there is a CMakeLists.txt in the root, the CMakeUserPreset.json file that is generated will have an include to the CMakePresets.json file. Using this, it seems quite feasible to obtain the generators_folder from your own toolchain file.

What would be the best mechanism to communicate the generators_folder, which might be different in different packages (if all use the same cmake_layout(), then it would be direct to deduce the location), to a user toolchain_file otherwise?

Please let me know if this helps.

[rollebolle]

Thanks for taking the time to think about this. Would it make sense for Conan to always append a valid CMAKE_MODULE_PATH to CMakeUserPresets.json, so that even if we use a custom toolchain we can find the dependencies?

I don't really know the details around this, but in Conan 2, a dependency on using the generated toolchains to be able to find any requirement packages was introduced. This in combination with the common cross-compile scenario where we want to use a custom toolchain file certainly makes it a bit more complicated to "get it right".

[memsharded]

Thanks for taking the time to think about this. Would it make sense for Conan to always append a valid CMAKE_MODULE_PATH to CMakeUserPresets.json, so that even if we use a custom toolchain we can find the dependencies?

Yes, this might be something worth exploring.

I don't really know the details around this, but in Conan 2, a dependency on using the generated toolchains to be able to find any requirement packages was introduced. This in combination with the common cross-compile scenario where we want to use a custom toolchain file certainly makes it a bit more complicated to "get it right".

yes, it is true that the toolchain_file conf, fully replacing the toolchain, hasn't been that used so far, it seems the injection of toolchains via include() with user_toolchain conf is way more popular, because there is some important configuration that Conan knows already how to map, that would require an extra effort from users. That is, the mapping from Conan settings to CMake variables, for example to set the VS runtime, to select BUILD_SHARED_LIBS or not, enable/disable fPIC, etc, is a variability that otherwise the user toolchain_file would have to implement, most likely requiring different toolchain_file files, and that is more difficult to maintain that using the include() mechanism of user_toolchain and letting Conan complete the information to adjust to Conan settings.

But in any case, I understand the use case, let me discuss with the team, it sounds that to minimize risks we could add those variables when the toolchain_file is defined. I am assigning this to have a look for Conan 2.5, thanks for the feedback!

[memsharded]

I am exploring the idea in #16455, but it is a draft, I am not very convinced, I might explore other approaches too.

[memsharded]

I have been syncing with the team about #16455 we are still not fully convinced about the approach. We are going to explore the possibility of fully controlling the CMakeToolchain blocks from the outside or inject the toolchain at the end so it has higher priority.