In-source vs. out-of-source recipes

[theodelrieu]

Hello,

There are two kinds of recipes, those which are embedded in a project, and those which belong in their own repository. I'll call the former in-source recipes and the latter out-of-source recipes.

I've been having problems with in-sources recipes lately, so I wanted to share my thoughts about them.

Out-of-source recipes

Those are the most commonly found, especially in OSS (e.g. bincrafters), and they have some strong advantages.

Every release can be packaged

It is possible to go all the way back in a project's release history, and package each release without interfering with upstream at all.

Recipe bug-fixes do not require a new upstream's release

Instead, you have to push a new recipe revision, which has no impact on upstream.

In-source recipes

The biggest advantage of in-source recipes is the possibility to develop the project and package it at the same time, which can be time saving especially when dealing with CI. Indeed, one can build, test, and package at the same time.

However, it has a big drawback:

Recipes bug-fixes require a new release

When the packaging files are part of the project, releasing a recipe bug-fix is no longer trivial.
The buggy recipe is tagged alongside the rest of the project, thus being carved into immaterial stone. Even if it has no impact on the real code!

So what can you do? Make a new release only containing the recipe fix?
It works, but it's a lot of noise for the users (not even mentioning those who do not use Conan), even more when there are more discovered recipe bugs.

Unfortunately, that's what we're doing at work at the moment.

Possible solution

I really think development and packaging should be separate processes, for the reasons I've briefly mentioned above.

The first thing I thought about was to have two recipes. An in-source one for development, used to resolve deps and build easily (i.e. local workflow), and an out-of-source one for packaging.

Unfortunately, this duplicates quite a lof of information (e.g. about requirements), which increases the risk of having mismatches between both.

And it gets worse when a project uses workspaces, because you end up with N in-source recipes...

Does someone have a magical solution for this? 😄

[jgsogo]

(My thoughts about it, I don't have an answer)

This is a major concern, for sure.

An in-source recipe helps the author with the CI, not only packaging in the same run, but also helps to configure and run different platform configurations, better with Conan than with CI scripts. This is something very useful for library writers, also for open source ones.
But it has the big drawback you mentioned, the release cycle of the Conan recipe is not the same as the library one, so unless you consider Conan packaging something needed in your release, then having to fix anything in the recipe could require a new library version and it is annoying.

I don't have any magical solution for this, the only one idea that comes to my mind is to have external recipes, make it easy to grab the corresponding recipe in the CI and build the local sources instead of using the source() method to retrieve the sources...

I feel like it was the same process when some library is moving to CMake from other build system, if it is the only build system then the realese will include proper CMake implementation, if not, any fix to the CMakeLists.txt would require a new release. Is Conan in the same path? Does CMake faced this issue before?

Please, anyone share your thoughs. Thanks!

[solvingj]

I have thought about this a lot. I will try to post more later, but one thing that has come to mind as an idea (although it may be completely flawed and not work), is having a pattern where both exist, and supporting composition to eliminate the redundancy.

So, there are quite a few OSS projects that just want to have a conanfile.txt in the root of their repository which can optionally provide the dependencies of the project. They don't want it to be required, and they don't want to be in the business of packaging their own project for Conan Center. An example of this is the stlab/libraries project maintained largely by sean-parent:
https://github.com/stlab/libraries

In this case, as a community packager, I would like to package the library without losing the information in conanfile.txt. This all feels like a reasonable situation to think about.

Obviously, the high-level arrangement would be:

conanfile.py checks for conanfile.txt
--Perhaps this happens after source()? Probably won't work.
--Perhaps the conanfile.txt needs to be obtained extremely early, like in the around when python_requires takes place.

conanfile.py adopts all the attributes of conanfile.txt and somehow merges them with values defined in conanfile.py. There would obviously be a LOT of questions here around precedence and such.

Again, at a high level, this would be a hybrid approach we could think about. It may very well be impossible or infeasible to implement, I'm not sure.

[jgsogo]

That is something that crossed through my mind too: conanfile.txt inside the repo and conanfile.py outside, and compose them... but then I thought about recipes that could be using the source() method to retrieve data from somewhere else, would it be a fair use-case or it is abusing Conan? should that additional-sources/data be packaged as a dependency?

I'm not scared about implementation, if the idea works and make sense.

[theodelrieu]

This is still embryonary in my head, but I feel that a clear separation of concerns between building/CI and packaging would be really helpful.
To achieve this, it might be required to rethink the conanfiles model.

Right now, the conanfile.txt is simpler to write than a conanfile.py but is also much less flexible. I'm not convinced of its usefulness, apart from being a good "quick start" method. I've always transitioned to a conanfile.py in the end, but YMMV.

The conanfile.py is a two-headed beast, and it might be useful to split it between the in-source kind and out-of-source kind, e.g. "build helper" recipe vs. "packaging" recipe. If we manage to specify those two concepts well, we might design a good solution to the OP issue.

It might add complexity for packagers in the end, but having a clear cut separation would avoid conanfiles to become a huge monolith/mush of complex things. They already are quite complicated and have a lot of features.

@jgsogo

An in-source recipe helps the author with the CI, not only packaging in the same run, but also helps to configure and run different platform configurations, better with Conan than with CI scripts.

True, this is mainly the "build helper" aspect of Conan at work here, i.e. dependency installer, and build system wrapper.

I feel like it was the same process when some library is moving to CMake from other build system, if it is the only build system then the realese will include proper CMake implementation, if not, any fix to the CMakeLists.txt would require a new release.

If we agree that a project is minimally composed of its code and a build system responsible to build it properly, then I'd argue that an bug fix in the build system files should indeed result in a new release.

[solvingj]

It's an interesting idea. I have also noticed that conanfile.py is dual-purpose. It gets deps for the current build, and packages the artifacts for the downstreams. While we're currently feeling some motivation to handle these two things separately, but in a composeable way, it's worth pointing out that this is probably how most package managers work. I think it's also interesting to point out that 95% of the recipe is automating the BUILD of the library or executable. I would say the package...() methods represent about 5% of the complexity and logic in the recipe. I don't know if that's useful.

[theodelrieu]

it's worth pointing out that this is probably how most package managers work

It would be interesting to look at how other PMs handle this issue (if they do).

[solvingj]

Maven/Nuget/Pypi/NPM . In all cases I am aware of, you list your dependencies and define your package in the same specification. C/C++ is the rare case in which project maintainers have to leave the door open to having dependencies satisfied in a wide variety of ways, and not forcing users to get them from a single package manager.

OS Package managers (DEB/RPM/Brew) are an exception here (despite being C/C++), because they do effectively force all the deps to come from a single PM. However, that is just a very different situation.

[elizagamedev]

I have a bit of a radical, kind of idealist suggestion for this sort of feature, based on my own experience working with Conan in production at my company.

First, the existing conanfile.py/conanfile.txt mechanism would remain as-is and could continue to be used for packages which are not "native" to Conan, e.g. zlib, openssl, etc. They could also be used for "native" Conan packages, but this usage should be discouraged.

Instead, native packages would have something like "conan-package.yml" (or .json) that would be somewhat reminiscent of npm's package.json. This would enforce some constants: namely, it would require building with CMake and an accompanying CMakeLists.txt with a standard header similar to the include(${CMAKE_BINARY_DIR}...) one. The generated include file in this case would provide macros, functions, and variables specific to native Conan packages. Binaries, PDB files, etc would be generated in a standard, well-documented way. This allows for an incredibly easy cycle of installing dependencies, building, testing, and eventually packaging, and should reduce a lot of the headache of C++ development, especially once debugging features are more fleshed out in Conan.

Given that CMake can provide generators for many different build systems, this would provide a stable, but not too-restrictive standard for all serious C++ development in the future and would make it possible to migrate a wide variety of projects over to become "native" packages. As part of my professional work I helped migrate a very old C++ codebase over to Conan/CMake already, and it's helped enormously. Maybe someday, almost every C/C++ codebase could be structured this way...

[theodelrieu]

What you describle looks more related to what SG15 is discussing (i.e. P1177r0, P1178r0, P1204r0, P1254r0 and P1313r0).

I agree that having a somewhat minimal package specification would be very profitable. Unfortunately this will take years, and I still do not see how it would solve the OP issue, i.e. having the packaging files in-source or out-of-source.

[Artalus]

I've had this idea for some time, but never got to try it out on a real project, though. Thus I might be missing some points already known to veteran packagers.

An in-source package can be published from develop or master branch into wip or ci or whatever channel by CPT via some CI. Maybe even not published, CPT can just package it to see if everything works and don't upload it anywhere.

When the author feels it's a good time to release, he doesn't tag the commit, but creates a release/xxx branch. CPT detects that the branch name matches stable regex and publishes the package into stable channel.

If the author finds some bug in the recipe, he pushes a commit to master and cherry-picks it into release/xxx (and if the bug's been hiding for too long, into release/yyy et.cetera). If there is a bug that should be immediately released as a hotfix, he branches from release/xxx to release/xxy and cherrypicks relevant commits there.

It seems to me that both CPT features and git-flow encourage this sort of release "cycle".

[solvingj]

I agree, the suggestion here and those proposals are a bit orthogonal and long-tailed. They do relate a bit, but my current focus is the versioning questions raised here.

Regarding the versioning questions:

From my perspective, the process of packaging a library, including the long-term maintenance and revision of packaging logic, is orthogonal to the API/ABI of the version of a library. It is fundamentally flawed to try to version the two together.

Historically, for OSS packages, we have co-opted the LIBRARY version to describe the PACKAGE version for Conan packages. I think this is the right choice for now and the foreseeable future. We do this because anything more robust would appear complicated and confusing to the community, especially for a fairly young tool like Conan. The story needed to remain simple, and still does to some degree. For these reasons, we want the most common and simple cases and users to be able to work while only being aware of ONE VERSION NUMBER.

This is the case now, and it is satisfactory in terms of SIMPLICITY and USABILITY, which is the high priority to the broad user community. At the same time, it is utterly insufficient in terms of DETERMINISM and REPRODUCIBILITY, which are high priorities to the more advanced and professional user community. The challenge for Conan is satisfying the priorities of both sets of user communities and use cases. I think the package revision system in the latest version of Conan is the first of it's kind, and should be hailed as a miracle if it indeed satisfies both user communities. I am optimistic, but honestly have not had time to fully test it out in either OSS or enterprise setting.

Still, I think it's important for anyone interested in this topic to really understand the fundamental problem in detail. That is, trying to use one variable (library version) to represent the sum of two separate states (library version and package version), without any variable for tracking of the second state separately (package revision). The only way a single version could do the job is to create the separate variable for tracking package revision, and then create some new abstract "consumer version" which represented the sum two lower-level states , which incremented any time either of the other two did. Indeed, this would be convoluted and highly undesirable for OSS packaging. Again, this whole problem domain exists uniquely for native binaries (C/C++) because the contents of a binary package are nearly infinite. There are infinite number of ways and so the recipe is FAR more likely to evolve independently of the library. For these reasons, in order to satisfy requirements of DETERMINISM and REPRODUCIBILITY for the cases that need it, a separate version tracking system is required for the package/recipe. And, as stated earlier, I think Conan's new revision system implicitly handling this so that we don't have to complicate the simple cases un-necessarily is a brilliant design.

Of note, there's one community to think about which has the same problem, but on a much grander scale. That's docker container community. There, you have a "tag" system which represents the sum total of states of an infinite number of different components. This system is anything but simple, has been the topic of much discussion and debate since it's inception. In short, it's certainly not a solved problem, so I hope people do acknowledge that there's inherent complexity here that can't be reduced.