to be or not to be: CMake install() #38
Comments
|
Another con: We can consider separate CMakefile, which is going to be provided as a courtesy and only present some install rules. This CMakefile will not be generated and can get out of sync. |
|
Without the install step you can't use it in SDKs and Linux Distros. |
|
@rogeeff Bazel has a very specific workflow where everything is built inside a single workspace without any installation. This is not the case for almost every other built system and packaging system existing in this world that follow a configure / make / make install pattern. Bazel might be omipresent at Google, but it is not the case (yet) in the open source world, and will probably never be the case some professional environment. If maintainability of Bazel build file is an issue, why not "annotate" the target that are installable in the bazel recipe and generate the CMake Files accordingly ? |
|
Another pro: "workspace build system", based on CMake or not (often not), often rely on the install output. Some tools I know of:
I don't know of such tools that don't depend on install actually. I'm wondering if there any CMake user that does not want the install target and feel strongly about it? |
|
@Sarcasm 👍 Yep, and you can add spack to the list for HPC environments |
|
@Sarcasm Add to that Distros package managers(RPM for example, where it has a "buildroot" and takes the files from there after you run "make install" into it). |
|
I would like to follow the advice to live on head, have only workspaces and source builds, but in reality I have projects where I need go back in time. Also, I need to deliver exactly a known binary package of today software to developers for today development. The way to ensure this is, the same. If I can not do this I can not use a library at work, I am not allowed to. Now, it is not that hard, just annoying, to write an install script yourself, but, And I would like to use abseil, because todays development should be on head, and the description for this lib and the concept behind sound just too good and fits exactly some of my needs. Could I please have it? |
|
From the previous issue, it seems that people assume that make install means precompiled binaries by default. |
|
In the spirit of the documentation - it clearly says "Do not depend on a compiled representation of Abseil. We do not promise any ABI compatibility — we intend for Abseil to be built from source, hopefully from head" |
Please, consider one moment that most persons in this issue understand that.
It's not so black and white:
The build tools I mentioned earlier, #38 (comment), and spack mentioned by @adevress, #38 (comment), all relies on And for the subset I actually used: They all respect this contract:
|
|
Interesting argument @Sarcasm, thank you.
I tend to think about install() as a binary distribution, copy the build artifacts within the file systems accessible from where this build is running. It would perhaps be useful to consider this from a different point of view, could you please elaborate, how you see what install() does? Thank you again. |
|
Interesting quote from tituswinters, @Sarcasm , and true what you write In terms of ABI compatibility , since it is normal to rebuild always everything that depends on top a component when a component is updated, this is not an issue at all. (as long as not the MS way of doing things is chosen to break ABI between Debug/Release builds. Abseil is not planning to do this?) Seriously, in some working environment, when you say every workspace (team) needs to have a source version from Abseil, this is the guarantee for non equal Abseil version used in a product because some team will update today, others tomorrow .... I find it very interesting that there are 2 such different point of views and requirements to reach same target, what is, the same version, hopefully the latest, of a component in a product development environment. Well, its up to the repo owner/developer to decide, I provided a different point of view, everything else could only be repeat my self, so my 2 cents are donated with this and my previous post. ;-) |
@gennadiycivil That's the source of the mis-understanding I think. For many package manager and build system, install() is just an intermediate step in the build pipeline what can be understood as "please, put the public interface of component X of revision Y under path Z for reuse by the next component in the build". For package manager like Nix or Spack, install() of a intermediate component (like abseil) is just a step in the build process before continuing the build for the other component that depend on it. Nix detects the changes in both the revision and the build configuration, and determine a unique, immutable install() path based on a cryptographic hash(recipe, source, dependencies) where the component (abseil) will be installed. This has several advantages:
Spack and some others works the same way. It seems to me that you guys are afraid of install() phase because you suffered in the past of the limitations of classical Linux distro package manager (rpm, dpkg). This is not what many of us rely on. |
|
@adevress It's not an issue with RPM/DPKG per se but more with how Linux distros traditionally work. After each merged commit, the SDK is updated in Git(LFS) and when somebody does A team can basically have just an SDK that is always at HEAD and only the source code of their own project(a team can have their own manifest, the CI needs to import them all with its own manifest, make the build and update the SDK with libraries, headers, RPM database, etc...). |
|
@lilianmoraru I totally agree. The problem is not so much RPM / DEB itself, more how it is traditionally used. |
|
I have tried to produce a "simple" CMake file that ressemble what some of the mentioned tools do. The examples build cctz and its dependencies, Google Mock and Google/benchmark. First create the project: Then build it with CMake: Now, to understand what is going on:
If we take a look at the build layout, in the This is similar to some of the tools I mentioned. Here is the content of the directories,
The build happens in sequence of dependencies:
Also, for each of these projects, the following stages can happen, in the following order:
When a package like cctz depends on other projects, we point it to the earlier projects install tree, You can see this for the This means find_package(benchmark REQUIRED)
find_package(GMock REQUIRED)CMake external projects is not the best example, tools like It also demonstrates that in the current state, without |
|
@Sarcasm This is potentially using different ABIs. If your top level project uses -std=c++14 and wants to propagate it, you will have to pass it manually to each dependency. This version of ExternalProject is broken in that regard, and you should have a look at the GoogleTest documentation for a better usage. Abseil internally changes depending on what is available in the standard library. If you can have different standards used when you compile it and when you use it, you will have issues linking, or worse, difficult to debug crashes at runtime. |
|
It's true that this toy example is not perfect. To be more correct, a toolchain file may be needed, e.g.: set(CMAKE_CXX_FLAGS "-std=c++14")
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)the projects may need to be patched when they don't honor the toolchain, ... Actually, I'm wondering if the question of whether or not having an Without giving it too much thoughts, a configuration could reasonably be made at build time, and installed, to either:
Since Abseil is not a header only, I personally don't see any strong benefits of being configuration-free. I understand Bazel may not need configuration as there is only one possible toolchain and build settings at anytime, or something along these lines. Also, I think this is a goal of modern CMake versions to support this kind of scenario, for example, looking at this recent discussion: Ideally people could specify the minimum standard required in a CMakeLists.txt, but the user can request a specific version from outside. |
|
Just dropping in here (I don't have a stake in this library, but some experience with CMake).. The typical workflow with CMake is to build everything from source. Especially on windows, you need to download all of your dependencies and then build them. Everything uses the same compiler. On OSX, you have homebrew, which uses an ABI-compatible method of building (everything is build with the same compiler, with the same options). I don't think being obstinate about using install is going to fix any problems or make anyone more likely to use People don't read the documentation and then complain. They may file issues when they shouldn't. The way that homebrew deals with that problem is to ban people who don't read the documentation. If you're that concerned about extra work, then you should take the same approach. |
|
In order for RE2 to depend on Abseil without abandoning or alienating our external users, I believe that it will be necessary for Abseil to accommodate traditional use cases in some way. Perhaps the "long-term support" branches could serve as a basis for packaging? And then Debian could handle Abseil much like they handle Boost, I imagine, and likewise for other distributions. |
This is exactly what the maintainers are afraid of: people conflating The problem is that linux distributions have a completely different packaging model than OSX or windows. In the latter two, you just use the latest version of the library, or something close to it. But in the case of linux, the packages are frozen in time for years. One solution could be to only offer |
|
@xoviat Now you've made it very clear why the I thought initially that the main argument against the I understand that the idea is to not leave developers any other option than to directly use abseil(because distros could not "make install" abseil, for example :) ), potentially living at "HEAD". Looking at other open-source projects, the reality is that projects usually pin the submodule or copy the code at one point in time and leave it there for "stability" or because nobody considers it important to have the code up-to-date(even when there are critical security fixes :( ). What I'm trying to say is that this still won't ensure that the projects live at "HEAD"(but does improve the chance of it, because it might exclude support for abseil in Linux distros), it's just that it makes it considerably harder for the SDK, big projects(many smaller components/repos depending on abseil), etc... use-case to happen. I am not sure yet which solution is better or which issue is actually more important(did not yet take time to thoroughly think about this), but I can propose a somewhat weird compromise. I don't remember which programming language had this(I thought Ruby, but can't find it) but there was a very specific problem in the language that you could not solve easily with the usual API, but also you could not apply the obvious "workaround" because it was awful(for performance or something like this). Maybe we can still have the |
https://abseil.io/about/philosophy states:
I understand that Abseil folks don't wish to endorse or even encourage packaging, but considering their promise to maintain such branches, I don't understand why they would obstruct packaging. There will be users of RE2, for example, who won't know or care about Abseil and its philosophy. In many cases, they will be using language bindings, so they won't know or care about C++ problems. In all cases, they will want new versions of RE2 to work just as easily as old versions of RE2. I can't condemn them to using outdated software simply because they had the temerity to use RE2 before Abseil existed. |
|
@xoviat can you please go on about the linux packaging model? I thought apt-get largely worked similarly to homebrew et al. |
|
some difference between Windows and Linux packaging: on Windows MSVC break ABI from version to version and even between debug and release build. This is why in Windows project often are huge monoliths, with external dependency in a subdirectory, 3rdParty or so. On Linux ABI does not break that often, I think I had it only once, some when during gcc 3.x on RHEL you use for example software collections to get a new gcc-63 or whatever, the latest python or ruby, ..... (I totally understand why abseil would break in such a scenario, I have no understanding why a lib like google test should not be shipped via a package, but this just as a site note) Of course RHEL, Suse, or debian do not update basic packages to the latest version, they add latest version in parallel, this is for a reason. So there is no one story about package managers on Linux. On Windows such a scenarios, any of them, are impossible with MSVC, therefore you need those monolithic workspaces with insane build times and everything included. I have no idea about Mac. |
No, it doesn't. Because In contrast, homebrew usually stays a few months behind HEAD at most. Specifically, let's compare the effort to update a package in both package managers.
|
|
To me, it feels like you want the compiler to do its job correctly, that what it compiles and what it uses is not just compatible, but really the same. Another funny thing, is to promote "live at head", but require ancient time CMake (2.8.12). You want to force people to have a coherent/correct workspace built with CMake, but CMake is not good to deal with multiple dependencies.
Real tools for dependency management, rely on Personnally, I'm thinking that "Configure-time configuration" is possible for Abseil, and it would make it easier to integrate while still being correct. I hope the tone is not too harsh. |
|
@Sarcasm the only part I disagree with is the way end. CMake does have inherent support for building external projects. That is the basis of one of my favorite projects Hunter (https://github.com/ruslo/hunter). I really believe Hunter is exactly what this project wants, but for some reason or another the motive to keep using classic CMake is too powerful. For the record, Hunter uses backwards compatible extensions so the install works even without downloading. I highly recommend building something with it. I guarantee you'll like it. |
|
@Sarcasm , re: CMake 2.8.12 - good probability that Abseil will require much newer CMake soon. |
|
@Sarcasm I'm a little confused at your response. I'm basically proposing doing https://github.com/google/googletest/blob/master/googletest/CMakeLists.txt#L114 for Abseil, allowing it to be used in the same way as googletest. I'm not sure what I'm missing. |
|
I hate to use reddit as a source, but this is a pretty good summary of the issue: "I've used both pretty extensively at work, and use Bazel for my hobby projects. They're great to work with, so long as you buy into the monorepo pitch. Hermetic, reproducible builds... https://www.reddit.com/r/cpp/comments/6euc7b/_/die6g1y?context=1000 |
|
The GoogleTest CMake snippet you linked to, installs the binary targets: cxx_library(gtest "${cxx_strict}" src/gtest-all.cc)
cxx_library(gtest_main "${cxx_strict}" src/gtest_main.cc)
target_link_libraries(gtest_main gtest)
...
install(TARGETS gtest gtest_main
RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}"
ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}"
LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}")
install(DIRECTORY "${gtest_SOURCE_DIR}/include/gtest"
DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}")If that's what you propose, then I'm all for it, and I'm sorry for the misunderstanding. Sadly, I don't think that's what you proposed by rereading this (emphasis mine):
So, if your actual plan, is to do like GoogleTest except that you don't install the binary targets, then I maintain my comment, and additionally the comments I made here, against |
|
I'm wondering if there isn't a miscommunication here, after talking with some other people internally. Is the concern that you'd like to do something like |
|
While I do not expect the Abseil developers to provide packages for every platform out there ( Otherwise each package maintainer has to reimplement the |
|
I firmly believe the clean way to use packages with CMake is to use I'd like the Sure, these tools can be misused, but they can also be used perfectly well, and when a package like Abseil just add difficulties because of "fear of misuse", it's seems wrong. The CMake documentation documents how dependencies are to be used: I don't want the Abseil team to teach me how I should use CMake in a different way than recommended by the CMake documentation itself. Either you adopt the build system and its quirks, or you give up and just say "use Bazel". Adopting CMake and propagating the unconventional usage is not doing any good IMHO. Disclaimer: I understand the complications of system wide installation, the ABI issues you can have as a system installation cannot guarantee every use of the library will use the correct settings. However, for projects at my companies, we don't do system wide installations, we have "workspaces" where we build the library, and make sure we have a set of correct build settings. We rely on autotools projects, CMake projects, Boost.Build, ...All these tools provide installation targets and we rely on them. |
|
Okay great, I think I understand the issues here. Let me make my goals clear from my end, and hopefully we can move forward with this: What I'm aiming to do right now is support live-at-head use of Abseil with CMake. This necessarily involves the configuration-time dance that googletest does, or else users have a superbuild and use ExternalPRoject In the near future, abseil will start to cut long-term support branches about every 6 months, likely with versioned inline namespaces. I and other members of the team are pushing that we should work with package manager maintainers such that all of their packages can, as much as possible, use the same long term support version of Abseil. Now to your point, @Sarcasm
I follow you here, but allow me to ask a stupid question -- how are you installing Abseil in this situation without a package manager?
For headers, we may be able to cobble together some kind of "includes" subdirectory of an install destination. What we probably can't do is actually structure abseil's code in this way -- it would be way too much work to restructure our code at this point. As for libraries, abseil is actively shedding our cctz dependency, at which point we will have no non-test dependencies. This seems to me like then there are no libraries to install. I have to admit I don't know what pkg-config is, I'll have to research that. For CMake config files, what I got out of our training with Kitware is largely that absl/base/config.h does what would otherwise be done with a cmake config file, unless I'm misunderstanding you. Again, apologies if I'm being super dense here. I'm coming at this with the best intentions and learning on the fly. |
|
Thank you @JonathanDCohen for your dedication here. First, I'm not sure how A CMake config (and pkg-config for that matter) solve a pretty basic problem. If I have library B that depends on library A, how do I determine the necessary header directories, libraries, and flags to compile with? CMake developed two solutions to the problem.
The problem with the first solution should be clear. The package requirements should be dictated by the package itself. The burden of determining required flags should not be placed on the dependent library. Unfortunately, most CMake projects to this day still rely almost entirely on this solution. The second solution is much more sane, and is compounded by the fact that CMake can do all the work for you. If library A has a CMake build, CMake can easily create a config for consumption by library B (see https://cmake.org/cmake/help/v3.1/module/CMakePackageConfigHelpers.html for details). The primary issue here is the inability to apply this to non-CMake projects. Now that we have that sorted out, you can see why I strongly suggest you take the second, more sensible approach. |
|
Thank you for clarifying the short-term goals. You may want to open a separate bug or something called "Partial improvements to CMake support" or show folks a roadmap (without dates) where you explain how this fits into the plan to have releases with versioned namespaces and so forth.
Agree,
I was skipping steps and thinking about your longer term goals. In that case you do create |
In a way - thats the whole point why we nag about an install target. Normally you would do this: # thats your workspace with all your libraries in whatever version you want.
MYBUILD_DIR=<someplace>
# git clone / unpack your sources to /tmp/abseil/src
cd /tmp/abseil/build
cmake -DCMAKE_PREFIX_PATH=$MYBUILD_DIR ../src
make
DESTDIR=$MYBUILD_DIR make install
# repeat for all dependencies
cd <my_project>/build
cmake -DCMAKE_PREFIX_PATH=$MYBUILD_DIR ../src
make
DESTDIR=/tmp/my_project_package make install
# now I can pack up /tmp/my_project_package, and unpack this to its final destinationit should not matter whether you use a package manager or build from source, atleast for your project files. I don't know how the part before can be solved easily in an automated foolproof flexible configuration (I am leaning toward an extra CMakeList.txt thats independent from the my_project build). but the key point is that you have an directory that's your build environment and you dont want to leak out details from individual builds, and you dont want to burden the users of your library by having to manually extract all files and configuration. You do so by providing an install target which can be redirected (CMake mostly does this for you, as do Automake and everything else). Or you could use some Projects like Buildroot / Yocto to create this build environment for you. At any rate you will need a way to install your libs/headers/source and a way to use them in depended projects. |
If I want to try a new library, like Abseil, I proceed as follow: First, I have an existing workspace with just my app: I will first get the Abseil cctz dependency: Then Abseil (notice how this could be the same steps as cctz, (1), (2) and (3) are all the reasons why departing from standard CMake is really painful, (2) the following patch was sufficient: +find_package(cctz REQUIRED)
+set(ABSL_CCTZ_TARGET cctz::cctz)
- if(NOT ABSL_CCTZ_TARGET)
- set(ABSL_CCTZ_TARGET cctz)
- endif()Now I have the following workspace: And to test it in Edit my_app/CMakeLists.txt: +find_package(absl REQUIRED)
+target_link_libraries(my_app absl::base)Reconfigure CMake to specify where to find the dependencies: What's really important here is that you build on existing knowledge/practices. You can see this is already a bit painful this way, not providing the install targets makes it even more so. As @nolange says, tools like Buildroot/Yocto really works with this kind of workflow.
Yes, please read about it.
For me a CMake config file is xxx-config.cmake, is not the same as absl/base/config.
If you can, you should say hi to the |
|
Could the FindFoo.cmake or foo-config.cmake be used to do a |
|
@Quincunx271 I do not believe so. Those files are always used to find precompiled content, not source files. |
|
downloading anything during the configure stage is a anti-pattern to me, but no one forces the libraries that are found with There are object libraries in CMake which could be used for that. |
|
Well as I've said before, if that is what you are looking for, use ExternalProject or Hunter (https://github.com/ruslo/hunter). |
|
This is now tracked in #111 |
|
Finally. I can attempt now to push Abseil at work :) |
Issue following the discussions on #8 : should we support install() in the cmake build:
Pro:
Cons:
The text was updated successfully, but these errors were encountered: