Cmake release targets #438

Ingvord · 2018-02-27T11:48:27Z

make package - wrap cpack
make deploy - wrap curl
~~make release aka increment version and git tag + make clean & make & make package & make deploy~~

Ingvord · 2018-02-27T15:12:13Z

As far as I understand there are only 3 pre-defined components in debian repository layout: main, contrib and non-free. But this is relevant only for official repo. We in our case may choose any arbitrary value e.g. tango as we deploy to bintray.

@picca could you please comment on this?

t-b · 2018-02-27T16:55:48Z

@Ingvord I think you are free to choose whatever you like for the names. But I would just use one of the official ones.

Ingvord · 2018-02-27T16:58:23Z

@t-b , thanks for the input! It is done so in b98f1fe ;)

bourtemb · 2018-03-01T13:25:08Z

We need to find a way to pass the Tango library version to appveyor without having to modify the appveyor file at each new version.
We could set an environment variable to appveyor using Set-AppveyorBuildVariable . We still need to get the Tango library version number from somewhere.
We could also customize the build version using update_AppveyorBuild.

Ingvord · 2018-03-02T09:11:39Z

@bourtemb , Unfortunately cmake can set env properties only for its own process [1], so I can only imaging to extract version into a separate file and then do

$> source ./VERSION
$> cmake

Not sure if it will help with AppVeyor though, @NexeyaSGara could you please comment?

Anyway extracting VERSION seems to be a good idea - it will be much easier than to automatically increment it...

NexeyaSGara · 2018-03-02T09:31:36Z

I think we could possibily read a file and then set the environment variable with the content.
Maybe with classic cmd or with a powershell script.

t-b · 2018-03-02T12:27:08Z

I would:

Createe a header file with the version number triplett
write a tiny c++ programm which outputs the version numbers
build and execute that with cmake via

TRY_RUN(VERSION_RUN_RESULT VERSION_COMPILE_RESULT
  ${CMAKE_SOURCE_DIR}/cmake ${CMAKE_SOURCE_DIR}/cmake/getVersion.c
  CMAKE_FLAGS "-DINCLUDE_DIRECTORIES=${CMAKE_SOURCE_DIR}"
  RUN_OUTPUT_VARIABLE_VERSION)

IF(NOT VERSION_COMPILE_RESULT)
  MESSAGE(FATAL_ERROR "Could not compile version program")
ELSEIF(VERSION_RUN_RESULT)
  MESSAGE(FATAL_ERROR "Could not run version program")
ENDIF()

and then you have the version numbers in cmake. This is CI agnostic and works also locally.

Ingvord · 2018-03-26T10:19:38Z

As for me this one is ready to be merged. I skipped the third point because as for now one needs to update version, tag repository and push it to perform a release (travis will deploy everything). So there is no need to over complicate cmake and anyway it is better to control release manually, for instance one could also check that RELEASE_NOTES are updated accordingly etc

picca · 2018-03-26T11:54:19Z

liblog4tango is now merged with libtango. The database is not part of the Debian packages generated by this Pull Request.

In that case, you should also conflict with liblog4tango, -dev

What is the impact of declaring a package as conflicting with another one? Does this mean that the user must remove the conflicting package(s) in order to install the new package?

Inspire yourself from this https://wiki.debian.org/PackageTransition

I've seen there are packages named libtango7-dev in wheezy and libtango8-dev in wheezy and jessie so I think Igor is right to add these packages in the conflicting packages list.

Yes no problem, I just ask to add libtango-dev ;)

bourtemb · 2018-03-26T11:55:48Z

configure/cpack_linux.cmake

+elseif(CMAKE_BUILD_TYPE STREQUAL "RELEASE")
+    set(CPACK_PACKAGE_NAME libtango9lts)
+    set(CPACK_DEBIAN_PACKAGE_DEPENDS "libcos4, libzmq5 | libzmq3")
+    set(CPACK_DEBIAN_PACKAGE_CONFLICTS "libtango7, libtango8, libtango9")


In release mode, it looks like there is no conflict with libtango7 and libtango8 since the library files installed by these packages have different names (E.g.: /usr/lib/x86_64-linux-gnu/libtango.so.8. See https://packages.debian.org/jessie/amd64/libtango8/filelist).
This would not be true for the development packages since there might be a conflict with the installed include files(?), so in DEBUG CMake mode, I would keep the conflicts with libtango7-dev and libtango8-dev.

bourtemb

I played a bit with the Stretch Debian packages which are currently on BinTray.
I have seen they do not install the include files and library in the same directories as the official Tango Debian packages.
Includes are installed under /usr/local/include instead of /usr/include/tango for the official package.
Libraries are installed under /usr/local/lib instead of /usr/lib/x86_64-linux-gnu directory.
So they actually do not fully conflict with each other this way... libtango9lts is getting higher priority in the default library path. The question is: do we keep it like that or do we install the library and include files at the same locations as the official Debian packages?
I also noticed that there does not seem to be big differences between libtango9lts and libtango9lts-dev.
The only difference I noticed being that libtango9lts-dev is installing the library compiled in DEBUG mode instead of RELEASE mode (so, with debugging symbols).
I was expecting libtango9-lts package to contain only the library and not the include files as it is currently done for the Debian libtango9 official package.
I would have expected include files to be only in -dev package.
There is also a conflict between libtango9lts and libtango9lts-dev packages because they are trying to install the same files.
I think the libtango9lts and libtango9lts-dev packages are actually 2 development packages, so I would rename them in libtango9lts-dev and libtango9lts-dev-dbg for instance.
Another possibility would be to install both the release and debug versions of the libraries in the same package. We would need to agree on where to install the debug version (in another directory, named debug for instance?).
The Tango developer could then use one version or the other by setting the appropriate LD_LIBRARY_PATH for instance.
These are just suggestions from a modest Debian user with very limited experience with Debian packages ;-)...

Ingvord · 2018-03-29T14:08:38Z

@bourtemb , thanks a lot for testing this PR. I have changed package names accordingly in c47f039

I think it is OK if we leave -dev and -dbg suffixes and install headers in both cases thus stress the difference between these packages and official ones (maybe we can add really release package at some point in the future)

Not sure if we need to put headers in tango subfolder. If apt-get remove libtango9lts-dev[-dbg] removes them than leave things as they are is OK for me...

bourtemb · 2018-03-29T15:47:06Z

I think it is OK if we leave -dev and -dbg suffixes and install headers in both cases thus stress the difference between these packages and official ones (maybe we can add really release package at some point in the future)

Fine with me but we need to update again the conflicts because the new -dev version (Result of RELEASE build) will be in conflict with all the libtango*-dev and liblog4tango-dev packages as well since it is installing the include files.
And by default, POGO is generating Makefiles looking first in $(TANGO_HOME)/include/tango (/usr/include/tango by default) and then in $(OMNI_HOME)/include (/usr/include by default) and finally in $(ZMQ_HOME)/include (/usr/local/include by default) so I think we should avoid this situation where the include files are installed from different packages. Adding the proper potential conflicts should be enough.

Not sure if we need to put headers in tango subfolder.

I know a guy who may have an opinion about that ;-) @chedburgh, would you like to comment on that?
I think one issue with that is the default option in POGO (coming from libtango-java Debian package available here: https://people.debian.org/~picca) which seems to be expecting the Tango include files inside $(TANGO_HOME)/include/tango directory (/usr/include/tango by default if TANGO_HOME env. variable is not set).
This is what we can find in /usr/share/pogo/preferences/tango.opt file deployed with libtango-java Debian package for instance. Of course this tango.opt could be changed (we are using a different file at the ESRF for instance, matching better our installation paths where the include files are not under a directory named tango).

Actually, I was more worried about the potential issues which could arise if a user manages to install both libtango9 and libtango9lts-dev or libtango9lts-dbg.
In the current situation it seems this would lead to a situation where a user using a POGO generated device server with default options would compile using the Tango 9.2.5 headers and would link with libtango9.2.8.so by default.
If we define correctly the conflicts, this should not happen.

Ingvord · 2018-04-03T14:25:32Z

@bourtemb, @chedburgh , I have found this in our previous emails (from October 2017):

Currently the Debian package (and source install) installs all header files to /usr/include/tango. Igor has indicated for the next revision of libtango, the model will change (with tango.h under /usr/include, other files under /usr/include/tango).

So I have changed the default installation path in 3c37bba to {prefix}/tango to be consistent with the present packages

We can change it to tango9lts if you think it is necessary. Anyway if you have no other changes requests I would merge this one and then #423 to deliver the events patch ASAP

chedburgh · 2018-04-05T07:42:41Z

Sorry, I did not get a notification when I was mentioned above. Catching up now

OK, so, IMHO… (Sorry if I comment on something resolved, this is a long thread, and sorry if I stir up any controversy…)

Regarding install locations. Headers should install to /usr/include/tango, and looking at the CMake files, it looks like the binaries install correctly to /usr/lib/x86_64-linux-gnu

I think you have this correct in the patch? Local should NOT be in the path.

Do not put the headers somewhere unusual (i.e. usr/include/tango-lts). You will regret having to support some unusual install location for headers at a later point. If a user wants to run multiple development installs, let them set it up themselves with source installs. If the user wants to install the headers somewhere with a version number, let them get the source and do it themselves. Do not over complicate the packages and install location for what power user should be able to manage themselves.

Packages

The development package should install the headers and anything else required for development. The binary package should NOT install headers, just the finished product (you do not compile against this package, you compile against the development package). A dbg package installs symbols etc. See A1 here

Regarding the package names.

In my opinion, I would simply have libtango9.deb and libtango9-dev.deb etc. They can just be made to conflict with the official debian packages as pica pointed out, and their version information will make them distinct from the official packages (i.e they will be versioned 9.2.8-pre-release-1 or something?). Unless you plan for the official packages to be called libtango9lts in future? Maintaining differently named packages can create confusion.

Question, when a new official lts debian package release comes out, can they be made to conflict with these snapshot lts releases ok if they have a difference a package name?

Header Include Paths

Since we are talking about header paths....

These are could be improved. For example, tango.h includes tango_const.h with no path. It should look like this:

#include <tango/tango_const.h>

And device servers should include

#include <tango/tango.h>

This is a standard way of stopping filenames clashing. You are relying on gcc searching system includes last to make sure it picks up tango headers in the case where they may have the same name as a system header (or another badly installed development package). I understand how POGO is set up it works around this, but it is just a work around. Check other package headers, or system stuff, i.e. /usr/include/linux/raw.h, you will see they use the path.

It would be great to move this to the standard way at some point.

Why is this important? When you build against the debian package (or any system with a /usr/include header install) without a pogo build file, you have to supply additional paths that should not be nessacessy. See the debian rules for libhdbpp-cassandra here, I have to include the path /usr/include/tango so it will compile.. This is not obvious for new users.

Edit: Corrected mistake libtango-dev.deb -> libtango9-dev.deb

Ingvord · 2018-04-06T17:55:07Z

@chedburgh , Wow! Thanks for a such exhaustive comment!!!

Honestly, I think we are over complicating things here...

First of all include headers are controlled by pkg-config, so we can change it in the future. And I totally agree that default location must be preferred. I guess this is what we get from ${CMAKE_INSTALL_FULL_INCLUDEDIR}

There is not so many users of these packages at the moment (even I do not use it... mainly because all of our Tango servers are in Java, i.e. maven). So it is not that critical to change rules afterwards. Do you use it for hdb++?

And finally I firmly believe we do not decide rock solid solution here. This PR is just a starting point for continuous delivery. Everything can be altered next day if you want so :)

chedburgh · 2018-04-09T07:52:19Z

Yes, sorry, I have a lot of opinions on release procedures (from bad experiences in the past).

I agree on the headers, it is a separate change, but I wanted to bring it up, since it is a bit messy having to work around this, and it would improve the projects general acceptance if it included headers in a regular way.

The hdb++ debian packages depend on the libtango debian package. I use docker to do much of my work with tango, and often I use the debian packages inside it.

I looked at the CMake files. It appears you not building proper debian packages, instead a debug/release package? Are the headers in both? I can not tell what is in either. Are you intending to release these to testers?

If possible, I would suggest using traditional debian packages, and I would name them in the same way, so a future debian packages can conflict against them.

libtango9 - just the binary shared libraries
libtango-dev - the headers and other development files
libtango9-dbg - the debug symbols only

You just need to version these correctly, i.e. 9.2.8a-build-45 etc, or similar.

Why? Well, for example, I use docker images for compiling tango projects, I could easily run packages with the same name (simple script call for me). I could run my hdb++ testing images against these pre releases libraries, if they are named correctly (I do not really want development files for release images, or to support multiple package names for libtango).

Just some suggestions! I can probably offer some CMake pointers if you decide to do any of this.

Ingvord · 2018-04-09T08:28:56Z

@chedburgh

libtango9 - just the binary shared libraries
libtango-dev - the headers and other development files
libtango9-dbg - the debug symbols only

This means we need to produce 3 packages aka artifacts from a single repo... that is something conterintuitve for me (my problem maybe). It is possible to do it in CMake?

I totally agree - it is better to stay with standard way of doing things. I am just not sure I can do it alone.

picca · 2018-04-09T08:36:04Z

The hdb++ debian packages depend on the libtango debian package. I use docker to do much of my work with tango, and often I use the debian packages inside it.

Do you provide these packages and do you think that we should integrate them into Debian ? If you have other packages do not hesitate to upload them into Debain mentors and I can help do the review :)

libtango9 - just the binary shared libraries libtango-dev - the headers and other development files libtango9-dbg - the debug symbols only

Now the -dbg packages are obsolet, and we generate automatically dbgsym during the debian build process. BUT, I do not know if cpack use the standard debian tools in order to generates the Debain packages.

You just need to version these correctly, i.e. 9.2.8a-build-45 etc, or similar.

I use most of the time this dch --bpo "" dch -l soleil 'Rebuilt for SOLEIL'" In order to generate the right changelog and version of my local packges. Cheers

chedburgh · 2018-04-09T08:38:28Z

This means we need to produce 3 packages aka artifacts from a single repo... that is something conterintuitve for me (my problem maybe). It is possible to do it in CMake?

Yes, I believe you define components on the install rules, then use these components in the packaging rules. Let me see if I can come up with a quick example today.

picca · 2018-04-09T08:38:42Z

In that case what about integrating the debian directory in the tango sources. then using a small shell script in order to generate the Debian package with the usual DEbian tools. dpkg-buildpackage. Maybe this is possible with cpack. (I do not know about this tool) Cheers

chedburgh · 2018-04-09T08:44:32Z

Do you provide these packages and do you think that we should integrate them into Debian ?
If you have other packages do not hesitate to upload them into Debain mentors and I can help do the review :)

Not yet, it has been my intention to submit them after I have had time to ensure the source package is created correctly, they are just binary packages hosted on bintray currently.

BUT, I do not know if cpack use the standard debian tools in order to generates the Debain packages.

I have not seen reference to CPack creating these files when I did some searching last week.

chedburgh · 2018-04-09T08:46:41Z

In that case what about integrating the debian directory in the tango sources.

then using a small shell script in order to generate the Debian package with the usual DEbian tools.

This would be the best solution. Use the full debian build process and files and just revision it correctly.

picca · 2018-04-09T09:43:04Z

Not yet, it has been my intention to submit them after I have had time to ensure the source package is created correctly, they are just binary packages hosted on bintray currently.

Great, it shold be nice to have this for Buster.

bourtemb · 2018-04-09T14:55:47Z

So, if I understood correctly, what we would need is a recipe to build libtango standard Debian packages with the Debian tools.
Could you provide this Frédéric, please?

picca · 2018-04-10T07:41:20Z

So, if I understood correctly, what we would need is a recipe to build libtango standard Debian packages with the Debian tools. Could you provide this Frédéric, please?

If you have the debian directory in the sources, you can do this gbp-buildpackage What do you want in fact. A wait to geenrate packages on each commit ? In that case you need to use the dch command in order to generate the right changelog numbers. It is nice also to follow the dep14 instruction in order to manage the debian packaging. https://lwn.net/Articles/620197/ Cheers Fred

bourtemb · 2018-04-27T08:44:23Z

As I wrote in an e-mail involving some of the persons who commented on this ticket, it seems clear that making the things clean to build high quality Debian packages will require more time.
The original intention behind #438 was to generate and deploy on Bintray automatically unstable libtango development Debian packages every time we tag a version, to ease the life of adventurous people who are interested in testing the very latest unstable version of the Tango C++ library.
This Pull Request is also fixing a bug related to appveyor and changing the default include install path to match what was done by tango 9.2.5a source distribution.

So, for the moment, I would be in favour of accepting this pull request but we need to make it clear somehow in the web pages where we advertise the Debian packages (if such web pages exist) that this is an experimental development version. On Bintray, the Maturity is set to "Development" for this package, so hopefully, this is clear for the users.

We can create nicer Debian packages (more in line with the official Debian packages) in the future using another dedicated Pull Request.
If we decide to use http://travis.debian.net or git-buildpackage, we will have to create dedicated Debian branches.
We will have to create a debian folder in these branches too.
This link can be useful too: https://wiki.debian.org/PackagingWithGit

I think the only thing missing now in this PR before the merge is a short description of the changes in the RELEASE NOTES. I think it is also important to inform the user that from now on, the include files will be installed in <install_prefix>/include/tango directory instead of <install_prefix>/include.

…targets

Ingvord closed this Feb 27, 2018

Ingvord force-pushed the cmake-release-targets branch from 109c0de to e0d625d Compare February 27, 2018 11:51

start

57c2a1e

Ingvord reopened this Feb 27, 2018

Ingvord added this to the Q1'2018 milestone Feb 27, 2018

Progress

b68f74d

Ingvord added 4 commits February 27, 2018 16:13

Progress

3cfe752

Progress

0792b74

Progress: looks like it is better to use main component

b98f1fe

Progress

747322d

Progress

2916e29

Ingvord and others added 11 commits March 5, 2018 16:23

Progress

366b85d

dump version into a file to use by AppVeyor

53fd74d

fix appveyor

8d004fc

fix appveyor

2346b54

Progress #438 add travis deploy

768fe06

Progress #438 add bintray user and api_key

9add0d8

Progress #438 override existing packages

57a2ce2

Progress #438 enhance package name

b92b16e

Progress #438 install curl

cb90c86

Progress #438 install lsb-release

637aa0d

Progress #438 minor changes to cmake

fa1ccef

bourtemb reviewed Mar 26, 2018

View reviewed changes

Ingvord added 2 commits March 26, 2018 14:00

Progress #438 update conflict info

2f7343a

Progress #438 update conflict info

2f91aeb

bourtemb reviewed Mar 29, 2018

View reviewed changes

Progress #438 change packages names

c47f039

Ingvord added 2 commits April 3, 2018 16:20

Progress #438 update conflict info

a18f07d

Progress #438 change headers installation folder

3c37bba

Ingvord added the help wanted label Apr 9, 2018

bourtemb mentioned this pull request Apr 27, 2018

tango_config.h: Add clang detection #442

Merged

bourtemb added 2 commits April 27, 2018 14:04

Merge remote-tracking branch 'origin/tango-9-lts' into cmake-release-…

b88a137

…targets

Update RELEASE_NOTES

583d39b

bourtemb merged commit 9c91432 into tango-9-lts Apr 27, 2018

t-b deleted the cmake-release-targets branch October 29, 2019 15:38

Cmake release targets #438

Cmake release targets #438

Uh oh!

Conversation

Ingvord commented Feb 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ingvord commented Feb 27, 2018

Uh oh!

t-b commented Feb 27, 2018

Uh oh!

Ingvord commented Feb 27, 2018

Uh oh!

bourtemb commented Mar 1, 2018

Uh oh!

Ingvord commented Mar 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NexeyaSGara commented Mar 2, 2018

Uh oh!

t-b commented Mar 2, 2018

Uh oh!

Ingvord commented Mar 26, 2018

Uh oh!

picca commented Mar 26, 2018 via email

Uh oh!

bourtemb Mar 26, 2018

Choose a reason for hiding this comment

Uh oh!

bourtemb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ingvord commented Mar 29, 2018

Uh oh!

bourtemb commented Mar 29, 2018

Uh oh!

Ingvord commented Apr 3, 2018

Uh oh!

chedburgh commented Apr 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Packages

Header Include Paths

Uh oh!

Ingvord commented Apr 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chedburgh commented Apr 9, 2018

Uh oh!

Ingvord commented Apr 9, 2018

Uh oh!

picca commented Apr 9, 2018 via email

Uh oh!

chedburgh commented Apr 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picca commented Apr 9, 2018 via email

Uh oh!

chedburgh commented Apr 9, 2018

Uh oh!

chedburgh commented Apr 9, 2018

Uh oh!

picca commented Apr 9, 2018 via email

Uh oh!

bourtemb commented Apr 9, 2018

Uh oh!

picca commented Apr 10, 2018 via email

Uh oh!

bourtemb commented Apr 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Ingvord commented Feb 27, 2018 •

edited

Loading

Ingvord commented Mar 2, 2018 •

edited

Loading

bourtemb left a comment •

edited

Loading

chedburgh commented Apr 5, 2018 •

edited

Loading

Ingvord commented Apr 6, 2018 •

edited

Loading

chedburgh commented Apr 9, 2018 •

edited

Loading

bourtemb commented Apr 27, 2018 •

edited

Loading