Autotools generated scripts are under version control

[ryao]

I am told that projects only generate these scripts when tagging releases, but we seem to have them under version control. Why is that?

[behlendorf]

They're right. Typically, these files aren't tracked under version control and are just included in the tagged tarballs.

However, since I've yet to tag a stable release and we have lots of people still pulling from the git repo directly it was convenient to include these autotools products. Otherwise you might need to find a system with the right version of autotools installed to generate the configure script, so you could then build the project. The products of autotools are portable, but things like the configure.ac files and Makefiles.am. They are often tied more closely to specific versions of autotools than people would like to admit.

That said, perhaps we getting to the point now where most people shouldn't be pulling directly from the master branch on Github. They can use one of the tagged -rc tarballs or either the versions integrated with the downstream distributions. If that's now more than norm we could seriously consider removing these files from the repo.

[chrisrd]

For what it's worth, I'd vote to keep the generated scripts in git for (my) convenience.

[behlendorf]

Well certainly keep them for a while yet, but perhaps we can drop them shortly before tagging a stable 0.6.0 which I think it still a ways off.

[ryao]

@chrisrd Which distribution do you use?

If I provided a way for you to compile a kernel, an initramfs image, SPL and ZFS on your distribution using Gentoo's package manager (e.g. inside /opt/gentoo), would you use it?

[chrisrd]

Debian. No (but thanks!). I have my own workflow in place to manage the bits and pieces of kernel-land I'm interested in whilst they're in flux. But I'm looking forward to the day when I'm no longer on the bleeding edge and can simply use standard packages!

[behlendorf]

My current intention is to drop the generated autotools script for the repositories once the following RPM DKMS changes have been merged in to an official tag.

This should allow us to start posting just tarballs and source RPM packages to the website for general use. I'll of course also keep prominent links to both the Ubuntu/Debian PPA and Gentoo packages. Building directly from the repositories can be demoted to a developer only thing where they should be expected to have the autotools installed.

openzfs/spl#136
#830

It still may take a while to get these merged but I wanted to give everyone a heads up. Does this seem reasonable to everyone, are there any concerns?

[ryao]

This will cause no problems in Gentoo. The package manager will generate the autotools scripts as part of the build process when building directly from ZFSOnLinux GIT.

[Rudd-O]

Having the autotools scripts in the repo (makefile.in and the like) isn't a "convenience". It's a huge mess. It makes merging, rebasing and other repo operations (which should be straightforward) into a NIGHTMARE, because every single autotools out there generates different scripts (which are just a mess of shellscripting) which makes it impossible to fish out the relevant changes for merging.

I vote for them to be removed since they have caused me indescribable pain and completely demotivated me from contributing more.

In fact, I would fully support ditching autotools altogether and using something more rational like waf, which doesn't require at all littering the repository with meta-meta-meta-meta-meta files.

[ryao]

@Rudd-O Do what I do. Extend your system package manager to pull from a branch on github, run autotools, build and install. Then Ignore the autotools scripts when doing git operation. Next, push to the designated branch on github when you need to test a change and finally, let your package manager handle autotools.

I find that to be the only sane way of dealing with the autotools scripts.

[behlendorf]

More convenient for users, not for developers. By this point I hope most users are sticking to the tarballs or using whatever solution is best packaged for their distribution.

Anyway, once we get the RPM DKMS changes in place I'm happy to drop the autogen.sh products. Just be patient with me for a few more weeks.

[Rudd-O]

@ryao: my scripts already do autogen.sh (well, they used to, when I wasn't using DKMS -- I am now) because that was the only sane way to operate. The problem is that the garbage autotools-generated scripts make it nigh impossible to merge branches. My process today consists on reverting my own autotools-script changes, generating a diff between the upstream autotools scripts and mine, applying that diff as a commit, then doing the merge, then backing out that autotools-scripts-diff out using rebase --interactive. It's crazy enough that I fear it will give me trichotilomania.

Would you guys entertain patches of the build sys to a full port to waf? No more garbage scripts!

[behlendorf]

Would you guys entertain patches of the build sys to a full port to waf? No more garbage scripts!

No. Autotools may have it flaws but it's still the most flexible and portable thing for a complicated project out there.

[Rudd-O]

I respectfully beg to differ with your judgment in this case.

waf is much more portable (it's fully self-contained, works everywhere, doesn't require users or developers to install anything, no autoconf, no autoheader, no automake, just to extend the build system), it's way more flexible (for example, it supports multiple configurations like DKMS or no-DKMS out of the box, unlike the autoconf hacks we have had to use at make time), its configuration is WAY easier to understand and extend than the arcane autotools files (m4, really?), and waf doesn't litter the project at all with random files.

Oh, and it's much faster too. Not only does waf build everything in parallel, its configure tests (the lengthiest part) are much, much faster than configure, and it only rebuilds what's necessary since waf understands the whole dependency chain. For me, this is the biggest, by far, the biggest win. If anything else, you should expect build and compile times to be slashed in half (or more for multicore systems).

The SAMBA project successfully replaced autocancer with waf (and they have vastly more complicated portability requirements than us). I personally used waf to replace an ant build system at Cloud.com, slashing compile times from 15minutes+ to less than 90 seconds.

The hard part is porting the M4 tests, really, and I would be doing that part.

There are real benefits to using something like waf. I respectfully suggest you reconsider your initial assessment.

[ryao]

@Rudd-O Why are you having problems merging branches due to the autotools scripts? If the only changes to the files that ./autogen.sh touches are being made at upstream until you try building the software, then there should be no issues.

I do not think that anyone will commit to switch from autotools before seeing it in action. If you want to change people's minds, you will need to rewrite the build system in waf, without any guarantee that your work will be adopted. If you do that, I am sure that @behlendorf would evaluate it.

[Rudd-O]

Yes, absolutely, the plan is for me to rewrite the build system using waf, then be evaluated by you guys, and then be committed if it passes muster. I just didn't want to do the whole endeavor (it is going to be a few days' work) if there was a GOOD reason why the build system should not be migrated to waf.