Skip to content

Build system needs replacing #200

@zultron

Description

@zultron

There have been a number of discussions about replacing the build system, but I can't seem to find an issue for it. (Apologies for the length; deficiencies are probably overly-detailed after I've encountered scepticism about whether the current build system in fact even has any major problems.)

The problem

The build system is decrepit and needs replacing. The problems of the build system can be explained in the context of its origins.

  • Began with the original, plain Makefile build system with Linux Kbuild integration
    • $EMC_HOME environment variable configures both build and run-time systems
      • This resembles other CAD and EDA packages from the '90s (TCL was also a favorite of the era)
    • Each type of build artifact must be dealt with manually
      • Introducing a new type of artifacts requires writing new rules
      • No out-of-the-box macros or scripts, like in high-level build systems
    • Additional complexity to integrate with the Linux Kbuild build system for building .ko modules becomes an issue later
    • "Shotgun approach" to includes: Simply include every subdirectory with gcc -I in the %.o: %.c: target!
  • GNU Autoconf added later to ease build parameter configuration
    • $EMC_HOME conflated with $prefix, but semantic differences
      • Makes it impossible to ./configure --prefix=/usr (the default in normal Autotools build systems), run tests or run the application, and then make install for a system install
      • Instead, need separate "RIP" build to run locally; $prefix artificially set to $EMC_HOME, the source directory by default
        • Setting $prefix inside configure.ac introduces subtle problems, e.g. requires crazy hacks to template files that include variables based on $prefix
    • Automake support was not built in at the same time
      • Perhaps because it's difficult to integrate with Kbuild, or perhaps just to avoid a lot of rework with $EMC_HOME
      • The build rules and libtool support are another ingredient missing to run out of the source directory and then make install to /usr without rebuilding from zero
    • As long as a separate RIP build is required to run from the source tree, tests cannot be run during the package build
  • (Somewhere in there) pthreads "sim" build was introduced
    • The "sim" build is the granddaddy of today's RT_PREEMPT, etc. user-space threads
    • Build system now has to build userland .so plugins
      • Userland threads share module sources but build them differently
      • The Kbuild integration became even more complex
      • New build rules and new build configuration settings
  • The "unified build" came in along with Machinekit (an atrocity committed by YT)
    • Introduced a disgusting recursive make modules for each configured thread flavor
    • Needed crazy hacks to build differently configured .o files from the same .c files to integrate with the "sim" module build rules
    • Introduced some build dependency bugs that were never resolved
  • All along the way, dozens of contributors added their own rules
    • Trivial changes to add a new module with no external dependencies are easy to add
    • But non-trivial changes take special knowledge to integrate cleanly
      • How to integrate new rules with Kbuild? src/subdir/Submakefile or src/Makefile?
      • Which CFLAGS or LDFLAGS variable to use?
        • Userspace or Kbuild? Where are all those *FLAGS variables documented?
        • Solution: Make a new variable!
      • Where do build artifacts for new types of builds go?
        • Normally, build artifacts go in the src/objects directory
        • But sometimes it's just easier to put them next to the sources!
      • Where do the artifacts' dependencies get plugged in?
        • Worst case: shotgun approach (cf. make clean; make threads=posix objects/posix/hal/components/encoder.o)
    • As a result, the build system's complexity and disorganization only leads to increasing disorganization over time

The end result, the build system of today, is very difficult to modify. It's difficult to understand the structure of dependencies, files and variables in order to know how to introduce the change at all, much less where to add it so that it doesn't make the structure even more disorganized and hard to understand for the next person.

The only way to fix it at this point is by replacing it. Fortunately, this will be easier today than ever before. The split machinekit-hal repository reduces the size of the task. The upcoming per-flavor module merge will simplify build rules. Eliminating kernel threads also eliminates the requirement to support kbuild. And today there are two options that are already 50% complete.

Possible build system replacements

Among alternatives, these make the best sense, given the enormous head start, and given that both are widely-used and well-supported in general, and that the Machinekit community has more or less experience with both.

  • CMake:
    • Several top Machinekit contributors have experience already
    • There have been a few attempts already that would get us most of the way
    • Needs selecting the most complete one, pulling from others where possible, and finishing the configuration
  • Autotools:
    • Machinekit community already familiar with Autoconf portion
    • The existing Autoconf configure.ac gets us halfway to a full Autotools configuration
    • Needs developing the Automake configuration; starting point is zero, AFAIK

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions