New build system (avast#730)

* something working

* progress

* progress

* progress

* cmake: fix common and add ctypes

* src/serdes: new build system

* new build system for pdbparser

* new build system for yaracpp

* new build system for crypto component

* new build system for config

* refactor cmake for retdec-configtool

* new build system for idr2pat

* new build system for ar-extractor and its LLVM dependency

* refactor the new build system

* new build for ar-extractortool

* new build system for macho-extractor and macho-extractortool

* new build for ctypesparser, demangler, and demanglertool

* new build for llvm-support and llvmir-emul

* new build for capstone2llvmir and capstone2llvmirtool

* new build for fileformat, loader, cpdetect, fileinfo

* new build for most remaining modules

* cmake/options.cmake: fix debugformat enable settings

* some build system fixes

* cmake/options.cmake: make RETDEC_ENABLE_ALL into option

this allows us to set it to ODD and build only documentation

* tests: new build system for unit tests

* pelib: fix doxygen comment

* retdec/retdec: remove unused include of llvm-support/diagnostics.h

* fileformat: do not include openssl in headers

* cmake: make install paths relative.

When they are absolute, then all the paths in instaled cmake scripts are absolute - install dir could not be moved.

* deps/yara: refactor cmake to properly install libs

* deps/yara: small cosmetic changes

* deps/llvm: refactor cmake to properly install libs

* deps/capstone: refactor cmake to properly install libs

* deps: refactor cmake files

* deps: refactor cmake files

* deps/yaramod: refactor cmake to properly install libs

* CMakeLists.txt: fix files installation

* config/retdec-config-config.cmake: fix required component order

Looks like this may matter (on some machines or cmake version).

* deps/llvm: fix include installation

* fileformat/cmake: do not use openssl-crypto

For some reasons, this can cause linking problems on some machines.

* deps/yaramod: fix link library order

* deps/googletests: fix target link library order

* rtti-finder: make deps PUBLIC.

I have no idea why it doesn't work with PRIVATE.

* deps/yaramod: fix formatting

* cmake: more refactorization...

1. Protect against including the same target file multiple times. It looks like this was not a problem, but who knows.
2. Use find_package() instead of find_dependency(). It looks like the later does not work correctly for components on some CMake versions.

* cmake: do not create version files for all components

Only the main RetDec version file is needed.

* cmake: propagate current project version to package config files

It is used to find the same version retdec components.

* CMakeLists.txt: replace AnyNewerVersion -> ExactVersion in compatibility settings

Since we probably will make breaking changes, make retdec installation compatible only with the same version.

* cmake: remove all COMPONENT options, these are not really needed

* cmake: move dependency finders inside double target protection conditions

* cmake: prefix all deps targets with retdec::deps::

* fileinfo: add openssl-crypto dep

* Revert "fileinfo: add openssl-crypto dep"

This reverts commit e1645d7.

* remove RETDEC_FORCE_OPENSSL_BUILD option, always build OpenSSL

* cmake: refactorization

* deps/whereami: fix include paths

* deps/tinyxml+whereami: fix included dirs

* deps/yaramod: fix pog_fmt lib installation

* deps/whereami: properly install target, not as part of utils.

Previous version should work, but there is a CMake bug in older versions which screws it up on Windows and macOS.

* crypto: link retdec::deps::openssl-crypto as PUBLIC

* bin2llvmir/providers/debugformat: fix segfault

Co-authored-by: Peter Matula <imatula@fit.vutbr.cz>

CMakeLists.txt

@@ -1,28 +1,62 @@
 
-cmake_minimum_required(VERSION 3.6)
+cmake_minimum_required(VERSION 3.8)
 
-project(retdec C CXX)
-
-include(GNUInstallDirs)
+project(retdec
+	LANGUAGES C CXX
+	VERSION 4.0
+)
 
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 set(CMAKE_CXX_EXTENSIONS OFF)
 
-include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/utils.cmake)
-
 # Set the default build type to 'Release'.
 if(NOT CMAKE_BUILD_TYPE)
 	set(default_build_type "Release")
 	message(STATUS "Setting build type to '${default_build_type}' as none was specified.")
 	set(CMAKE_BUILD_TYPE "${default_build_type}" CACHE STRING "Choose the type of build." FORCE)
 endif()
 
+## Includes.
+include(GNUInstallDirs)
+include(CMakePackageConfigHelpers)
+include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/utils.cmake)
 include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/options.cmake)
 
 # RetDec, and some dependencies (e.g. LLVM, Keystone), require Python 3.
 find_package(PythonInterp 3.4 REQUIRED)
 
+### Variables.
+
+## Repository directories.
+set(RETDEC_CMAKE_DIR                "${CMAKE_CURRENT_SOURCE_DIR}/cmake")
+set(RETDEC_DEPS_DIR                 "${CMAKE_CURRENT_SOURCE_DIR}/deps")
+set(RETDEC_DOC_DIR                  "${CMAKE_CURRENT_SOURCE_DIR}/doc")
+set(RETDEC_INCLUDE_DIR              "${CMAKE_CURRENT_SOURCE_DIR}/include")
+set(RETDEC_SCRIPTS_DIR              "${CMAKE_CURRENT_SOURCE_DIR}/scripts")
+set(RETDEC_SOURCE_DIR               "${CMAKE_CURRENT_SOURCE_DIR}/src")
+set(RETDEC_SUPPORT_DIR              "${CMAKE_CURRENT_SOURCE_DIR}/support")
+set(RETDEC_TESTS_DIR                "${CMAKE_CURRENT_SOURCE_DIR}/tests")
+## Installation directories.
+# Bins.
+set(RETDEC_INSTALL_BIN_DIR          "${CMAKE_INSTALL_BINDIR}")
+set(RETDEC_INSTALL_BIN_DIR_ABS      "${CMAKE_INSTALL_PREFIX}/${RETDEC_INSTALL_BIN_DIR}")
+set(RETDEC_INSTALL_TESTS_DIR        "${RETDEC_INSTALL_BIN_DIR}")
+# Includes.
+set(RETDEC_INSTALL_INCLUDE_DIR      "${CMAKE_INSTALL_INCLUDEDIR}")
+set(RETDEC_INSTALL_DEPS_INCLUDE_DIR "${RETDEC_INSTALL_INCLUDE_DIR}/retdec/deps")
+# Libs.
+set(RETDEC_INSTALL_LIB_DIR          "${CMAKE_INSTALL_LIBDIR}")
+set(RETDEC_INSTALL_LIB_DIR_ABS      "${CMAKE_INSTALL_PREFIX}/${RETDEC_INSTALL_LIB_DIR}")
+set(RETDEC_INSTALL_DEPS_LIB_DIR     "${RETDEC_INSTALL_LIB_DIR}/deps")
+set(RETDEC_INSTALL_DEPS_LIB_DIR_ABS "${CMAKE_INSTALL_PREFIX}/${RETDEC_INSTALL_DEPS_LIB_DIR}")
+# Data.
+set(RETDEC_INSTALL_DATA_DIR         "${CMAKE_INSTALL_DATADIR}/retdec")
+set(RETDEC_INSTALL_CMAKE_DIR        "${RETDEC_INSTALL_DATA_DIR}/cmake")
+set(RETDEC_INSTALL_DOC_DIR          "${RETDEC_INSTALL_DATA_DIR}/doc")
+set(RETDEC_INSTALL_SUPPORT_DIR      "${RETDEC_INSTALL_DATA_DIR}/support")
+set(RETDEC_INSTALL_SUPPORT_DIR_ABS  "${CMAKE_INSTALL_PREFIX}/${RETDEC_INSTALL_SUPPORT_DIR}")
+
 # On Linux and macOS, set RPATH relative to the origin of the installed
 # executables (i.e. relative to the bin directory). This allows us to move the
 # installation directory into a different location after installation, which is
@@ -111,3 +145,26 @@ cond_add_subdirectory(scripts RETDEC_ENABLE_SCRIPTS)
 add_subdirectory(src)
 cond_add_subdirectory(support RETDEC_ENABLE_SUPPORT)
 add_subdirectory(tests)
+
+# Create config version file.
+write_basic_package_version_file(
+	"${CMAKE_CURRENT_BINARY_DIR}/retdec-config-version.cmake"
+	VERSION ${PROJECT_VERSION}
+	COMPATIBILITY ExactVersion
+)
+
+# Create main RetDec CMake config file.
+configure_file(
+	"${CMAKE_CURRENT_LIST_DIR}/retdec-config.cmake"
+	"${CMAKE_CURRENT_BINARY_DIR}/retdec-config.cmake"
+	@ONLY
+)
+
+# Install CMake files.
+install(
+	FILES
+		"${CMAKE_CURRENT_BINARY_DIR}/retdec-config.cmake"
+		"${CMAKE_CURRENT_BINARY_DIR}/retdec-config-version.cmake"
+	DESTINATION
+		"${RETDEC_INSTALL_CMAKE_DIR}"
+)

README.md

@@ -155,7 +155,6 @@ sudo pacman --needed -S base-devel cmake git perl python3 autoconf automake libt
 * [CMake](https://cmake.org/) (version >= 3.6)
 * [Git](https://git-scm.com/)
 * [Active Perl](https://www.activestate.com/activeperl). It needs to be the first Perl in `PATH`, or it has to be provided to CMake using `CMAKE_PROGRAM_PATH` variable, e.g. `-DCMAKE_PROGRAM_PATH=/c/perl/bin`. Does NOT work with Strawberry Perl or MSYS2 Perl (you would have to install a pre-built version of OpenSSL, see below).
-  * Alternatively, you can install OpenSSL directly from [here](https://slproweb.com/products/Win32OpenSSL.html). This means OpenSSL won't be built and you don't need to install any Perl. Do not install Light version of OpenSSL as they don't contain development files.
 * [Python](https://www.python.org/) (version >= 3.4)
 * Optional: [Doxygen](http://ftp.stack.nl/pub/users/dimitri/doxygen-1.8.13-setup.exe) and [Graphviz](https://graphviz.gitlab.io/_pages/Download/windows/graphviz-2.38.msi) for generating API documentation
 
@@ -238,7 +237,6 @@ You can pass the following additional parameters to `cmake`:
 * `-DRETDEC_DOC=ON` to build with API documentation (requires Doxygen and Graphviz, disabled by default).
 * `-DRETDEC_TESTS=ON` to build with tests (disabled by default).
 * `-DRETDEC_DEV_TOOLS=ON` to build with development tools (disabled by default).
-* `-DRETDEC_FORCE_OPENSSL_BUILD=ON` to force OpenSSL build even if it is installed in the system (disabled by default).
 * `-DRETDEC_COMPILE_YARA=OFF` to disable YARA rules compilation at installation step (enabled by default).
 * `-DCMAKE_BUILD_TYPE=Debug` to build with debugging information, which is useful during development. By default, the project is built in the `Release` mode. This has no effect on Windows, but the same thing can be achieved by running `cmake --build .` with the `--config Debug` parameter.
 * `-DCMAKE_PROGRAM_PATH=<path>` to use Perl at `<path>` (probably useful only on Windows).

cmake/options.cmake

@@ -4,7 +4,6 @@
 option(RETDEC_DOC "Build public API documentation (requires Doxygen)." OFF)
 option(RETDEC_TESTS "Build tests." OFF)
 option(RETDEC_DEV_TOOLS "Build dev tools." OFF)
-option(RETDEC_FORCE_OPENSSL_BUILD "Force OpenSSL build." OFF)
 option(RETDEC_COMPILE_YARA "Compile YARA rules at installation." ON)
 option(RETDEC_MSVC_STATIC_RUNTIME "Use a multi-threaded statically-linked runtime library." OFF)
 
@@ -59,7 +58,7 @@ include(${PROJECT_SOURCE_DIR}/cmake/deps.cmake)
 
 # Default: ALL build is enabled.
 #
-set(RETDEC_ENABLE_ALL ON)
+option(RETDEC_ENABLE_ALL "" ON)
 
 # Convert target list in RETDEC_ENABLE to RETDEC_ENABLE_<component> flags.
 #
@@ -298,20 +297,12 @@ set_if_at_least_one_set(RETDEC_ENABLE_DEBUGFORMAT
 		RETDEC_ENABLE_ALL
 		RETDEC_ENABLE_BIN2LLVMIR)
 
-set_if_at_least_one_set(RETDEC_ENABLE_CTYPESPARSER
-		RETDEC_ENABLE_ALL
-		RETDEC_ENABLE_BIN2LLVMIR
-		RETDEC_ENABLE_DEMANGLER)
-
 set_if_at_least_one_set(RETDEC_ENABLE_DEMANGLER
 		RETDEC_ENABLE_ALL
 		RETDEC_ENABLE_BIN2LLVMIR
-		RETDEC_ENABLE_CTYPESPARSER
 		RETDEC_ENABLE_DEBUGFORMAT
 		RETDEC_ENABLE_DEMANGLERTOOL)
 
-# TODO: second run because demangler and ctypesparser are cyclically dependent.
-# This dependency should be avoided.
 set_if_at_least_one_set(RETDEC_ENABLE_CTYPESPARSER
 		RETDEC_ENABLE_ALL
 		RETDEC_ENABLE_BIN2LLVMIR
@@ -322,12 +313,12 @@ set_if_at_least_one_set(RETDEC_ENABLE_CTYPES
 		RETDEC_ENABLE_CTYPESPARSER)
 
 set_if_at_least_one_set(RETDEC_ENABLE_PDBPARSER
-		RETDEC_ENABLE_CPDETECT
 		RETDEC_ENABLE_DEBUGFORMAT)
 
 set_if_at_least_one_set(RETDEC_ENABLE_LOADER
 		RETDEC_ENABLE_ALL
 		RETDEC_ENABLE_BIN2LLVMIR
+		RETDEC_ENABLE_DEBUGFORMAT
 		RETDEC_ENABLE_FILEINFO
 		RETDEC_ENABLE_RTTI_FINDER
 		RETDEC_ENABLE_STACOFIN
@@ -339,6 +330,7 @@ set_if_at_least_one_set(RETDEC_ENABLE_FILEFORMAT
 		RETDEC_ENABLE_ALL
 		RETDEC_ENABLE_BIN2LLVMIR
 		RETDEC_ENABLE_CPDETECT
+		RETDEC_ENABLE_DEBUGFORMAT
 		RETDEC_ENABLE_FILEINFO
 		RETDEC_ENABLE_GETSIG
 		RETDEC_ENABLE_LOADER
@@ -362,7 +354,6 @@ set_if_at_least_one_set(RETDEC_ENABLE_CONFIG
 		RETDEC_ENABLE_ALL
 		RETDEC_ENABLE_BIN2LLVMIR
 		RETDEC_ENABLE_CONFIGTOOL
-		RETDEC_ENABLE_DEBUGFORMAT
 		RETDEC_ENABLE_FILEINFO
 		RETDEC_ENABLE_LLVMIR2HLL
 		RETDEC_ENABLE_STACOFIN)
@@ -391,6 +382,7 @@ set_if_at_least_one_set(RETDEC_ENABLE_UTILS
 		RETDEC_ENABLE_BIN2PAT
 		RETDEC_ENABLE_CAPSTONE2LLVMIRTOOL
 		RETDEC_ENABLE_CONFIG
+		RETDEC_ENABLE_COMMON
 		RETDEC_ENABLE_CRYPTO
 		RETDEC_ENABLE_CTYPES
 		RETDEC_ENABLE_CTYPESPARSER
@@ -402,7 +394,7 @@ set_if_at_least_one_set(RETDEC_ENABLE_UTILS
 		RETDEC_ENABLE_LOADER
 		RETDEC_ENABLE_MACHO_EXTRACTOR
 		RETDEC_ENABLE_MACHO_EXTRACTORTOOL
-		RETDEC_ENABLE_PRETDEC_ENABLE_CPDETECTAT2YARA
+		RETDEC_ENABLE_CPDETECT
 		RETDEC_ENABLE_PATTERNGEN
 		RETDEC_ENABLE_RTTI_FINDER
 		RETDEC_ENABLE_STACOFIN
@@ -518,7 +510,8 @@ set_if_at_least_one_set(RETDEC_ENABLE_RAPIDJSON
 
 set_if_at_least_one_set(RETDEC_ENABLE_TINYXML2
 		RETDEC_ENABLE_TINYXML2
-		RETDEC_ENABLE_FILEINFO)
+		RETDEC_ENABLE_FILEINFO
+		RETDEC_ENABLE_CPDETECT)
 
 set_if_at_least_one_set(RETDEC_ENABLE_VARIANT
 		RETDEC_ENABLE_UTILS)

deps/CMakeLists.txt

@@ -1,6 +1,20 @@
+
+include(ExternalProject)
+
+include(ProcessorCount)
+ProcessorCount(CPUS)
+
+if(CMAKE_C_COMPILER)
+	set(CMAKE_C_COMPILER_OPTION "-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}")
+endif()
+if(CMAKE_CXX_COMPILER)
+	set(CMAKE_CXX_COMPILER_OPTION "-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}")
+endif()
+
 if(MSVC)
-	set(DEBUG_DIR "Debug/")
 	set(RELEASE_DIR "Release/")
+	set(DEBUG_DIR "Debug/")
+	set(DEBUG_LIB_SUFFIX "d")
 endif()
 
 cond_add_subdirectory(capstone RETDEC_ENABLE_CAPSTONE)

deps/capstone/CMakeLists.txt

@@ -1,11 +1,3 @@
-include(ExternalProject)
-
-if(CMAKE_C_COMPILER)
-	set(CMAKE_C_COMPILER_OPTION "-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}")
-endif()
-if(CMAKE_CXX_COMPILER)
-	set(CMAKE_CXX_COMPILER_OPTION "-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}")
-endif()
 
 if(CAPSTONE_LOCAL_DIR)
 	message(STATUS "Capstone: using local Capstone directory.")
@@ -104,10 +96,76 @@ endif()
 ExternalProject_Get_Property(capstone-project source_dir)
 ExternalProject_Get_Property(capstone-project binary_dir)
 
-# Add libraries.
+set(CAPSTONE_LIB_NAME  ${CMAKE_STATIC_LIBRARY_PREFIX}capstone${CMAKE_STATIC_LIBRARY_SUFFIX})
+set(CAPSTONE_LIB_DEBUG ${binary_dir}/${DEBUG_DIR}${CAPSTONE_LIB_NAME})
+set(CAPSTONE_LIB_OPT   ${binary_dir}/${RELEASE_DIR}${CAPSTONE_LIB_NAME})
+
+# Create target.
 add_library(capstone INTERFACE)
+add_library(retdec::deps::capstone ALIAS capstone)
 add_dependencies(capstone capstone-project)
-target_include_directories(capstone SYSTEM INTERFACE ${source_dir}/include)
-target_include_directories(capstone SYSTEM INTERFACE ${source_dir}/arch)
-target_link_libraries(capstone INTERFACE debug ${binary_dir}/${DEBUG_DIR}${CMAKE_STATIC_LIBRARY_PREFIX}capstone${CMAKE_STATIC_LIBRARY_SUFFIX})
-target_link_libraries(capstone INTERFACE optimized ${binary_dir}/${RELEASE_DIR}${CMAKE_STATIC_LIBRARY_PREFIX}capstone${CMAKE_STATIC_LIBRARY_SUFFIX})
+
+target_include_directories(capstone
+	SYSTEM INTERFACE
+		$<BUILD_INTERFACE:${source_dir}/include>
+		$<BUILD_INTERFACE:${source_dir}/arch>
+		$<INSTALL_INTERFACE:${RETDEC_INSTALL_DEPS_INCLUDE_DIR}>
+)
+
+target_link_libraries(capstone INTERFACE
+		$<INSTALL_INTERFACE:retdec::deps::capstone-libs>
+	debug
+		$<BUILD_INTERFACE:${CAPSTONE_LIB_DEBUG}>
+	optimized
+		$<BUILD_INTERFACE:${CAPSTONE_LIB_OPT}>
+)
+
+# Install includes.
+install(
+	DIRECTORY
+		${source_dir}/include/
+	DESTINATION
+		${RETDEC_INSTALL_DEPS_INCLUDE_DIR}
+)
+
+# Install libs.
+# Install both Release and Debug variant to the same location.
+# We assume that only one variant will be present at the time.
+install(
+    FILES
+		${CAPSTONE_LIB_DEBUG}
+		${CAPSTONE_LIB_OPT}
+    DESTINATION
+		${RETDEC_INSTALL_DEPS_LIB_DIR}
+	OPTIONAL
+)
+
+# Install targets.
+install(TARGETS capstone
+	EXPORT capstone-targets
+)
+
+# Export targets.
+install(EXPORT capstone-targets
+	FILE "retdec-capstone-targets.cmake"
+	NAMESPACE retdec::deps::
+	DESTINATION ${RETDEC_INSTALL_CMAKE_DIR}
+)
+
+# Configure config file.
+set(CAPSTONE_LIB_INSTALLED "${RETDEC_INSTALL_DEPS_LIB_DIR_ABS}/${CAPSTONE_LIB_NAME}")
+configure_package_config_file(
+	"retdec-capstone-config.cmake"
+	"${CMAKE_CURRENT_BINARY_DIR}/retdec-capstone-config.cmake"
+	INSTALL_DESTINATION ${RETDEC_INSTALL_CMAKE_DIR}
+	PATH_VARS
+		CAPSTONE_LIB_INSTALLED
+)
+
+# Install CMake files.
+install(
+	FILES
+		"${CMAKE_CURRENT_BINARY_DIR}/retdec-capstone-config.cmake"
+	DESTINATION
+		"${RETDEC_INSTALL_CMAKE_DIR}"
+)

deps/capstone/retdec-capstone-config.cmake

@@ -0,0 +1,12 @@
+@PACKAGE_INIT@
+
+if(NOT TARGET retdec::deps::capstone-libs)
+    add_library(retdec::deps::capstone-libs STATIC IMPORTED)
+    set_target_properties(retdec::deps::capstone-libs PROPERTIES
+        IMPORTED_LOCATION @PACKAGE_CAPSTONE_LIB_INSTALLED@
+    )
+endif()
+
+if(NOT TARGET retdec::deps::capstone)
+    include(${CMAKE_CURRENT_LIST_DIR}/retdec-capstone-targets.cmake)
+endif()

deps/elfio/CMakeLists.txt

@@ -1,3 +1,36 @@
 
 add_library(elfio INTERFACE)
-target_include_directories(elfio SYSTEM INTERFACE ${CMAKE_CURRENT_LIST_DIR}/include)
+add_library(retdec::deps::elfio ALIAS elfio)
+
+target_include_directories(elfio
+    SYSTEM INTERFACE
+        $<BUILD_INTERFACE:${RETDEC_DEPS_DIR}/elfio/include>
+        $<INSTALL_INTERFACE:${RETDEC_INSTALL_DEPS_INCLUDE_DIR}>
+)
+
+# Install includes.
+install(
+	DIRECTORY ${RETDEC_DEPS_DIR}/elfio/include/
+	DESTINATION ${RETDEC_INSTALL_DEPS_INCLUDE_DIR}
+)
+
+# Install targets.
+install(TARGETS elfio
+	EXPORT elfio-targets
+	# Header only -> there are no actual libs to install to destination.
+)
+
+# Export targets.
+install(EXPORT elfio-targets
+	FILE "retdec-elfio-targets.cmake"
+	NAMESPACE retdec::deps::
+	DESTINATION ${RETDEC_INSTALL_CMAKE_DIR}
+)
+
+# Install CMake files.
+install(
+	FILES
+		"${CMAKE_CURRENT_LIST_DIR}/retdec-elfio-config.cmake"
+	DESTINATION
+		"${RETDEC_INSTALL_CMAKE_DIR}"
+)

deps/elfio/retdec-elfio-config.cmake

@@ -0,0 +1,4 @@
+
+if(NOT TARGET retdec::deps::elfio)
+    include(${CMAKE_CURRENT_LIST_DIR}/retdec-elfio-targets.cmake)
+endif()