Meta build system multi-thread

From time to time, the topic of why meta-build systems like CMake and Meson are single-threaded sequentially executing processes is brought up. With desktop workstations (not to mention build farms) having 32, 64, and even 128+ CPU cores increasingly widespread, and the configure / generation step of meta-build systems taking tens of seconds to a few minutes on large projects, developers are understandably frustrated by the lack of parallelism.

A fundamental issue with CMake, Meson and equivalently for other meta-build systems is that the user’s CMake scripts would then have to be dataflow / declarative versus imperative. This would require reworking of script syntax and meta-build internal code radically.

In Meson, Python threading (single thread executes at one time) is used in subprojects, giving a speed boost in download time of subproject source code. There is no Python multiprocessing or ProcessPoolExecutor in Meson configure step. Meson parallel execution is for build (Ninja) and test. Both build and test are also done in parallel in CMake. For CMake, the ExternalProject steps can already be run in parallel (including download) via the underlying build system.

A way to speed up meta-build configure time–here specific to CMake–is to stuff the CMakeCache.txt file with precomputed values and/or use CMake Toolchain to do likewise, skipping configure tests when the host build system is static. CMakeCache.txt stuffing is a technique Meson uses to speed up configure time of CMake-based subprojects from Meson projects.