build-system.rst (18226B)
1================================== 2The QEMU build system architecture 3================================== 4 5This document aims to help developers understand the architecture of the 6QEMU build system. As with projects using GNU autotools, the QEMU build 7system has two stages, first the developer runs the "configure" script 8to determine the local build environment characteristics, then they run 9"make" to build the project. There is about where the similarities with 10GNU autotools end, so try to forget what you know about them. 11 12 13Stage 1: configure 14================== 15 16The QEMU configure script is written directly in shell, and should be 17compatible with any POSIX shell, hence it uses #!/bin/sh. An important 18implication of this is that it is important to avoid using bash-isms on 19development platforms where bash is the primary host. 20 21In contrast to autoconf scripts, QEMU's configure is expected to be 22silent while it is checking for features. It will only display output 23when an error occurs, or to show the final feature enablement summary 24on completion. 25 26Because QEMU uses the Meson build system under the hood, only VPATH 27builds are supported. There are two general ways to invoke configure & 28perform a build: 29 30 - VPATH, build artifacts outside of QEMU source tree entirely:: 31 32 cd ../ 33 mkdir build 34 cd build 35 ../qemu/configure 36 make 37 38 - VPATH, build artifacts in a subdir of QEMU source tree:: 39 40 mkdir build 41 cd build 42 ../configure 43 make 44 45For now, checks on the compilation environment are found in configure 46rather than meson.build, though this is expected to change. The command 47line is parsed in the configure script and, whenever needed, converted 48into the appropriate options to Meson. 49 50New checks should be added to Meson, which usually comprises the 51following tasks: 52 53 - Add a Meson build option to meson_options.txt. 54 55 - Add support to the command line arg parser to handle any new 56 ``--enable-XXX``/``--disable-XXX`` flags required by the feature. 57 58 - Add information to the help output message to report on the new 59 feature flag. 60 61 - Add code to perform the actual feature check. 62 63 - Add code to include the feature status in ``config-host.h`` 64 65 - Add code to print out the feature status in the configure summary 66 upon completion. 67 68 69Taking the probe for SDL2_Image as an example, we have the following pieces 70in configure:: 71 72 # Initial variable state 73 sdl_image=auto 74 75 ..snip.. 76 77 # Configure flag processing 78 --disable-sdl-image) sdl_image=disabled 79 ;; 80 --enable-sdl-image) sdl_image=enabled 81 ;; 82 83 ..snip.. 84 85 # Help output feature message 86 sdl-image SDL Image support for icons 87 88 ..snip.. 89 90 # Meson invocation 91 -Dsdl_image=$sdl_image 92 93In meson_options.txt:: 94 95 option('sdl', type : 'feature', value : 'auto', 96 description: 'SDL Image support for icons') 97 98In meson.build:: 99 100 # Detect dependency 101 sdl_image = dependency('SDL2_image', required: get_option('sdl_image'), 102 method: 'pkg-config', 103 kwargs: static_kwargs) 104 105 # Create config-host.h (if applicable) 106 config_host_data.set('CONFIG_SDL_IMAGE', sdl_image.found()) 107 108 # Summary 109 summary_info += {'SDL image support': sdl_image.found()} 110 111 112 113Helper functions 114---------------- 115 116The configure script provides a variety of helper functions to assist 117developers in checking for system features: 118 119``do_cc $ARGS...`` 120 Attempt to run the system C compiler passing it $ARGS... 121 122``do_cxx $ARGS...`` 123 Attempt to run the system C++ compiler passing it $ARGS... 124 125``compile_object $CFLAGS`` 126 Attempt to compile a test program with the system C compiler using 127 $CFLAGS. The test program must have been previously written to a file 128 called $TMPC. The replacement in Meson is the compiler object ``cc``, 129 which has methods such as ``cc.compiles()``, 130 ``cc.check_header()``, ``cc.has_function()``. 131 132``compile_prog $CFLAGS $LDFLAGS`` 133 Attempt to compile a test program with the system C compiler using 134 $CFLAGS and link it with the system linker using $LDFLAGS. The test 135 program must have been previously written to a file called $TMPC. 136 The replacement in Meson is ``cc.find_library()`` and ``cc.links()``. 137 138``has $COMMAND`` 139 Determine if $COMMAND exists in the current environment, either as a 140 shell builtin, or executable binary, returning 0 on success. The 141 replacement in Meson is ``find_program()``. 142 143``check_define $NAME`` 144 Determine if the macro $NAME is defined by the system C compiler 145 146``check_include $NAME`` 147 Determine if the include $NAME file is available to the system C 148 compiler. The replacement in Meson is ``cc.has_header()``. 149 150``write_c_skeleton`` 151 Write a minimal C program main() function to the temporary file 152 indicated by $TMPC 153 154``feature_not_found $NAME $REMEDY`` 155 Print a message to stderr that the feature $NAME was not available 156 on the system, suggesting the user try $REMEDY to address the 157 problem. 158 159``error_exit $MESSAGE $MORE...`` 160 Print $MESSAGE to stderr, followed by $MORE... and then exit from the 161 configure script with non-zero status 162 163``query_pkg_config $ARGS...`` 164 Run pkg-config passing it $ARGS. If QEMU is doing a static build, 165 then --static will be automatically added to $ARGS 166 167 168Stage 2: Meson 169============== 170 171The Meson build system is currently used to describe the build 172process for: 173 1741) executables, which include: 175 176 - Tools - qemu-img, qemu-nbd, qga (guest agent), etc 177 178 - System emulators - qemu-system-$ARCH 179 180 - Userspace emulators - qemu-$ARCH 181 182 - Unit tests 183 1842) documentation 185 1863) ROMs, which can be either installed as binary blobs or compiled 187 1884) other data files, such as icons or desktop files 189 190All executables are built by default, except for some ``contrib/`` 191binaries that are known to fail to build on some platforms (for example 19232-bit or big-endian platforms). Tests are also built by default, 193though that might change in the future. 194 195The source code is highly modularized, split across many files to 196facilitate building of all of these components with as little duplicated 197compilation as possible. Using the Meson "sourceset" functionality, 198``meson.build`` files group the source files in rules that are 199enabled according to the available system libraries and to various 200configuration symbols. Sourcesets belong to one of four groups: 201 202Subsystem sourcesets: 203 Various subsystems that are common to both tools and emulators have 204 their own sourceset, for example ``block_ss`` for the block device subsystem, 205 ``chardev_ss`` for the character device subsystem, etc. These sourcesets 206 are then turned into static libraries as follows:: 207 208 libchardev = static_library('chardev', chardev_ss.sources(), 209 name_suffix: 'fa', 210 build_by_default: false) 211 212 chardev = declare_dependency(link_whole: libchardev) 213 214 As of Meson 0.55.1, the special ``.fa`` suffix should be used for everything 215 that is used with ``link_whole``, to ensure that the link flags are placed 216 correctly in the command line. 217 218Target-independent emulator sourcesets: 219 Various general purpose helper code is compiled only once and 220 the .o files are linked into all output binaries that need it. 221 This includes error handling infrastructure, standard data structures, 222 platform portability wrapper functions, etc. 223 224 Target-independent code lives in the ``common_ss``, ``softmmu_ss`` and 225 ``user_ss`` sourcesets. ``common_ss`` is linked into all emulators, 226 ``softmmu_ss`` only in system emulators, ``user_ss`` only in user-mode 227 emulators. 228 229 Target-independent sourcesets must exercise particular care when using 230 ``if_false`` rules. The ``if_false`` rule will be used correctly when linking 231 emulator binaries; however, when *compiling* target-independent files 232 into .o files, Meson may need to pick *both* the ``if_true`` and 233 ``if_false`` sides to cater for targets that want either side. To 234 achieve that, you can add a special rule using the ``CONFIG_ALL`` 235 symbol:: 236 237 # Some targets have CONFIG_ACPI, some don't, so this is not enough 238 softmmu_ss.add(when: 'CONFIG_ACPI', if_true: files('acpi.c'), 239 if_false: files('acpi-stub.c')) 240 241 # This is required as well: 242 softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('acpi-stub.c')) 243 244Target-dependent emulator sourcesets: 245 In the target-dependent set lives CPU emulation, some device emulation and 246 much glue code. This sometimes also has to be compiled multiple times, 247 once for each target being built. Target-dependent files are included 248 in the ``specific_ss`` sourceset. 249 250 Each emulator also includes sources for files in the ``hw/`` and ``target/`` 251 subdirectories. The subdirectory used for each emulator comes 252 from the target's definition of ``TARGET_BASE_ARCH`` or (if missing) 253 ``TARGET_ARCH``, as found in ``default-configs/targets/*.mak``. 254 255 Each subdirectory in ``hw/`` adds one sourceset to the ``hw_arch`` dictionary, 256 for example:: 257 258 arm_ss = ss.source_set() 259 arm_ss.add(files('boot.c'), fdt) 260 ... 261 hw_arch += {'arm': arm_ss} 262 263 The sourceset is only used for system emulators. 264 265 Each subdirectory in ``target/`` instead should add one sourceset to each 266 of the ``target_arch`` and ``target_softmmu_arch``, which are used respectively 267 for all emulators and for system emulators only. For example:: 268 269 arm_ss = ss.source_set() 270 arm_softmmu_ss = ss.source_set() 271 ... 272 target_arch += {'arm': arm_ss} 273 target_softmmu_arch += {'arm': arm_softmmu_ss} 274 275Module sourcesets: 276 There are two dictionaries for modules: ``modules`` is used for 277 target-independent modules and ``target_modules`` is used for 278 target-dependent modules. When modules are disabled the ``module`` 279 source sets are added to ``softmmu_ss`` and the ``target_modules`` 280 source sets are added to ``specific_ss``. 281 282 Both dictionaries are nested. One dictionary is created per 283 subdirectory, and these per-subdirectory dictionaries are added to 284 the toplevel dictionaries. For example:: 285 286 hw_display_modules = {} 287 qxl_ss = ss.source_set() 288 ... 289 hw_display_modules += { 'qxl': qxl_ss } 290 modules += { 'hw-display': hw_display_modules } 291 292Utility sourcesets: 293 All binaries link with a static library ``libqemuutil.a``. This library 294 is built from several sourcesets; most of them however host generated 295 code, and the only two of general interest are ``util_ss`` and ``stub_ss``. 296 297 The separation between these two is purely for documentation purposes. 298 ``util_ss`` contains generic utility files. Even though this code is only 299 linked in some binaries, sometimes it requires hooks only in some of 300 these and depend on other functions that are not fully implemented by 301 all QEMU binaries. ``stub_ss`` links dummy stubs that will only be linked 302 into the binary if the real implementation is not present. In a way, 303 the stubs can be thought of as a portable implementation of the weak 304 symbols concept. 305 306 307The following files concur in the definition of which files are linked 308into each emulator: 309 310``default-configs/devices/*.mak`` 311 The files under ``default-configs/devices/`` control the boards and devices 312 that are built into each QEMU system emulation targets. They merely contain 313 a list of config variable definitions such as:: 314 315 include arm-softmmu.mak 316 CONFIG_XLNX_ZYNQMP_ARM=y 317 CONFIG_XLNX_VERSAL=y 318 319``*/Kconfig`` 320 These files are processed together with ``default-configs/devices/*.mak`` and 321 describe the dependencies between various features, subsystems and 322 device models. They are described in :ref:`kconfig` 323 324``default-configs/targets/*.mak`` 325 These files mostly define symbols that appear in the ``*-config-target.h`` 326 file for each emulator [#cfgtarget]_. However, the ``TARGET_ARCH`` 327 and ``TARGET_BASE_ARCH`` will also be used to select the ``hw/`` and 328 ``target/`` subdirectories that are compiled into each target. 329 330.. [#cfgtarget] This header is included by ``qemu/osdep.h`` when 331 compiling files from the target-specific sourcesets. 332 333These files rarely need changing unless you are adding a completely 334new target, or enabling new devices or hardware for a particular 335system/userspace emulation target 336 337 338Support scripts 339--------------- 340 341Meson has a special convention for invoking Python scripts: if their 342first line is ``#! /usr/bin/env python3`` and the file is *not* executable, 343find_program() arranges to invoke the script under the same Python 344interpreter that was used to invoke Meson. This is the most common 345and preferred way to invoke support scripts from Meson build files, 346because it automatically uses the value of configure's --python= option. 347 348In case the script is not written in Python, use a ``#! /usr/bin/env ...`` 349line and make the script executable. 350 351Scripts written in Python, where it is desirable to make the script 352executable (for example for test scripts that developers may want to 353invoke from the command line, such as tests/qapi-schema/test-qapi.py), 354should be invoked through the ``python`` variable in meson.build. For 355example:: 356 357 test('QAPI schema regression tests', python, 358 args: files('test-qapi.py'), 359 env: test_env, suite: ['qapi-schema', 'qapi-frontend']) 360 361This is needed to obey the --python= option passed to the configure 362script, which may point to something other than the first python3 363binary on the path. 364 365 366Stage 3: makefiles 367================== 368 369The use of GNU make is required with the QEMU build system. 370 371The output of Meson is a build.ninja file, which is used with the Ninja 372build system. QEMU uses a different approach, where Makefile rules are 373synthesized from the build.ninja file. The main Makefile includes these 374rules and wraps them so that e.g. submodules are built before QEMU. 375The resulting build system is largely non-recursive in nature, in 376contrast to common practices seen with automake. 377 378Tests are also ran by the Makefile with the traditional ``make check`` 379phony target, while benchmarks are run with ``make bench``. Meson test 380suites such as ``unit`` can be ran with ``make check-unit`` too. It is also 381possible to run tests defined in meson.build with ``meson test``. 382 383Useful make targets 384------------------- 385 386``help`` 387 Print a help message for the most common build targets. 388 389``print-VAR`` 390 Print the value of the variable VAR. Useful for debugging the build 391 system. 392 393Important files for the build system 394==================================== 395 396Statically defined files 397------------------------ 398 399The following key files are statically defined in the source tree, with 400the rules needed to build QEMU. Their behaviour is influenced by a 401number of dynamically created files listed later. 402 403``Makefile`` 404 The main entry point used when invoking make to build all the components 405 of QEMU. The default 'all' target will naturally result in the build of 406 every component. Makefile takes care of recursively building submodules 407 directly via a non-recursive set of rules. 408 409``*/meson.build`` 410 The meson.build file in the root directory is the main entry point for the 411 Meson build system, and it coordinates the configuration and build of all 412 executables. Build rules for various subdirectories are included in 413 other meson.build files spread throughout the QEMU source tree. 414 415``tests/Makefile.include`` 416 Rules for external test harnesses. These include the TCG tests, 417 ``qemu-iotests`` and the Avocado-based acceptance tests. 418 419``tests/docker/Makefile.include`` 420 Rules for Docker tests. Like tests/Makefile, this file is included 421 directly by the top level Makefile, anything defined in this file will 422 influence the entire build system. 423 424``tests/vm/Makefile.include`` 425 Rules for VM-based tests. Like tests/Makefile, this file is included 426 directly by the top level Makefile, anything defined in this file will 427 influence the entire build system. 428 429Dynamically created files 430------------------------- 431 432The following files are generated dynamically by configure in order to 433control the behaviour of the statically defined makefiles. This avoids 434the need for QEMU makefiles to go through any pre-processing as seen 435with autotools, where Makefile.am generates Makefile.in which generates 436Makefile. 437 438Built by configure: 439 440``config-host.mak`` 441 When configure has determined the characteristics of the build host it 442 will write a long list of variables to config-host.mak file. This 443 provides the various install directories, compiler / linker flags and a 444 variety of ``CONFIG_*`` variables related to optionally enabled features. 445 This is imported by the top level Makefile and meson.build in order to 446 tailor the build output. 447 448 config-host.mak is also used as a dependency checking mechanism. If make 449 sees that the modification timestamp on configure is newer than that on 450 config-host.mak, then configure will be re-run. 451 452 The variables defined here are those which are applicable to all QEMU 453 build outputs. Variables which are potentially different for each 454 emulator target are defined by the next file... 455 456 457Built by Meson: 458 459``${TARGET-NAME}-config-devices.mak`` 460 TARGET-NAME is again the name of a system or userspace emulator. The 461 config-devices.mak file is automatically generated by make using the 462 scripts/make_device_config.sh program, feeding it the 463 default-configs/$TARGET-NAME file as input. 464 465``config-host.h``, ``$TARGET-NAME/config-target.h``, ``$TARGET-NAME/config-devices.h`` 466 These files are used by source code to determine what features 467 are enabled. They are generated from the contents of the corresponding 468 ``*.h`` files using the scripts/create_config program. This extracts 469 relevant variables and formats them as C preprocessor macros. 470 471``build.ninja`` 472 The build rules. 473 474 475Built by Makefile: 476 477``Makefile.ninja`` 478 A Makefile include that bridges to ninja for the actual build. The 479 Makefile is mostly a list of targets that Meson included in build.ninja. 480 481``Makefile.mtest`` 482 The Makefile definitions that let "make check" run tests defined in 483 meson.build. The rules are produced from Meson's JSON description of 484 tests (obtained with "meson introspect --tests") through the script 485 scripts/mtest2make.py.