GNU Make: From Beginner to Expert
Translated from: https://interrupt.memfault.com/blog/gnu-make-guidelines
GNU Make is a popular and commonly used program for building C language software. It is used to build the Linux kernel and other commonly used GNU/Linux programs and software libraries.
Most embedded software developers will use GNU Make at some point in their careers, either to compile small libraries or to build entire projects. Although there are many options available that can replace Make, it is still often chosen as the build system for new software due to its feature set and widespread support.
This article explains the general concepts and features of GNU Make and includes suggestions on how to make the most out of Make. It is a brief introduction to my favorite and most commonly used Make concepts and features.
What is GNU Make?
GNU Make is a program that can automatically run shell commands and help execute repetitive tasks. It is typically used to transform files into other forms, such as compiling source code files into programs or libraries.
It achieves this by tracking prerequisites and executing a hierarchy of commands to generate targets.
Although the GNU Make manual is lengthy, I recommend reading it as it is the best reference I have found: https://www.gnu.org/software/make/manual/html_node/index.html
When to Choose Make
Make is suitable for building small C/C++ projects or libraries that will be included in the build system of another project. Most build systems have ways to integrate Make-based subprojects.
For larger projects, you may find that more modern build systems are easier to use.
Here are situations where I recommend using a non-Make build system:
When the number of targets (or files) being built is (or will eventually be) in the hundreds. When a “configuration” step is needed to set and save variables, target definitions, and environment configurations. When the project will remain internal or private and will not need to be built by end users. When you find debugging to be a frustrating task. When you need the build to be cross-platform, capable of being built on macOS, Linux, and Windows. In these cases, you might find using CMake, Bazel, Meson, or other modern build systems to be a more pleasant experience.
Invoking Make
Running make will load a file named Makefile from the current directory and attempt to update the default target (which will be detailed later).
Make will sequentially search for files named GNUmakefile, makefile, and makefile.
You can specify a specific makefile using the -f/–file parameter:
$ make -f foo.mkYou can specify any number of targets, listing them as positional parameters:
# Typical targets<br/>$ make clean allYou can pass the Make directory with the -C parameter, which will run Make as if it first changed to that directory.
$ make -C some/sub/directoryFun fact: git can also run with -C to achieve the same effect!
Parallel Invocation
If the -j or -l options are provided, Make can run jobs in parallel. A guideline I have been told is to set the job limit to 1.5 times the number of processor cores:
# a machine with 4 cores: make -jInterestingly, I found that using the -l “load limit” option had slightly better CPU utilization than using the -j “jobs” option. Although YMMV!
There are several ways to programmatically find the current machine’s CPU count. A simple method is to use the python multiprocessing.cpu_count() function to get the number of threads supported by the system (note with hyperthreading systems, this will consume a lot of computer resources, but may be preferable to let it produce infinite work).
# Call the cpu_count() function of python in a subshell<br/>python -c "import multiprocessing; print(multiprocessing.cpu_count())"Output During Parallel Invocation
If Make is executing commands in parallel that produce a lot of output, you may see interleaved output on stdout. To handle this, Make has an option –output-sync.
I recommend using –output-sync=recurse, which will print the full output of the recipe when each target is completed without interleaving outputs from other recipes.
If the recipe uses recursive Make, it will also output the entire recursive Make output together.
Analysis of Makefile
A Makefile contains rules for generating targets. Some basic components of a Makefile are as follows:
# Comments are prefixed with the '#' symbol<br/># A variable assignment<br/>FOO = "hello there!"<br/># A rule creating target "test", with "test.c" as a prerequisite<br/>test: test.c<br/> # The contents of a rule is called the "recipe", and is<br/> # typically composed of one or more shell commands.<br/> # It must be indented from the target name (historically with<br/> # tabs, spaces are permitted)<br/><br/> # Using the variable "FOO"<br/> echo $(FOO)<br/><br/> # Calling the C compiler using a predefined variable naming<br/> # the default C compiler, '$(CC)'<br/> $(CC) test.c -o testLet’s look at each part of the example above.
Variables
Variables use the syntax $(FOO), where FOO is the variable name.
Variables contain plain strings since Make has no other data types. Appending to a variable will add a space and new content:
FOO = one<br/>FOO += two<br/># FOO is now "one two"<br/><br/>FOO = one<br/>FOO = $(FOO)two<br/># FOO is now "onetwo"Variable Assignment
In GNU Make syntax, there are two ways to assign variables:
The expression on the right is assigned literally to the variable—similar to macros in C/C++, where the expression is evaluated when the variable is used:
FOO = 1<br/>BAR = $(FOO)<br/>FOO = 2<br/># prints BAR=2<br/>$(info BAR=$(BAR))Assigning the result of an expression to a variable; the expression is expanded at assignment:
FOO = 1<br/>BAR := $(FOO)<br/>FOO = 2<br/># prints BAR=1<br/>$(info BAR=$(BAR))Note: The $(info…) function above is used to print expressions, which is very useful for debugging makefiles!
Variables that are not explicitly set, implicitly set, or automatically set will evaluate to an empty string.
Environment Variables
Environment variables are carried into the Make execution environment. For example, with the following makefile:
$(info YOLO variable = $(YOLO))If we set the variable YOLO in the shell command while running make, we will set this value:
$ YOLO="hello there!" make<br/>YOLO variable = hello there!<br/>make: *** No targets. Stop.Note: Make prints the “No targets” error because our makefile does not list any targets!
If you use the ?= assignment syntax, Make will assign the value only if the variable has no value:
# Default CC is gcc<br/>CC ?= gccThen we can override $(CC) in the makefile:
$ CC=clang makeAnother common pattern is to allow additional flags to be inserted. In the makefile, we will append to the variable instead of assigning it directly:
CFLAGS += -WallThis allows extra flags to be passed from the environment:
$ CFLAGS='-Werror=conversion -Werror=double-promotion' makeThis is very useful!
Most Important Variables
A special category of variables used is called override variables. Using this command line option will override values set in the environment or in the Makefile!
Makefile:
# any value set elsewhere<br/>YOLO = "not overridden"<br/>$(info $(YOLO))Command:
# setting "YOLO" to different values in the environment + makefile + overriding<br/># variable, yields the overriding value<br/>$ YOLO="environment set" make YOLO='overridden!!'<br/>overridden!!<br/>make: *** No targets. Stop.Target Variables
These variables are only available in the context of recipes. They also apply to any prerequisite recipes!
# set the -g value to CFLAGS<br/># applies to the prog.o/foo.o/bar.o recipes too!<br/>prog : CFLAGS = -g<br/>prog : prog.o foo.o bar.o<br/> echo $(CFLAGS) # will print '-g'Implicit Variables
These are predefined by Make (unless overridden by any other variable type with the same name). Some common examples:
$(CC) - the C compiler (gcc)<br/>$(AR) - archive program (ar)<br/>$(CFLAGS) - flags for the C compiler<br/>Full list here:<br/><a href="https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html">https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html</a>Automatic Variables
These are special variables set by Make that are available in the context of recipes. They are useful for preventing repetitive naming (Don’t Repeat Yourself).
Some common automatic variables:
# $@ : the target name, here it would be "test.txt"<br/>test.txt:<br/> echo HEYO > $@<br/><br/># $^ : name of all the prerequisites<br/>all.zip: foo.txt test.txt<br/> # run the gzip command with all the prerequisites "$^", outputting to the<br/> # name of the target, "$@"<br/> gzip -c $^ > $@<br/>See more at: <a href="https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html">https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html</a>Targets
Targets are to the left of the rule syntax:
target: prerequisite<br/> recipeTargets are almost always named files. This is because Make uses the last modified time to track whether the target is newer than its prerequisites and whether it needs to be rebuilt!
When calling Make, you can specify the target you want to build by listing it as a positional parameter:
# make the 'test.txt' and 'all.zip' targets<br/>make test.txt all.zipIf you do not specify a target in the command, Make will use the first target specified in the makefile, known as the “default target” (which can also be overridden if desired).
Phony Targets
Sometimes it is useful to set meta-targets, such as all, clean, test, etc. In these cases, you do not want Make to check for files named all/clean, etc.
Make provides .PHONY target syntax to mark a target as not pointing to a file:
Assuming our project builds a program and a library foo and foo.a; if we want to create an ‘all’ rule to build both by default:
.PHONY: all<br/>all : foo foo.aIf you have multiple phony targets, a good pattern might be to append each target to the .PHONY that defines it:
# the 'all' rule that builds and tests. Note that it's listed first to make it<br/># the default rule<br/>.PHONY: all<br/>all: build test<br/><br/># compile foo.c into a program 'foo'<br/>foo: foo.c<br/> $(CC) foo.c -o foo<br/><br/># compile foo-lib.c into a library 'foo.a'<br/>foo.a: foo-lib.c<br/> # compile the object file<br/> $(CC) foo-lib.c -c foo-lib.o<br/> # use ar to create a static library containing our object file. using the<br/> # '$@' variable here to specify the rule target 'foo.a'<br/> $(AR) rcs $@ foo-lib.o<br/><br/># a phony rule that builds our project; just contains a prerequisite of the<br/># library + program<br/>.PHONY: build<br/>build: foo foo.a<br/><br/># a phony rule that runs our test harness. has the 'build' target as a<br/># prerequisite! Make will make sure (pardon the pun) the build rule executes<br/># first<br/>.PHONY: test<br/>test: build<br/> ./run-tests.shNote! Phony targets are always considered out of date, so Make will always run these targets’ recipes (thus running any targets that have phony prerequisites)! Use with caution!
Implicit Rules
Implicit rules are provided by Make. I find using them confusing because there are too many behaviors happening behind the scenes. You may occasionally encounter them in the wild, so be cautious.
# this will compile 'test.c' with the default $(CC), $(CFLAGS), into the program<br/># 'test'. it will handle prerequisite tracking on test.c<br/>test: test.o<br/>Full list of implicit rules here:<br/><a href="https://www.gnu.org/software/make/manual/html_node/Catalogue-of-Rules.html">https://www.gnu.org/software/make/manual/html_node/Catalogue-of-Rules.html</a>Pattern Rules
Pattern rules allow you to write a generic rule that applies to multiple targets by pattern matching:
# Note the use of the '$<' automatic variable, specifying the first<br/># prerequisite, which is the .c file<br/>%.o: %.c<br/> $(CC) -c $< -o $@or
OBJ_FILES = foo.o bar.o<br/># Use CC to link foo.o + bar.o into 'program'. Note the use of the '$^'<br/># automatic variable, specifying ALL the prerequisites (all the OBJ_FILES)<br/>program: $(OBJ_FILES)<br/> $(CC) -o $@ $^Prerequisites
As mentioned above, Make will check these targets before running the rules. They can be files or other targets.
If any prerequisite is newer than the target (by modification time), Make will run the target’s rule.
In a C project, you may have a rule that converts C files to object files, and if the C file changes, you want the object file to be regenerated:
foo.o: foo.c<br/> # use automatic variables for the input and output file names<br/> $(CC) $^ -c $@Automatic Prerequisites
For C language projects, a very important consideration is that if the #include header files of a C file change, this will trigger a recompilation. This is done through the gcc/clang -M compiler flag, which outputs a .d file that is then imported with Make include directives.
.d files will contain the necessary prerequisites of the .c files, so any changes to header files will cause a rebuild. Click here for more details:
https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
The basic form might be:
# these are the compiler flags for emitting the dependency tracking file. Note<br/># the usage of the '$<' automatic variable<br/>DEPFLAGS = -MMD -MP -MF $<.d<br/><br/>test.o: test.c<br/> $(CC) $(DEPFLAGS) $< -c $@<br/><br/># bring in the prerequisites by including all the .d files. prefix the line with<br/># '-' to prevent an error if any of the files do not exist<br/>-include $(wildcard *.d)Order-only Prerequisites
These prerequisites will only be built if they do not exist; if they are newer than the target, they will not trigger the target to be rebuilt.
A typical use case is to create a directory for output files; sending files to a directory will update its mtime attribute, but we do not want this to trigger a rebuild.
OUTPUT_DIR = build<br/># output the .o to the build directory, which we add as an order-only<br/># prerequisite- anything right of the | pipe is considered order-only<br/>$(OUTPUT_DIR)/test.o: test.c | $(OUTPUT_DIR)<br/> $(CC) -c $^ -o $@<br/><br/># rule to make the directory<br/>$(OUTPUT_DIR):<br/> mkdir -p $@Recipe
A “recipe” is a list of shell commands to execute when creating a target. They are passed to a subshell (default is /bin/sh). If the target is updated after the recipe runs, the rule is considered successful (but if not updated, it is not considered an error).
foo.txt:<br/> # a simple recipe<br/> echo HEYO > $@If any line in the recipe returns a non-zero exit code, Make will terminate and print an error message. You can prefix the line with a – character to tell Make to ignore non-zero exit codes:
.PHONY: clean<br/>clean:<br/> # we don't care if rm fails<br/> -rm -r ./buildPrefixing the recipe line with @ will suppress echoing that line before execution:
clean:<br/> @# this recipe will just print 'About to clean everything!'<br/> @# prefixing the shell comment lines '#' here also prevents them from<br/> @# appearing during execution<br/> @echo About to clean everything!Make will expand variable/function expressions in the recipe context but will not evaluate them. If you want to access shell variables, use $:
USER = linus<br/><br/>print-user:<br/> # print out the shell variable $USER<br/> echo $$USER<br/><br/> # print out the make variable USER<br/> echo $(USER)Function
The syntax for calling Make functions is as follows:
$(function-name arguments) where arguments is a comma-separated list of parameters.
Built-in Functions There are several functions provided by Make. The most common ones I use are for text manipulation: https://www.gnu.org/software/make/manual/html_node/Text-Functions.html https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html
For example:
FILES=$(wildcard *.c)<br/><br/># you can combine function calls; here we strip the suffix off of $(FILES) with<br/># the $(basename) function, then add the .o suffix<br/>O_FILES=$(addsuffix .o,$(basename $(FILES)))<br/><br/># note that the GNU Make Manual suggests an alternate form for this particular<br/># operation:<br/>O_FILES=$(FILES:.c=.o)User-defined Functions
reverse = $(2) $(1)<br/><br/>foo = $(call reverse,a,b)<br/><br/># recursive wildcard (use it instead of $(shell find . -name '*.c'))<br/># taken from <a href="https://stackoverflow.com/a/18258352">https://stackoverflow.com/a/18258352</a><br/>rwildcard=$(foreach d,$(wildcard $1*),$(call rwildcard,$d/,$2) $(filter $(subst *,%,$2),$d))<br/><br/>C_FILES = $(call rwildcard,.,*.c)Shell Function
You can make Make call a shell expression and capture the result:
TODAYS_DATE=$(shell date –iso-8601)
However, I am cautious when using this feature; it increases the dependency on any program you use, so if you are calling more exotic programs, ensure your build environment is controlled (e.g., in a container or using Conda).
Conditional Expressions in Make
FOO=yolo<br/>ifeq ($(FOO),yolo)<br/>$(info foo is yolo!)<br/>else<br/>$(info foo is not yolo :( )<br/>endif<br/><br/># testing if a variable is set; unset variables are empty<br/>ifneq ($(FOO),) # checking if FOO is blank<br/>$(info FOO is unset)<br/>endif<br/><br/># "complex conditional"<br/>ifeq ($(FOO),yolo)<br/>$(info foo is yolo)<br/>else ifeq ($(FOO), heyo)<br/>$(info foo is heyo)<br/>else<br/>$(info foo is not yolo or heyo :( )<br/>endifMake Include
sources.mk:
SOURCE_FILES := bar.c foo.c \
Makefile:
include sources.mk
OBJECT_FILES = $(SOURCE_FILES:.c=.o)
%.o: %.c^ -o $@
Make Eval
# generate rules for xml->json in some weird world<br/>FILES = $(wildcard inputfile/*.xml)<br/><br/># create a user-defined function that generates rules<br/>define GENERATE_RULE =<br/>$(eval<br/># prereq rule for creating output directory<br/>$(1)_OUT_DIR = $(dir $(1))/$(1)_out<br/>$(1)_OUT_DIR:<br/> mkdir -p $@<br/><br/># rule that calls a script on the input file and produces $@ target<br/>$(1)_OUT_DIR/$(1).json: $(1) | $(1)_OUT_DIR<br/> ./convert-xml-to-json.sh $(1) $@<br/>)<br/><br/># add the target to the all rule<br/>all: $(1)_OUT_DIR/$(1).json<br/>endef<br/># produce the rules<br/>.PHONY: all<br/>all:<br/><br/>$(foreach file,$(FILES),$(call GENERATE_RULE,$(file)))Note that the way of using this feature of Make may be confusing, adding some useful comments to explain what the intent is will be helpful for your future self!
VPATH
VPATH is a special Make variable that contains a list of directories that Make should search when looking for prerequisites and targets.
It can be used to send object files or other derived files to the ./build directory, rather than cluttering the src directory:
# This makefile should be invoked from the temporary build directory, e.g.:<br/># $ mkdir -p build && cd ./build && make -f ../Makefile<br/><br/># Derive the directory containing this Makefile<br/>MAKEFILE_DIR = $(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))<br/><br/># now inform Make we should look for prerequisites from the root directory as<br/># well as the cwd<br/>VPATH += $(MAKEFILE_DIR)<br/><br/>SRC_FILES = $(wildcard $(MAKEFILE_DIR)/src/*.c)<br/><br/># Set the obj file paths to be relative to the cwd<br/>OBJ_FILES = $(subst $(MAKEFILE_DIR)/,,$(SRC_FILES:.c=.o))<br/><br/># now we can continue as if Make was running from the root directory, and not a<br/># subdirectory<br/><br/># $(OBJ_FILES) will be built by the pattern rule below<br/>foo.a: $(OBJ_FILES)<br/> $(AR) rcs $@ $(OBJ_FILES)<br/><br/># pattern rule; since we added ROOT_DIR to VPATH, Make can find prerequisites<br/># like `src/test.c` when running from the build directory!<br/>%.o: %.c<br/> # create the directory tree for the output file 👍<br/> echo $@<br/> # compile<br/> $(CC) -c $^ -o $@Touch File
# our tools are stored in tools.tar.gz, and downloaded from a server<br/>TOOLS_ARCHIVE = tools.tar.gz<br/>TOOLS_URL = https://httpbin.org/get<br/><br/># the rule to download the tools using wget<br/>$(TOOLS_ARCHIVE):<br/> wget $(TOOLS_URL) -O $(TOOLS_ARCHIVE)<br/><br/># rule to unpack them<br/>tools-unpacked.dummy: $(TOOLS_ARCHIVE)<br/> # running this command results in a directory.. but how do we know it<br/> # completed, without a file to track?<br/> tar xzvf $^<br/> # use the touch command to record completion in a dummy file<br/> touch $@Debugging Makefile
For small issues, I typically use the Make equivalent of printf, which is the $(info/warning/error) function, for example when checking a condition path that is not working:
ifeq ($(CC),clang)<br/>$(error whoops, clang not supported!)<br/>endifTo debug why a rule runs when it shouldn’t (or vice versa), you can use the –debug option:
https://www.gnu.org/software/make/manual/html_node/Options-Summary.html
I recommend redirecting stdout to a file when using this option, as it generates a lot of output.
Profile
For profiling a make invocation (e.g., for attempting to improve compilation times), this tool can be useful:
https://github.com/rocky/remake
Check out the tips here for compilation-related performance improvements:
https://interrupt.memfault.com/blog/improving-compilation-times-c-cpp-projects
Verbose Flag
# Makefile for building the 'example' binary from C sources<br/><br/># Verbose flag<br/>ifeq ($(V),1)<br/>Q :=<br/>else<br/>Q := @<br/>endif<br/><br/># The build folder, for all generated output. This should normally be included<br/># in a .gitignore rule<br/>BUILD_FOLDER := build<br/><br/># Default all rule will build the 'example' target, which here is an executable<br/>.PHONY:<br/>all: $(BUILD_FOLDER)/example<br/><br/># List of C source files. Putting this in a separate variable, with a file on<br/># each line, makes it easy to add files later (and makes it easier to see<br/># additions in pull requests). Larger projects might use a wildcard to locate<br/># source files automatically.<br/>SRC_FILES = \
src/example.c \
src/main.c<br/><br/># Generate a list of .o files from the .c files. Prefix them with the build<br/># folder to output the files there<br/>OBJ_FILES = $(addprefix $(BUILD_FOLDER)/,$(SRC_FILES:.c=.o))<br/><br/># Generate a list of depfiles, used to track includes. The file name is the same<br/># as the object files with the .d extension<br/>DEP_FILES = $(addsuffix .d,$(OBJ_FILES))<br/><br/># Flags to generate the .d dependency-tracking files when we compile. It's<br/># named the same as the target file with the .d extension<br/>DEPFLAGS = -MMD -MP -MF [email protected]<br/><br/># Include the dependency tracking files<br/>-include $(DEP_FILES)<br/><br/># List of include dirs. These are put into CFLAGS.<br/>INCLUDE_DIRS = \
src/<br/><br/># Prefix the include dirs with '-I' when passing them to the compiler<br/>CFLAGS += $(addprefix -I,$(INCLUDE_DIRS))<br/><br/># Set some compiler flags we need. Note that we're appending to the CFLAGS<br/># variable<br/>CFLAGS += \
-std=c11 \
-Wall \
-Werror \
-ffunction-sections -fdata-sections \
-Og \
-g3<br/><br/># Our project requires some linker flags: garbage collect sections, output a<br/># .map file<br/>LDFLAGS += \
-Wl,--gc-sections,-Map,[email protected]<br/><br/># Set LDLIBS to specify linking with libm, the math library<br/>LDLIBS += \
-lm<br/><br/># The rule for compiling the SRC_FILES into OBJ_FILES<br/>$(BUILD_FOLDER)/%.o: %.c<br/> @echo Compiling $(notdir $<)<br/> @# Create the folder structure for the output file<br/> @mkdir -p $(dir $@)<br/> $(Q) $(CC) $(CFLAGS) $(DEPFLAGS) -c $< -o $@<br/><br/># The rule for building the executable "example", using OBJ_FILES as<br/># prerequisites. Since we're not relying on an implicit rule, we need to<br/># explicity list CFLAGS, LDFLAGS, LDLIBS<br/>$(BUILD_FOLDER)/example: $(OBJ_FILES)<br/> @echo Linking $(notdir $@)<br/> $(Q) $(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@<br/><br/># Remove debug information for a smaller executable. An embedded project might<br/># instead using [arm-none-eabi-]objcopy to convert the ELF file to a raw binary<br/># suitable to be written to an embedded device<br/>STRIPPED_OUTPUT = $(BUILD_FOLDER)/example-stripped<br/><br/>$(STRIPPED_OUTPUT): $(BUILD_FOLDER)/example<br/> @echo Stripping $(notdir $@)<br/> $(Q)objcopy --strip-debug $^ $@<br/><br/># Since all our generated output is placed into the build folder, our clean rule<br/># is simple. Prefix the recipe line with '-' to not error if the build folder<br/># doesn't exist (the -f flag for rm also has this effect)<br/>.PHONY: clean<br/>clean:<br/> - rm -rf $(BUILD_FOLDER)$ V=1 make
Make Suggestions
Here is a list of suggestions to make the most out of Make:
Targets should typically be real files. Always use @ as the recipe output path when issuing sub-MAKE commands, so your rules and Make’s paths are the same. Feel free to use comments in the makefile, especially when using complex behaviors or subtle syntax. Your colleagues (and future self) will thank you. Use -j or -l options to run Make in parallel! Avoid using the touch command to track rule completion.
Others
You may also encounter automake in open source projects (look for ./configure scripts). This is a related tool for generating makefiles, worth looking into (especially if you are writing C software that needs to be widely portable).
Today there are many competitors to GNU Make, and I encourage everyone to research them. Some examples:
CMake is very popular (used by the Zephyr project) and worth looking into. It makes out-of-tree builds very easy. Bazel uses a declarative syntax (vs. Make’s imperative approach). Meson is a meta-builder like cmake, but defaults to using Ninja as a backend, which can be very fast.