Autodependencies with GNU make

Scott McPeak, November 2001

The Problem

A compiler compiles a C source file (.c file) plus some header files (.h files) into an object file (.o file). make is a tool to organize a build process, so that whenever a source file changes, the files which depend upon it get rebuilt.

However, while make tends to do a good job of understanding the dependency of .o files upon .c files, it has no built-in facility for determining, nor convenient way of expressing, the dependencies on .h files. What's more, any solution must deal well with source files which are automatically generated.

This article outlines my solution to this problem, which is actually quite simple. I originally wrote this up because I thought it was original, but it turns out Paul Smith had already documented this solution.

The Situation

Suppose I have a (GNU) Makefile like this:

OBJS := foo.o bar.o

# link
proggie: $(OBJS)
	gcc $(OBJS) -o proggie

# compile
%.o: %.c
	gcc -c $(CFLAGS) $*.c -o $*.o

# remove compilation products
clean:
	rm -f proggie *.o

This Makefile says there are two source files, foo.c and bar.c, which are compiled and linked to make the executable proggie. It also describes how, in general, to build a .o file given a .c file.

However, suppose that both foo.c and bar.c include foo.h. That means the respective .o files both depend on the contents of foo.h, but that fact isn't expressed in the Makefile. As a result, if the programmer changes foo.h, the program will likely be inconsistent when rebuilt.

One could of course add more lines like

foo.o: foo.h
bar.o: bar.h

but it's obvious this would become a tedious and error-prone maintenance hassle in anything but the smallest programs.

The Solution

The solution is fairly simple: every time we build a .o file, we also create a file with extension .d (for dependency) that records which files were used to create the corresponding .o file. (Note that we do not create the .d files ahead of time, in contrast to some approaches.) The .d files will be written in the make language itself, and included into the primary Makefile. We can generate the files using the -MM option of gcc (changes in green):

OBJS := foo.o bar.o

# link
proggie: $(OBJS)
	gcc $(OBJS) -o proggie

# pull in dependency info for *existing* .o files
-include $(OBJS:.o=.d)

# compile and generate dependency info
%.o: %.c
	gcc -c $(CFLAGS) $*.c -o $*.o
	gcc -MM $(CFLAGS) $*.c > $*.d

# remove compilation products
clean:
	rm -f proggie *.o *.d

Given the situation described above, we'd get two .d files after compiling. One of them, bar.d, would look like:

bar.o: bar.c foo.h

When make reads this line, since no shell commands are specified, it appends the list of dependencies to any that bar.o already has, without affecting the commands used to build it.

Note that .d files exist if and only if the corresponding .o file exists. This makes sense, since if the .o file doesn't exist yet, we don't need a .d file to tell us it has to be rebuilt.

More subtly, we never try to build a .d file until we have the necessary ingredients to build the corresponding .o file. This is important when a project has some source files built automatically (e.g. Bison output), since any attempt to build the .d files prematurely would fail.

The -include $(OBJS:.o=.d) syntax may need some explanation. First, $(OBJS:.o=.d) takes the value of $(OBJS), and replaces all occurrences of .o at the end of a name with .d. Next, the leading hyphen ("-") means that if some .d files don't exist, make should continue without complaining (again, if the .d file doesn't exist, then neither will the .o file, so the .o file will be properly rebuilt).

An Improvement

The Makefile above has a problem. Suppose I rename foo.h to foo2.h, and change foo.c and bar.c accordingly. When I try to recompile, make will complain that (e.g.) bar.o depends on foo.h, which does not exist. I will have to do make clean or similar to get things working again.

What is needed is a way to say that a particular prerequisite file, if missing, should be treated as changed (so the target will be rebuilt), but not cause an error. GNU make has an obscure feature that does just this: if a file (1) appears as a target in a rule with no prerequisites and no commands, and (2) that file does not exist and cannot be remade, then make will rebuild anything which depends on that file and not report an error. (See Chapter 4 of the make manual, "Rules without Commands or Prerequisites".)

To exploit this feature, dependency generation must rewrite the .d file to list one command-less, prerequisite-less rule for every file named as a dependency. There are several ways to do this; I choose to use a combination of sed and fmt. I've also chosen to prepend the at-sign ("@") to the new commands, so they won't get echoed when make is running (changes in green):

OBJS := foo.o bar.o

# link
proggie: $(OBJS)
	gcc $(OBJS) -o proggie

# pull in dependency info for *existing* .o files
-include $(OBJS:.o=.d)

# compile and generate dependency info;
# more complicated dependency computation, so all prereqs listed
# will also become command-less, prereq-less targets
#   sed:    strip the target (everything before colon)
#   sed:    remove any continuation backslashes
#   fmt -1: list words one per line
#   sed:    strip leading spaces
#   sed:    add trailing colons
%.o: %.c
	gcc -c $(CFLAGS) $*.c -o $*.o
	gcc -MM $(CFLAGS) $*.c > $*.d
	@cp -f $*.d $*.d.tmp
	@sed -e 's/.*://' -e 's/\\$$//' < $*.d.tmp | fmt -1 | \
	  sed -e 's/^ *//' -e 's/$$/:/' >> $*.d
	@rm -f $*.d.tmp

# remove compilation products
clean:
	rm -f proggie *.o *.d

Now dependency files look like this:

bar.o: bar.c foo.h
bar.c:
foo.h:

For those interested in experimenting with the effect of renaming dependencies, I have a tarball with a sample Makefile and a script which simulates the build-rename-build process: example.tar.gz.

One More Tweak

As I later discovered, the above commands don't quite work if the source file (and destination .o file) are in different directories than where make is run. It turns out that gcc -MM will write a dependency file naming bar.o, even when the actual object file it creates is called dir/bar.o.

For example, the system above might create

bar.o: dir/bar.c dir/foo.h
dir/bar.c:
dir/foo.h:

and this won't work because nothing else in the Makefile refers to bar.o.

To work around this (arguably a bug in gcc), one more sed command is needed in the block that builds the dependencies (changes in green again):

%.o: %.c
	gcc -c $(CFLAGS) $*.c -o $*.o
	gcc -MM $(CFLAGS) $*.c > $*.d
	@mv -f $*.d $*.d.tmp
	@sed -e 's|.*:|$*.o:|' < $*.d.tmp > $*.d
	@sed -e 's/.*://' -e 's/\\$$//' < $*.d.tmp | fmt -1 | \
	  sed -e 's/^ *//' -e 's/$$/:/' >> $*.d
	@rm -f $*.d.tmp

This will write a dependency file like

dir/bar.o: dir/bar.c dir/foo.h
dir/bar.c:
dir/foo.h:

Related Work

My starting point for this was the advice in Peter Miller's Recursive Make Considered Harmful. This paper talks about many things besides autodependencies, so it's a good read even if I think the particular autodependency solution in that paper is suboptimal. In particular, Miller's scheme tries to build all the .d files before anything else. I tried this, and ran into lots of problems with automatically-generated source files, for a variety of reasons.

The GNU make manual adopts Miller's solution. My hope is the technique described here might also make its way into the manual.

The inspiration for the close coupling between .o and .d file lifetimes is the Borland C++ compiler, which actually puts the dependency information into the .o file itself (and then optionally caches that info in the GUI). At one point, a company called CodeSourcery had a competition of sorts called Software Carpentry, for alternatives to traditional compile tools (make among others). They have a good list of people's improvements to make on their build tools page (this link is now broken and I don't know of a replacement). You could also try saying "make replacement" to google.

After I originally wrote this, I learned that Paul Smith (the current maintainer of GNU make) already wrote up this technique. My first solution had overlooked the command-less, prerequisite-less rule feature, and instead relied on a patch to make's sources.

Links

Collected important links from the above page for convenience, in no particular order: