subdirmk - assistance for non-recursive use of make =================================================== Introduction ------------ Peter Miller's 1997 essay _Recursive Make Considered Harmful_ persuasively argues that it is better to arrange to have a single make invocation with the project's complete dependency tree, rather than the currently conventional `$(MAKE) -C subdirectory' approach. However, actually writing a project's build system in a non-recursive style is not very ergonomic. The main difficulties are: - constantly having to write out long file and directory names - the lack of a per-directory make variable namespace means long make variables (or namespace clashes) - it is difficult to arrange that one can cd to a subdirectory and say `make all' and have something reasonable happen (to wit, build an appropriate subset) `subdirmk' is an attempt to solve these problems (and it also slightly alleviates some of the boilerplate needed to support out-of-tree builds well). Basic approach -------------- The developer is expected to write a makefile fragment, in each relevant subdirectory, called `Subdir.sd.mk'. These fragments may contain ordinary make language. Unqualified filenames are relative to the build toplevel, and all commands all run there. However, the sigil & is treated specially. By and large, it refers to `the build directory corresponding to this .sd.mk file', etc. There are a variety of convenient constructions. The result is that to a large extent, the Subdir.sd.mk has an easy way to namespace its "local" make variables, and an easy way to refer to its "local" filenames (and filenames in general). The Subdir.sd.mk's are filtered, fed through autoconf in the usual way (for @..@-substitutions) and included by one autogenerated toplevel makefile. So all of the input is combined and passed to one make invocation. (A corollary is that there is no enforcement of the namespacing: discipline is required to prefix relevant variable names with &, etc.) Each subdirectory is also provided with an autogenerated `Makefile' which exists purely to capture ordinary make invocations and arrange for something suitable to happen. Where there are dependencies between subdirectories, each Subdir.sd.mk can simply refer to files in other subdirectories directly. Invocation, "recursive" per-directory targets --------------------------------------------- Arrangements are made so that when you run `make foo' in a subdirectory, it is like running the whole toplevel makefile, from the toplevel, as `make subdir/foo'. If `subdir/foo' is a file that might be built, that builds it. But `foo' can also be a conventional target like `all'. Each subdirectory has its own `all' target. For example a subdirectory `src' has a target `src/all'. The rules for these are automatically generated from the settings of the per-directory &TARGETS variables. &TARGETS is magic in this way. (In src/Subdir.sd.mk, &TARGETS of course refers to a make variable called src_TARGETS.) The `all' target in a parent directory is taken to imply the `all' targets in all of its subdirectories, recursively. And in the autogenerated stub Makefiles, `all' is the default target. So if you just type `make' in the toplevel, you are asking for `&all' (/all) for every directory in the project. In a parallel build, the rules for all these various subdirectory targets may be in run in parallel: there is only one `make' invocation at a time. There is no sequencing between subdirectories, only been individual targets (as specified according to their dependencies). You can define other per-directory recursive targets too: simply mention (usually, by setting) the variable &TARGETS_zonk, or whatever. This will create a src/zonk target (for appropriate value of src/). Unlike `all', these other targets only exist in areas of the project where at least something mentions them. So for example, if &TARGETS_zonk is mentioned in src but not lib, `make zonk' in lib will fail. If you want to make a target exist everywhere, mention its name in Perdir.sd.mk (see below). Perdir.sd.mk, inclusion ----------------------- The file Perdir.sd.mk in the toplevel of the source is automatically processed after each individual directory's Subdir.sd.mk, and the &-substituted contents therefore appear once for each subdirectory. This lets you do per-directory boilerplate. Some useful boilerplate is already provided in subdirmk, for you to reference like this: &:include subdirmk/cdeps.sd.mk &:include subdirmk/clean.sd.mk For example you could put that in Perdir.sd.mk. The top-level Subdir.sd.mk is the first makefile included after the autogenerated `main.mk' which merely has some basic settings and includes. So if you want to get in early and set global variables, put them near the top of Subdir.sd.mk. subdirmk's filter script itself sets (only) these variables: top_srcdir abs_top_srcdir SUBDIRMK_MAKEFILES MAKEFILE_TEMPLATES You are likely to want to define $(PWD), and shorter names for top_srdir and abs_top_srcdir (we suggest $(src) and $(abs_src)). Global definitions ------------------ If want to set global variables, such as CC, that should only be done once. You can put them in your top-level Subdir.sd.mk, or a separate file you `include' and declare using SUBDIRMK_MAKEFILES. If you need different settings of variables like CC for different subdirectories, you should probably do that with target-specific variable settings. See the info node `(make) Target-specific'. Subdirectory templates `.sd.mk' vs plain autoconf templates `.mk.in' -------------------------------------------------------------------- There are two kinds of template files. Filename .sd.mk .mk.in Processed by &-substitution, autoconf only then autoconf Instantiated Usu. once per subdir Once only Need to be mentioned No, but Subdir.sd.mk All not in subdirmk/ in configure.ac? via SUBDIRMK_SUBDIRS via SUBDIRMK_MAKEFILES How to include `&:include foo.sd.mk' `include foo.mk' in all relevant .sd.mk in only one (but not needed for Subdir.sd.mk Subdir and Perdir) If you `include subdirmk/regen.mk', dependency management and automatic regeneration for all of this template substitution, and for config.status etc. is done for you. Tables of file reference syntaxes --------------------------------- In a nonrecursive makefile supporting out of tree builds there are three separate important distinctions between different file locations: (i) In the build tree, or in the source tree ? (ii) In (or relative to) the subdirectory to which this Subdir.sd.mk relates, or relative to the project's top level ? (iii) Absolute or relative pathname ? Usually relative pathnames suffice. Where an absolute pathname is needed, it can be built out of &/ and an appropriate make variable such as $(PWD). Path construction &-expansions are built from the following: Relative paths in... build source This directory & &, Top level . &; In more detail, with all the various options laid out: Recommended Relative paths in... Absolute paths in... for build source build source This lc &file &,file $(PWD)/&file $(abs_src)/&file directory any &/file &,/file $(PWD)&/file $(abs_src)/&/file several & f g h &, f g h $(addprefix...) Top lc file &;file level any file &;/file $(PWD)/file $(abs_src)/file .mk.in file $(src)/file $(PWD)/file $(abs_src)/file several f g h &; f g h $(addprefix...) (This assumes you have appropriate make variables src, PWD and abs_src.) Substitution syntax ------------------- In general & expands to the subdirectory name when used for a filename, and to the subdirectory name with / replaced with _ for variable names. Note that & is processed *even in makefile comments*. The substitutor does not understand make syntax, or shell syntax, at all. However, the substitution rules are chosen to work well with constructs which are common in makefiles. In the notation below, we suppose that the substitution is being in done in a subdirectory sub/dir of the source tree. In the RH column we describe the expansion at the top level, which is often a special case (in general in variable names we call that TOP rather than the empty string). &CAPS => sub_dir_CAPS or TOP_CAPS &lc => sub/dir/lc or lc Here CAPS is any ASCII letter A-Z and lc is a-z. The assumption is that filenames are usually lowercase and variables usually uppercase. Otherwise, use another syntax: &_ => sub_dir_ or TOP_ &=_ => sub_dir or TOP &/ => sub/dir/ or nothing &=/ => sub/dir or . &,lc => $(top_srcdir)/sub/dir/lc &,/ => $(top_srcdir)/sub/dir/ &;lc => $(top_srcdir)/lc &;/ => $(top_srcdir)/ In general: = return subdir without delimiter (not allowed with `,' `;') , pathname of this subdirectory in source tree ; pathname of top level of source tree / terminates the escape (needed if next is not lwsp or space) lwsp starts multi-word processing (see below) So pathname syntax is a subset of: '&' [ ',' | ';' ] [ lc | '/' ] && => && for convenience in shell runes \& => & general escaping mechanism & thing thing... & &, thing thing... & &; thing thing... & Convenience syntax for prefixing multiple filenames. Introduced by & followed by lwsp where lc could go. Each lwsp-separated non-ws word is prefixed by &/ etc. etc. respectively. No other & escapes are recognised. This processing continues until & preceded by lwsp, or until EOL (the end of the line), or \ then EOL. &: .... recognised at start of line only (possibly after lwsp) args are processed for & &:include filename filename should usually be foo.sd.mk &:-include filename tolerate nonexistent file filenames are relative to $(top_srcdir) &! disables & until EOL (and then disappears) &# delete everything to end of line (useful if the RHS contains unrecognised & constructions) &!STUFF changes the escape sequence from & to literally STUFF STUFF may be any series of of non-whitespace characters, and is terminated by EOL or lwsp. &!STUFF and the lwsp are discarded. After this, write STUFF instead of &, everywhere. The effect is global and lasts until the next setting. It takes effect on &:include'd files too, so maybe set it back before using &:include. Notably STUFFSTUFF => STUFFSTUFF \STUFF => STUFF STUFF!& set escape back to & &TARGETS_things Handled specially. If mentioned, declares that this subdir ought to have a target `things'. The rule will be &/things:: $(&TARGETS_things) You may extend it by adding more :: rules for the target, but the preferred style is to do things like this: &TARGETS_check += & test-passed.stamp It is important to mention &TARGETS_things at least once in the context of each applicable directory, because doing so arranges that the *parent* will also have a `things' target which recursively implies this directory's `things'. Must be spelled exactly &TARGETS_things. &_TARGETS_things, for example, is not magic. But mentioning &TARGETS_things in a #-comment *does* work because the & filter does not care about comments. `all' is extra special: every directory has an `all' target, which corresponds to &TARGETS. Subdirectory and variable naming -------------------------------- The simple variable decoration scheme does not enforce a strict namespace distinction between parts of variable names which come from subdirectory names, and parts that mean something else. So it is a good idea to be a bit careful with your directory naming. `TOP', names that contain `_', and names that are similar to parts of make variables (whether conventional ones, or ones used in your project) are best avoided. If you name your variables in ALL CAPS and your subdirectories in lower case with `-' rather than `_', there will be no confusion. Incorporating this into your project ------------------------------------ Use `git-subtree' to merge the subdirmk/ directory. You may find it useful to symlink the DEVELOPER-CERTIFICATE file (git can store symlinks as symlinks - just `git add' the link). And you probably want to mention the situation in your top-level COPYING. Symlink autogen.sh into your project toplevel. In your configure.ac, say m4_include([subdirmk/subdirmk.ac]) SUBDIRMK_SUBDIRS([...list of subdirectories in relative syntax...]) Write a Subdir.sd.mk in each directory. The toplevel one should probably contain: include subdirmk/usual.mk include subdirmk/regen.mk Write a Perdir.sd.mk in the toplevel, if you want. It should probably have: &:include subdirmk/cdeps.sd.mk &:include subdirmk/clean.sd.mk Hints ----- You can convert your project incrementally. Start with the top-level Makefile.in and rename it to Subdir.sd.mk, and add the appropriate stuff to configure.ac, and fix everything up. Leave the existing $(MAKE) -C for your existing subdirectories alone. Then you can convert individual subdirectories, or classes of subdirectories, at your leisure. (You must be /sure/ that each subdirectory will be entered only once at a time, but your existing recursive make descent system should already do that or you already have concurrency bugs.) Aside from this, be very wary of any invocation of $(MAKE) anywhere. This is a frequent source of concurrency bugs in recursive make build systems. When combined with nonrecursive make it's all in the same directory and there is nothing stopping the different invocations ending up trying to make the same targets at the same time. That causes hideous racy lossage. There are ways to get this to work reliably but it is advanced stuff. If you make syntax errors, or certain kinds of other errors, in your makefiles, you may find that just `make' is broken now and cannot get far enough to regenerate a working set of makefiles. If this happens just rerun ./config.status by hand. Legal information ----------------- subdirmk is Copyright 2019 Mark Wooding Copyright 2019 Ian Jackson subdirmk and its example is free software; you can redistribute it and/or modify it under the terms of the GNU Library General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Library General Public License for more details. You should have received a copy of the GNU Library General Public License along with this library as the file LGPL-2. If not, see https://www.gnu.org/. Individual files generally contain the following tag in the copyright notice, instead of the full licence grant text: SPDX-License-Identifier: LGPL-2.0-or-later As is conventional, this should be read as a licence grant. Contributions are accepted based on the git commit Signed-off-by convention, by which the contributors' certify their contributions according to the Developer Certificate of Origin version 1.1 - see the file DEVELOPER-CERTIFICATE. Where subdirmk is used by and incorporated into another project (eg via git subtree), the directory subdirmk/ is under GNU LGPL-2.0+, and the rest of the project are under that other project's licence(s). (The project's overall licence must be compatible with LGPL-2.0+.)