From: Ian Jackson Date: Sun, 16 Feb 2020 18:45:25 +0000 (+0000) Subject: Merge subdirmk 0.4 X-Git-Tag: v0.6.1~24 X-Git-Url: http://www.chiark.greenend.org.uk/ucgi/~ianmdlvl/git?p=secnet.git;a=commitdiff_plain;h=bf60f6943fca67ea86acdcb12db34e74fe43175c Merge subdirmk 0.4 git subtree pull. Fix up test-example/Dir.sd.mk for the incompatible change (`&${' needs to become `&{'). Signed-off-by: Ian Jackson --- bf60f6943fca67ea86acdcb12db34e74fe43175c diff --cc subdirmk/README index 3d18dc2,0000000..f79a280 mode 100644,000000..100644 --- a/subdirmk/README +++ b/subdirmk/README @@@ -1,551 -1,0 +1,575 @@@ +subdirmk - assistance for non-recursive use of make +=================================================== + +Introduction +------------ + +Peter Miller's 1997 essay _Recursive Make Considered Harmful_ +persuasively argues that it is better to arrange to have a single +make invocation with the project's complete dependency tree, rather +than the currently conventional `$(MAKE) -C subdirectory' approach. + +However, actually writing a project's build system in a non-recursive +style is not very ergonomic. The main difficulties are: + - constantly having to write out long file and directory names + - the lack of a per-directory make variable namespace means + long make variables (or namespace clashes) + - it is difficult to arrange that one can cd to a subdirectory + and say `make all' and have something reasonable happen + (to wit, build an appropriate subset) + +`subdirmk' is an attempt to solve these problems (and it also slightly +alleviates some of the boilerplate needed to support out-of-tree +builds well, and helps a bit with metaprogramming and rule writing). + +Basic approach +-------------- + +The developer is expected to write a makefile fragment, in each +relevant subdirectory, called `Dir.sd.mk'. + +These fragments may contain ordinary make language. Unqualified +filenames are relative to the build toplevel, and all commands all run +there. + +However, the sigil & is treated specially. By and large, it refers to +`the build directory corresponding to this .sd.mk file', etc. +There are a variety of convenient constructions. + +The result is that to a large extent, the Dir.sd.mk has an easy way +to namespace its "local" make variables, and an easy way to refer to +its "local" filenames (and filenames in general). + +The Dir.sd.mk's are filtered, fed through autoconf in the usual way +(for @..@-substitutions) and included by one autogenerated toplevel +makefile. + +So all of the input is combined and passed to one make invocation. +(A corollary is that there is no enforcement of the namespacing: +discipline is required to prefix relevant variable names with &, etc.) + +Each subdirectory is also provided with an autogenerated `Makefile' +which exists purely to capture ordinary make invocations and arrange +for something suitable to happen. + +Where there are dependencies between subdirectories, each Dir.sd.mk +can simply refer to files in other subdirectories directly. + +Substitution syntax +------------------- + +In general & expands to the subdirectory name when used for a +filename, and to the subdirectory name with / replaced with _ for +variable names. (If your variables start with capital letters and +your filenames with lowercase. Otherwise, use &/ or &_.) + +Note that & is processed *even in makefile comments*. The substitutor +does not understand make syntax, or shell syntax, at all. However, +the substitution rules are chosen to work well with constructs which +are common in makefiles. + +In the notation below, we suppose that the substitution is being in +done in a subdirectory sub/dir of the source tree. In the RH column +we describe the expansion at the top level, which is often a special +case (in general in variable names we call that TOP rather than the +empty string). + +&CAPS => sub_dir_CAPS or TOP_CAPS +&lc => sub/dir/lc or lc + Here CAPS is any ASCII letter A-Z and lc is a-z. + The assumption is that filenames are usually lowercase and + variables usually uppercase. Otherwise, use another syntax: + +&/ => sub/dir/ or nothing +&_ => sub_dir_ or TOP_ +&. => sub/dir or . + (This implies that `&./' works roughly like `&/', although + it can produce a needless `./') + +&= => sub_dir or TOP + +&^lc => ${top_srcdir}/sub/dir/lc +&^/ => ${top_srcdir}/sub/dir/ +&^. => ${top_srcdir}/sub/dir + +&~lc => ${top_srcdir}/lc +&~/ => ${top_srcdir}/ +&~. => ${top_srcdir} + +In general: + ^ pathname of this subdirectory in source tree + ~ pathname of top level of source tree + / terminates the path escape } needed if next is + _ terminates the var escape } not letter or space) + . terminates path escape giving dir name (excluding /) + = terminates var escape giving only prefix part (rarely needed) + lwsp starts multi-word processing (see below) + +So pathname syntax is a subset of: + '&' [ '^' | '~' ] [ lc | '/' | '.' ] + +&& => && for convenience in shell runes + +&\& => & general escaping mechanism +&\$ => $ provided for $-doubling regimes +&\NEWLINE eats the newline and vanishes + +&$VARIABLE => ${sub_dir_VARIABLE} or ${TOP_VARIABLE} + VARIABLE is ASCII starting with a letter and matching \w+ + +& thing thing... & +&^ thing thing... & +&~ thing thing... & + Convenience syntax for prefixing multiple filenames. + Introduced by & followed by lwsp where lc could go. + Each lwsp-separated non-ws word is prefixed by &/ etc. + etc. respectively. No other & escapes are recognised. + This processing continues until & preceded by lwsp, + or until EOL (the end of the line), or \ then EOL. + +&: .... + recognised at start of line only (possibly after lwsp) + ++&: => &: ++ for make multiple targets syntax ++ recognised anywhere *except* start of line ++ +&:include filename filename should usually be [&]foo.sd.mk +&:-include filename tolerate nonexistent file + RHS is &-expanded but filenames are relative to the top + srcdir. This implies that unqualified names are like &~/ + whereas &/ is like &^/. &^ and &~ do not work here because + they expand to constructions involving literally + `$(top_srcdir)', but the RHS is not make-expanded. + +&! disables & until EOL (and then disappears) + +&# delete everything to end of line + (useful if the RHS contains unrecognised & constructions) + +&TARGETS_things + Handled specially. If mentioned at the start of a line + (possibly following whitespace), declares that this + subdir ought to have a target `things'. The rule will be + &/things:: $(&TARGETS_things) + + You may extend it by adding more :: rules for the target, + but the preferred style is to do things like this: + &TARGETS_check += & test-passed.stamp + + It is important to mention &TARGETS_things at least once in + the context of each applicable directory, because doing so + arranges that the *parent* will also have a `things' target + which recursively implies this directory's `things'. + + Must be spelled exactly &TARGETS_things. &_TARGETS_things, + for example, is not magic. To make the target exist + without providing any prerequisites for it, write a line + containing just `&TARGETS_things +='. + + `all' is extra special: every directory has an `all' + target, which corresponds to &TARGETS. + ++Directives ++- - - - - ++ +&:warn [!]WARNTAG ... + Suppress (with !) or re-enable (without !) warnings tagged + WARNTAG (see section `Warnings', below). The suppression list + is reset at the start of processing in each subdirectory. + Warnings that appear at the end of processing are controlled + by the final warning state after processing all the toplevel + input files (including Final.sd.mk). + +&:local+global [!][&]VARIABLE ... + Suppresses any warnings relating to forthcoming mentions + to VARIABLE or &VARIABLE, as applicable. Scope ends at + the end of the current directory's Suffix.sd.mk. + Prefixing with ! removes [&]VARIABLE from the suppresion list. + +&:changequote NEWQUOTE + changes the escape sequence from & to literally NEWQUOTE + NEWQUOTE may be any series of of non-whitespace characters, + and is terminated by EOL or lwsp. The whole line is + discarded. + + After this, write NEWQUOTE instead of &, everywhere. + The effect is unscoped and lasts until the next setting, + or until the end of the current directory's Suffix.sd.mk. + It takes effect on &:include'd files too, so maybe set + it back before using &:include. + + Notably + NEWQUOTENEWQUOTE => NEWQUOTENEWQUOTE + NEWQUOTE\NEWQUOTE => NEWQUOTE + NEWQUOTE\$ => $ + NEWQUOTE:changequote & set escape back to & + + +Dollar doubling and macro assistance +------------------------------------ + +&$+ Starts dollar-doubling +&$- Stops dollar-doubling + Both are idempotent and local to the file or context. + +This is useful both for make macrology involving $(eval ...), and +possibly for helping write complicated recipes involving shell +variables, inline Perl code, etc. + +Sometimes we will show $'s being doubled inside another construct. +This means the content of the construct is $-doubled: $-doubling is +locally enabled, and restored afterwards. + +&:macro NAME => define NAME +STUFF $ THINGS .. STUFF $$ THINGS +&:endm .. endef + NAME is processed for & + - &${..$..} => ${eval ${call ..$$..}} ++&{..$..} => ${eval ${call ..$$..}} + (matches { } pairs to find the end) + content is $-doubled (unless it contains &$- to turn that off) ++ contrast &(...), see "Convenience syntax for call", below. + - Together &:macro and &${...} provide a more reasonable macro facility ++Together &:macro and &{...} provide a more reasonable macro facility +than raw make. They solve the problem that make expansions cannot - directly generate multiple rules, variable, etc.; instead, `$(eval )' ++directly generate multiple rules, variables, etc.; instead, `$(eval )' +must be used, but that re-expands the argument, meaning that all the +literal text must be $-doubled. This applies to the macro text and to +the arguments. Also `$(eval $(call ...))' is an unfortunate syntax. - Hence &:macro and &${...}. ++Hence &:macro and &{...}. + +While dollar-doubling: +- - - - - - - - - - - + +$ => $$ including $'s produced by other + &-expansions not mentioned here + +&\$ => $ - &$( => ${ (expands to { } so it is useable for shell too) ++&$( => $( +&$NN => ${NN} where N are digits + +A few contexts do not support $-doubling, such as directive arguments +or places where this might imply $-quadrupling. (There is no way to +get $-quadrupling.) + ++Convenience syntax for call ++- - - - - - - - - - - - - - ++ ++&(thing => $(call thing ++&( thing => $(call thing ++ and specially: ++&(&lc => $(call sub_dir_lc or $(call TOP_lc ++&( &lc => $(call sub_dir_lc or $(call TOP_lc ++ even though lc would normally be thought a filename ++ ++Unlike &{...}, this does not involve any dollar-doubling. ++ ++Use this when the expansion is going to be a piece of text to be used ++as part of a rule, filename, etc. When the expansion is top-level ++make text (eg, rules), use &:macro and &{...}. ++ + +Invocation, "recursive" per-directory targets +--------------------------------------------- + +Arrangements are made so that when you run `make foo' in a +subdirectory, it is like running the whole toplevel makefile, from the +toplevel, as `make subdir/foo'. If `subdir/foo' is a file that might +be built, that builds it. + +But `foo' can also be a conventional target like `all'. + +Each subdirectory has its own `all' target. For example a +subdirectory `src' has a target `src/all'. The rules for these are +automatically generated from the settings of the per-directory +&TARGETS variables. &TARGETS is magic in this way. (In +src/Dir.sd.mk, &TARGETS of course refers to a make variable called +src_TARGETS.) + +The `all' target in a parent directory is taken to imply the `all' +targets in all of its subdirectories, recursively. And in the +autogenerated stub Makefiles, `all' is the default target. So if you +just type `make' in the toplevel, you are asking for `&all' +(/all) for every directory in the project. + +In a parallel build, the rules for all these various subdirectory +targets may be in run in parallel: there is only one `make' invocation +at a time. There is no sequencing between subdirectories, only been +individual targets (as specified according to their dependencies). + +You can define other per-directory recursive targets too: set the +variable &TARGETS_zonk, or whatever (being sure to write &TARGETS_zonk +at the start of a line). This will create a src/zonk target (for +appropriate value of src/). Unlike `all', these other targets only +exist in areas of the project where at least something mentions them. +So for example, if &TARGETS_zonk is set in src but not lib, `make +zonk' in lib will fail. If you want to make a target exist +everywhere, += it with nothing in Prefix.sd.mk or Suffix.sd.mk (see +below). + +Prefix.sd.mk, Suffix.sd.mk, Final.sd.mk, inclusion +-------------------------------------------------- + +The files Prefix.sd.mk and Suffix.sd.mk in the toplevel of the source +are automatically processed before and after each individual +directory's Dir.sd.mk, and the &-substituted contents therefore +appear once for each subdirectory. + +This lets you do per-directory boilerplate. Some useful boilerplate +is already provided in subdirmk, for you to reference like this: + &:include subdirmk/cdeps.sd.mk + &:include subdirmk/clean.sd.mk +For example you could put that in Suffix.sd.mk. + +The top-level Dir.sd.mk is the first makefile included after the +autogenerated `main.mk' which merely has some basic settings and +includes. So if you want to get in early and set global variables, +put them near the top of Dir.sd.mk. + +The file Final.sd.mk in the toplevel directory is processed and +the result included after all the other files. Its subdirmk +filtering context inherits warning suppressions from the toplevel's +Dir.sd.mk etc., but not anything else. + +subdirmk's filter script itself sets (only) these variables: + top_srcdir + abs_top_srcdir + SUBDIRMK_MAKEFILES + MAKEFILE_TEMPLATES +You are likely to want to define $(PWD), and shorter names for +top_srdir and abs_top_srcdir (we suggest $(src) and $(abs_src)). + +Warnings +-------- + +subdirmk's `generate' program, which does the acual &-substitution, +can produce some warnings about your .sd.mk files. These can be +suppressed with the &:warn directive. The warning tags are: + + local+global + The same VARNAME was used both with and without an & prefix. + This can be confusing. Also, if you avoid this then you will + get a warning iff you accidentally leave off a needed &. + The generation of this warning depends on scanning your + makefile for things that look like variable references, which + subdirmk does not do completely perfectly. Exciting make + syntax may evade this warning, or require suppressions. + (You can suppress this warning for a particular VARNAME with + the &:local+global directive.) + + single-char-var + A variable expansion like $FBAR. make's expansion rules + interpret this as $(F)BAR. It's normally better to write + it this way, at least if the variable expansion is followed + by more letters. Note that &$FOO works differently to + raw make: it expands to ${sub_dir_FOO}. + + broken-var-ref + An attempt at variable expansion looking like $&... + You probably expected this to mean $(TOP_F)BAR but it + expands to $TOP_FBAR which make thinks means $(T)OP_FBAR. + + unknown-warning + &:warn was used to try to enable a warning that this version + of subdirmk does not understand. (Note that an attempt to + *dis*able an unknown warning is only reported if some other + warning was issued which might have been disabled.) + + +Guides, hints, and further explanations +======================================= + +Incorporating this into your project +------------------------------------ + +Use `git-subtree' to merge the subdirmk/ directory. You may find it +useful to symlink the DEVELOPER-CERTIFICATE file (git can store +symlinks as symlinks - just `git add' the link). And you probably +want to mention the situation in your top-level COPYING and HACKING. + +Symlink autogen.sh into your project toplevel. + +In your configure.ac, say + + m4_include([subdirmk/subdirmk.ac]) + SUBDIRMK_SUBDIRS([...list of subdirectories in relative syntax...]) + +Write a Dir.sd.mk in each directory. See the substitution syntax +reference, above, and the example/ directory here. The toplevel +Dir.sd.mk should probably contain: + + include subdirmk/usual.mk + include subdirmk/regen.mk + +Write a Suffix.sd.mk in the toplevel, if you want. It should probably +have: + + &:include subdirmk/cdeps.sd.mk + &:include subdirmk/clean.sd.mk + + +Hints +----- + +You can convert your project incrementally. Start with the top-level +Makefile.in and rename it to Dir.sd.mk, and add the appropriate +stuff to configure.ac, and fix everything up. Leave the existing +$(MAKE) -C for your existing subdirectories alone. Then you can +convert individual subdirectories, or classes of subdirectories, at +your leisure. (You must be /sure/ that each recursive (non-subdirmk) +subdirectory will be entered only once at a time, but your existing +recursive make descent system should already do that or you already +have concurrency bugs.) + +Aside from this, be very wary of any invocation of $(MAKE) anywhere. +This is a frequent source of concurrency bugs in recursive make build +systems. When combined with nonrecursive make it's all in the same +directory and there is nothing stopping the different invocations +ending up trying to make the same targets at the same time. That +causes hideous racy lossage. There are ways to get this to work +reliably but it is advanced stuff. + +If you make syntax errors, or certain kinds of other errors, in your +makefiles, you may find that just `make' is broken now and cannot get +far enough to regenerate a working set of makefiles. If this happens +just rerun ./config.status by hand. + +If you go back and forth between different versions of your code you +can sometimes find that `make' complains that one of your Dir.sd.mk +files is missing: typically, if iot was used and therefore a +dependency in some other version of your code. If you run `make +clean' (or `make realclean') these dependencies are suppressed, which +will clear up the problem. + + +Global definitions +------------------ + +If want to set global variables, such as CC, that should only be done +once. You can put them in your top-level Dir.sd.mk, or a separate +file you `include' and declare using SUBDIRMK_MAKEFILES. + +If you need different settings of variables like CC for different +subdirectories, you should probably do that with target-specific +variable settings. See the info node `(make) Target-specific'. + + +Directory templates `.sd.mk' vs plain autoconf templates `.mk.in' +-------------------------------------------------------------------- + +There are two kinds of template files. + + Filename .sd.mk .mk.in + + Processed by &-substitution, autoconf only + then autoconf + + Instantiated Usu. once per subdir Once only + + Need to be mentioned No, but Dir.sd.mk All not in subdirmk/ + in configure.ac? via SUBDIRMK_SUBDIRS via SUBDIRMK_MAKEFILES + + How to include `&:include foo.sd.mk' `include foo.mk' + in all relevant .sd.mk in only one + (but not needed for Dir.sd.mk + Prefix, Suffix, Final) + +If you `include subdirmk/regen.mk', dependency management and +automatic regeneration for all of this template substitution, and for +config.status etc. is done for you. + + +Tables of file reference syntaxes +--------------------------------- + +In a nonrecursive makefile supporting out of tree builds there are +three separate important distinctions between different file +locations: + + (i) In the build tree, or in the source tree ? + + (ii) In (or relative to) the subdirectory to which this Dir.sd.mk + relates, or relative to the project's top level ? + + (iii) Absolute or relative pathname ? Usually relative pathnames + suffice. Where an absolute pathname is needed, it can be built + out of &/ and an appropriate make variable such as $(PWD). + +Path construction &-expansions are built from the following: + + Relative paths in... + build source + + This directory & &^ + Top level . &~ + +In more detail, with all the various options laid out: + + Recommended Relative paths in... Absolute paths in... + for build source build source + + This lc &file &^file $(PWD)/&file $(abs_src)/&file + directory any &/file &^/file $(PWD)/&/file $(abs_src)/&/file + several & f g h &^ f g h $(addprefix...) + + Top lc file &~file + level any file &~/file $(PWD)/file $(abs_src)/file + .mk.in file $(src)/file $(PWD)/file $(abs_src)/file + several f g h &~ f g h $(addprefix...) + +(This assumes you have appropriate make variables src, PWD and +abs_src.) + + +Subdirectory and variable naming +-------------------------------- + +The simple variable decoration scheme does not enforce a strict +namespace distinction between parts of variable names which come from +subdirectory names, and parts that mean something else. + +So it is a good idea to be a bit careful with your directory naming. +`TOP', names that contain `_', and names that are similar to parts of +make variables (whether conventional ones, or ones used in your +project) are best avoided. + +If you name your variables in ALL CAPS and your subdirectories in +lower case with `-' rather than `_', there will be no confusion. + + +Legal information +================= + +subdirmk is ++ Copyright 2019-2020 Ian Jackson + Copyright 2019 Mark Wooding - Copyright 2019 Ian Jackson + + subdirmk and its example is free software; you can redistribute it + and/or modify it under the terms of the GNU Library General Public + License as published by the Free Software Foundation; either + version 2 of the License, or (at your option) any later version. + + This is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Library General Public License for more details. + + You should have received a copy of the GNU Library General Public + License along with this library as the file LGPL-2. + If not, see https://www.gnu.org/. + +Individual files generally contain the following tag in the copyright +notice, instead of the full licence grant text: + SPDX-License-Identifier: LGPL-2.0-or-later +As is conventional, this should be read as a licence grant. + +Contributions are accepted based on the git commit Signed-off-by +convention, by which the contributors' certify their contributions +according to the Developer Certificate of Origin version 1.1 - see +the file DEVELOPER-CERTIFICATE. + +Where subdirmk is used by and incorporated into another project (eg +via git subtree), the directory subdirmk/ is under GNU LGPL-2.0+, and +the rest of the project are under that other project's licence(s). +(The project's overall licence must be compatible with LGPL-2.0+.) diff --cc test-example/Dir.sd.mk index 2e79067,0000000..ff34215 mode 100644,000000..100644 --- a/test-example/Dir.sd.mk +++ b/test-example/Dir.sd.mk @@@ -1,34 -1,0 +1,34 @@@ +&TARGETS += & sites.conf sites-nonego.conf + +include common.make + +&/%.key: &^/%.key.b64 + base64 -d <$< >$@.new && mv -f $@.new $@ + +&sites-nonego.conf: $(src)/make-secnet-sites &^/sites &/Dir.mk + $(src)/make-secnet-sites --output-version=1 &^/sites $@ + +&sites.conf: $(src)/make-secnet-sites &^/sites &/Dir.mk + mkdir -p &pubkeys + &~/make-secnet-sites --pubkeys-dir=&pubkeys --pubkeys-install \ + &^/sites $@.tmp && mv -f $@.tmp $@ + +&clean:: + rm -rf &pubkeys + +&:macro &privkey +&/&$1.privkeys/priv.&$2: &/&$3 + mkdir -p $(dir $@) && cp $< $@.tmp && mv -f $@.tmp $@ +&PRIVKEYS += &/&$3 &/&$1.privkeys/priv.&$2 +&clean:: + rm -rf &/&$1.privkeys +&:endm + - &${&privkey,outside,5dc36a4700,rsa1-sites2.key} - &${&privkey,outside,0000000000,outside.key} - &${&privkey,inside,0000000000,inside.key} ++&{&privkey,outside,5dc36a4700,rsa1-sites2.key} ++&{&privkey,outside,0000000000,outside.key} ++&{&privkey,inside,0000000000,inside.key} + +&all-privkeys:: $(&PRIVKEYS) + +&TARGETS += $(&PRIVKEYS) +&CLEAN += *.new