From: Ian Jackson Date: Thu, 18 Jun 2020 18:03:25 +0000 (+0100) Subject: README.md: documentation X-Git-Tag: nailing-cargo/1.0.0~177 X-Git-Url: http://www.chiark.greenend.org.uk/ucgi/~ianmdlvl/git?a=commitdiff_plain;h=4e6e739d308290e3d47f2d7c1e94945ca78e98e9;p=nailing-cargo.git README.md: documentation Signed-off-by: Ian Jackson --- diff --git a/DEVELOPER-CERTIFICATE b/DEVELOPER-CERTIFICATE new file mode 100644 index 0000000..912d22e --- /dev/null +++ b/DEVELOPER-CERTIFICATE @@ -0,0 +1,38 @@ +Developer Certificate of Origin +Version 1.1 + +Copyright (C) 2004, 2006 The Linux Foundation and its contributors. +1 Letterman Drive +Suite D4700 +San Francisco, CA, 94129 + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + + +Developer's Certificate of Origin 1.1 + +By making a contribution to this project, I certify that: + +(a) The contribution was created in whole or in part by me and I + have the right to submit it under the open source license + indicated in the file; or + +(b) The contribution is based upon previous work that, to the best + of my knowledge, is covered under an appropriate open source + license and I have the right under that license to submit that + work with modifications, whether created in whole or in part + by me, under the same open source license (unless I am + permitted to submit under a different license), as indicated + in the file; or + +(c) The contribution was provided directly to me by some other + person who certified (a), (b) or (c) and I have not modified + it. + +(d) I understand and agree that this project and the contribution + are public and that a record of the contribution (including all + personal information I submit with it, including my sign-off) is + maintained indefinitely and may be redistributed consistent with + this project or the open source license(s) involved. + diff --git a/README.md b/README.md new file mode 100644 index 0000000..1f5e640 --- /dev/null +++ b/README.md @@ -0,0 +1,357 @@ +nailing-cargo +============= + +This is a wrapper tool for `cargo`, the Rust build tool and package +manager. Functions: + + * Conveniently use local crates, including completely + unpublished crates. + + * Perform out-of-tree builds, including in an account with + no write access to the source tree. + + * Provide convenience aliases for target architecture names. + + * Makes the default be offline (ie, not to access the internet) + +These functions are of course configurable. + +The primary source of information for `nailing-cargo` is the file +`../Cargo.nail` (which is in TOML syntax). You put `Cargo.nail` +alongside the top-level git repositories you are working with, and +invoke nailing-cargo from the git directory containing the Rust +package you want to build. + +Installing +---------- + +nailing-cargo is designed to be run out of a git clone: + +``` +$ git clone https://salsa.debian.org/iwj/nailing-cargo.git +$ ln -s `pwd`/nailing-cargo/nailing-cargo ~/bin +``` + +It is self-contained, depending only on a reasonably functional Perl +installation. + +Most basic example usage +------------------------ + +``` +$ cd myproject +$ cat >../Cargo.nail +subdirs=""" +myproject +mylibrary +""" +$ nailing-cargo cargo generate-lockfile +$ nailing-cargo cargo build +``` + +Using local crates, or locally modified crates +============================================== + +cargo does not work well with local crates, +especially completely unpublished ones. +(See [issue#6713](https://github.com/rust-lang/cargo/issues/6713), +[stackoverflow](https://stackoverflow.com/questions/33025887/how-to-use-a-local-unpublished-crate), +[issue#1481](https://github.com/rust-lang/cargo/issues/1481), +[my blog](https://diziet.dreamwidth.org/1805.html).) + +Using a local version of a crate should be possible without putting +paths into your `Cargo.toml` and without editing complex +configuration. + +How nailing-cargo helps with using local crates +----------------------------------------------- + +nailing-cargo temporarily edits all the `Cargo.toml` files in all the +subdirectories you mention, to refer to each other; then it runs +cargo; and then it puts eveyrthing back. + +Telling nailing-cargo how to massage Cargo.toml +----------------------------------------------- + +To find the subdirectories, and the packages, it looks for `subdirs` +and `packages` in `Cargo.nail`. + +`subdirs` can be a (usually multi-line) string containing a list of +subdirectory names one per line. (`#`-comments are supported.) Or it +can be a list of strings (`subdirs = ['myproject','mylibrary']`). In +each of these directories `Cargo.toml` will be massaged, and +the package there will be used for other massaged `Cargo.toml`s + +`packages` is a mapping from package names to dictionaries (e.g. in +`Cargo.nail`, write something like: `[packages.mylibrary]` +`subdir=mylibrary-test` or `[packages]` `mylibrary='mylibrary-test'`). +These override the locations for the specified packages (so you can, +for example, have multiple trees with the same package in). The +`subdir` values are also added to the list of directories where +`Cargo.toml` should be massaged. + +Out-of-tree builds +================== + +It is often desirable to run builds in a way that does not write to +the source tree. cargo's enthusiastic approach to the dependency +management task means that it is a good idea to try to insulate your +main working environment from the many things cargo has decided to +download and execute. + +However, when you tell cargo to do an out of tree build (using +`--manifest-path`) it will insist on `Cargo.lock` being in the source +directory, and often will insist on writing to it. + +How nailing-cargo helps with out-of-tree builds +----------------------------------------------- + +nailing-cargo (configured appropriately) copies files back and forth +to between the source and build directories, and runs cargo as your +build user. + +The `Cargo.lock` must still be saved in your source tree somewhere. +nailing-cargo arranges this for you. You can either put this file in +`.gitignore`; or commit it to git; or you can tell nailing-cargo to +save it as something like `Cargo.lock.example`. + +Configuring out-of-tree builds +------------------------------ + +To enable out-of-tree-builds, put an `[oot]` section in your +`Cargo.nail` or one of nailing-cargo's other config files. +In that section, specify at least `use`. + +The primary config keys here are: + + * `dir`: The build directory. If relative, it is relative to the + parent of the invocation directory (and could be a symlink then). + Default is `Build` (assuming `use` is specified). + + * `use`: How to become the build user. Needs to be combined + with other setting(s): + + * `ssh`: Use ssh. `user` must be given as well and is + the `@` argument to ssh. + + * `command_args`: `command` must be specified as a list, + specifying a command and arguments which work like `nice`. + + * `command_sh`: `command` must be specified as a list, + specifying a command and arguments which work like `sh -c`. + + * `null`: Run builds as the same user. + + * `really`: Use `really` from `chiark-really.deb`. + `user` must be given as well. + +Target architecture convenience aliases +======================================= + +If you are cross-building you may need to tell cargo `--target=`. +The architecture names are quite long and inconvenient. + +A simple shell alias would help a lot, except that cargo rejects +`--target=` when it thinks it's not needed. + +In your nailing-cargo config, you can write soemthing like +`arch.RPI='arm-unknown-linux-gnueabihf'`. Then `nailing-cargo -ARPI` +will DTRT. In fact, you do not even need to specify that particular +arch alias, since it is built-in to nailing-cargo. + +Default change to offline mode +============================== + +It seems to me that build tools should be explicit about their use of +the network. So by default, nailing-cargo passes `--offline` to +cargo. + +If you disagree with my opinion, write `misc.online=true` in your +nailing-cargo configuration. `misc.online=false`, and command line +options, are also available, for overriding. + +Command-line options +==================== + + * `-v`: Increase verbosity. Default is 1. + + * `-q`: Set verbosity ot 0. + + * `-D`: Increase amount of debugging dump. + + * `-n`: "No action": stop after writing Cargo.toml.nailing~ + everywhere, and do not run any build command. + + * `-A` | `--arch=` | `--target=` + + Specify target architecture. If starts with a + capital ascii letter, is an alias for some other arch + looked up in Cargo.nail and then in the builtin list: + RPI arm-unknown-linux-gnueabihf + Translates to a --target= option to the ultimate command, + unless that is a cargo subcommand which would reject it. + --arch and --target are simply aliases. + + * `-u` | `--cargo-lock-update` | `-U` | `--no-cargo-lock-update` + + Enables, or disables, the dance to allow Cargo.lock (or + alternative) to be updated in the source directory. + + With this dance enabled the Cargo.lock and Cargo.toml are + copied to the build directory along with a skeleton just big + enough to fool cargo. After cargo has run, the resulting + Cargo.lock is copied back to the source tree. + + This makes no sense with in-tree builds. + + Default is no update unless the ultimate command is a + cargo subcommand which we know needs it. + + * `-m` | `--cargo-manifest-args` | `-M` | `--no-cargo-manifest-args` + + Controls whether we add cargo command line options relating to + finding `Cargo.toml`, to the command to run. + + Default is to add them iff we are doing an out-of-tree build, + unless we are doing the dance to update the `Cargo.lock` (see + above) in which case the only relevant files are to be found in + the build directory). + + The arguments added are + + --manifest-path= + --locked + --target-dir=target + + * `-T` | `--no-cargo-target-arg` | `-t` | `--cargo-target-arg` + + `-T` suppresses `--target-dir`; `-t` un-suppresses it. Only + makes any difference with `-m`, since otherwise no + `--target-dir` would be passed anyway. Additionally this is + done automatically when nailing-cargo sees that the cargo + subcommand is one which needs it, eg `fetch'. + + * `-o` | `--online` | `--offline` | `-O` + + Whether to allow cargo to make network access. nailing-cargo + always passes `--offline` to cargo, unless `--online` is in + force. The default is offline, unless the cargo subcommand is + one which implies online (currently, `fetch'). + +Configuration +============= + +nailing-cargo reads these configuration files: +``` + /etc/nailing-cargo/cfg.toml + ~/.nailing-cargo.toml + ./.nailing-cargo.toml + ../Nailing-Cargo.toml + ../Cargo.nail +``` +Settings in later-mentioned files override ones in earlier-mentioned +files. + +Note that unlike everything else, `packages` and `subdirs` are read +only from `Cargo.nail` (see "Limitations and bugs", below). + +Alternative `Cargo.lock` filename +--------------------------------- + +To control use of alternative Cargo.lock filename, use the section +`[alt_cargolock]`. Settings here: + + * `file = false`: disables this feature + + * `file = true`: equivalent to `file = "Cargo.lock.example"` + + * `file = ` + + * `force = false`: Use the alternative file only if it + already exists. (This is the default.) + + * `force = true`: Always uses the alternative filename. + +Running the command +=================== + +Normally you pass `cargo` as an argument to `nailing-cargo`. But you +can also pass `make` or any other command. You may need to pass +`--no-cargo-manifest-args` to nailing-cargo. + +nailing-cargo passes these environment variables to the build command: + + * `NAILINGCARGO_WORKSPHERE`: invocation `..` (parent) + * `NAILINGCARGO_MANIFEST_DIR`: invocation `.` (invocation directory) + * `NAILINGCARGO_BUILD_DIR`: build directory (even if same as source) + * `NAILINGCARGO_BUILDSPHERE`: only set if out of tree: parent of build dir. + +All of these are absolute paths. + +Limitations and bugs +==================== + + * nailing-cargo dirties your source trees, including particularly + `Cargo.toml` and `Cargo.lock`, temporarily. If nailing-cargo crashes + or is interrupted these changes may be left behind; running + nailing-cargo again should clean up such a mess. Unfortunately it is + not possible to fix this bug because the cargo team have deliberately + made cargo inflexible. + [[issue#6715]](https://github.com/rust-lang/cargo/issues/6715).) + + * Out of tree builds require a unified filesystem view: eg, different + users on the same host, NFS, or something. This could be improved. + + The alternative Cargo.lock filename must currently be a leafname. I + think fixing this just involves review to check other values work + properly. + + * The alternative Cargo.lock file must be on smae filesystem. This is + not so easy to fix; we would want the existing algorithm but a + fallback for this case. + + * Cargo.nail is unconditionally looked for in `..`. Ideally should be + configurable, and also perhaps be able to combine multiple Cargo.nail + files? Relatedly, although nailing-cargo can read multiple config + filos, it can only handle one file specifying directories and + packages. + + * Contributions to address these would be welcome, of course. If you + plan to do substantial work, please do get in touch with a sketch of + your proposed changes. + +Contributing and legal +====================== + +nailing-cargo is Free Software. + +Please help improve it. Contributions are welcome by email to +`ijackson@chiark.greenend.org.uk` or via the +[Salsa project](https://salsa.debian.org/iwj/nailing-cargo). + +Legally, the project accepts contributions which follow the git commit +Signed-off-by convention, by which the contributors certify their +contributions according to the Developer Certificate of Origin version +1.1 - see the file DEVELOPER-CERTIFICATE. + +Copyright (C) 2019-2020 Ian Jackson + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Individual files generally contain the following tag in the copyright +notice, instead of the full licence grant text: +``` + SPDX-License-Identifier: AGPL-3.0-or-later +``` +As is conventional, this should be read as a licence grant. diff --git a/nailing-cargo b/nailing-cargo index cc7d41f..d1d3309 100755 --- a/nailing-cargo +++ b/nailing-cargo @@ -1,184 +1,6 @@ #!/usr/bin/perl -w - -# nailing-cargo: wrapper to use unpublished local crates -# -# Copyright (C) 2019-2020 Ian Jackson -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see . - -# example usages: -# ../nailing-cargo/nailing-cargo make -# ../nailing-cargo/nailing-cargo cargo build - -# Why do we need this ? -# -# https://github.com/rust-lang/cargo/issues/6713 -# https://stackoverflow.com/questions/33025887/how-to-use-a-local-unpublished-crate -# https://github.com/rust-lang/cargo/issues/1481 - -# Options: -# -v Increase verbosity. Default is 1. -# -q Set verbosity ot 0. -# -D Increase amount of debugging dump. -# -# -n "No action": stop after writing Cargo.toml.nailing~ -# everywhere, and do not run any build command. -# -# -T --target= -# Specify target architecture. If starts with a -# capital ascii letter, is an alias for some other arch -# looked up in Cargo.nail and then in the builtin list: -# RPI arm-unknown-linux-gnueabihf -# Translates to a --target= option to the ultimate command, -# unless that is a cargo subcommand which would reject it. -# -# -u | --cargo-lock-update -# -U | --no-cargo-lock-update -# Arranges to do a dance to allow Cargo.lock (or -# alternative) to be updated in the source directory. -# -# The Cargo.lock and Cargo.toml are copied to the build -# directory along with a skeleton just big enough to fool -# cargo. After cargo has run, the resulting Cargo.lock is -# copied back to the source tree. -# -# Makes no sense with in-tree builds. -# -# Default is no update unless the ultimate command is a -# cargo subcommand which we know needs it. -# -# -m | --cargo-manifest-args -# -M | --no-cargo-manifest-args -# Controls whether we add cargo command line options -# relating to finding Cargo.toml, to the command to -# run. -# -# Default is true if we are doing an out-of- tree build, -# unless we are updating the Cargo.lock (in which case the -# only relevant files are to be found in the build directory). -# -# The arguments are -# --manifest-path= -# --locked -# --target-dir=target -# -# -T | --no-cargo-target-arg -# -t | --cargo-target-arg -# Suppress --target (or un-suppress it). Only useful with -m. -# Done automatically when nailing-cargo sees that the cargo -# subcommand is one which needs it, eg `fetch'. -# -# --online | -o -# --offline | -O -# Whether to allow cargo to make network access. -# (nailing-cargo always passes --offline to cargo, unless -# --online is in force). The default is offline, -# unless the cargo subcommand is one which implies -# online (currently, `fetch'). -# -# Cargo.nail: -# -# # Adds each to the list of directories whose -# # Cargo.toml is to be nailed, and also arranges to use -# # the package found there for other directories if -# # no other source of that package is evident in Cargo.nail. -# subdirs = [ -# "" -# ... -# ] -# or: -# subdirs = """ -# "" -# ... -# """ -# -# # Adds to the list of directories whose Cargo.toml -# # is to be nailed, and overrides any other nailing for -# [packages] -# = -# = { = ... } -# -# Configuration is read from -# -# /etc/nailing-cargo/cfg.toml -# ~/.nailing-cargo.toml -# ./.nailing-cargo.toml -# ../Nailing-Cargo.toml -# ../Cargo.nail -# -# To control use of alternative Cargo.lock filename: -# [alt_cargolock] -# -# file = true # equivalent to "Cargo.lock.example" -# file = false # disables this feature -# file = "" -# -# force = false # default, uses alt file only if it already exists -# force = true # always uses alt file; creation would make Cargo.lock -# -# (you can also specify just alt_cargo_lock instead of alt_cargo_lock.file) -# -# To enable out of tree builds: -# [oot] -# dir = "" # default is Build, if use is specified -# and then -# use = "really" -# user = "" -# or -# use = "ssh" -# user = "@host" # NB must still share a filesystem! -# or -# use = "command_args" -# command = ["", "", "nice"] -# or -# use = "command_sh" -# command = ["", "", "sh -c"] -# or -# use = "null" -# -# Other settings: -# [misc] -# online = true # forces default to be --online -# online = false # forces default to be --offline -# -# Limitations: -# -# Always dirties everyone's Cargo.toml, but tries to put them -# back (if not, running it again should fix it). Cannnot be -# fixed without changes to cargo. -# -# Out of tree builds require a unified filesystem view: eg, -# different users on the same host, NFS, or something. This -# could be improved. -# -# Alternative Cargo.lock file must currently be a leafname. -# I think this just involves review to check other values work. -# -# Alternative Cargo.lock file must be on smae filesystem. -# This is not so easy; we would want the existing algorithm but -# a fallback for this case. -# -# Cargo.nail unconditionally looked for in .. -# Ideally should be configurable, and also perhaps be able -# to combine multiple Cargo.nail files ? -# -# Env vars we pass to the command: -# NAILINGCARGO_WORKSPHERE absolute path of invocation .. -# NAILINGCARGO_MANIFEST_DIR absolute path of invocation . -# NAILINGCARGO_BUILDSPHERE only if out of tree: abs parent of build dir -# NAILINGCARGO_BUILD_DIR absolute path of build dir (even if = src) - +# nailing-cargo: wrapper to use unpublished local crates +# SPDX-License-Identifier: AGPL-3.0-or-later our $self;