Ross Younger

Lignum

Tue, 14 Jul 2026 16:24:56 +1200

A source code repository visualiser with a 1,000’ view.

I wasn’t satisfied with the state of Git repository visualisers. I found they love to show you every last commit. When your repo is more than trivial, this is not useful for getting the big-picture.

Mine is different: it focusses on the macroscopic structure. Which are the active branches, where are the tags, where are the divergence and merge points?

In my first software job out of college – back when dinosaurs roamed the earth and CVS was the height of version control – we often ended up drawing what we termed railway diagrams on the whiteboard to visualise the structure. This is my attempt to automate creating these.

Features

Output in GraphViz dot format
Revision of interest filtering, with wildcards
Options to include all local branches (default), remotes, tags and root (parentless) commits
Edges are annotated with their size in commits (broadly the number on the direct line from A to B, though merge commits give different results)

Getting started

This is a Rust command-line application.

cargo install --locked lignum, or get it from the source repository.

Run lignum in a directory that contains a git repository, or give it --repo <path>. If you have xdot installed, the -x option will pipe the output directly to xdot.

Check the --help message for more.

A web toy: Forced ranking assistant

Mon, 21 Jul 2025 18:21:31 +1300

I wanted to arrange some items in a strongly ranked order of preference, but found it quite hard. So I made a little web tool that tries to help.

The idea? A sort of battle royale.

Consider two of the items chosen at random. Which do you prefer? Give it one point.
Repeat for every possible pairing of the items, and add up the scores.

The tool: https://crazyscot.github.io/forced-rank/

It’s a Javascript single-page application written in VueJS. Source code

Getting value from CI

Sun, 01 Dec 2024 10:53:36 +1300

Many of my peers within the software world know the value of Continuous Integration and don’t need convincing. This article is for everybody else.

Introduction

In my first job out of college we had what you’d recognise as CI, though the term wasn’t so popular then. It was powerful, very useful, but a source of Byzantine complexity.

I’ve also worked for people who didn’t think CI was worth doing because it was too expensive to set up and maintain. This is not totally unreasonable; the real question is to figure out where the value for your project might lie.

Recently, a friend wrote:

I don't really know very much about CI. I would be interested in knowing more and might even use some of the quick wins (...) I do not want to become completely reliant upon GitHub for anything.

So let’s start with a primer.

Terminology: What is CI?

Unfortunately the term “CI” is sometimes misused and/or confused.

The short answer is that it’s automation that regularly (continuously) does something useful with your codebase. These actions might take place on every commit, nightly, or be activated by some external trigger.

CI usually refers to a spectrum of practices, each step building on the last:

Continuous…	Typical activities
Build	Builds your code, usually to the unit or module level. Runs unit tests.
Integration	Assembles modules to a “finished application”, whatever that means. Runs integration tests.
Test	A full suite of automated tests. May include regression, performance, deployability and data migration.
Delivery	When the test suite passes, the latest version of the system is automatically released to a staging environment. This might involve building packages and putting them in a download area.
Deployment	When the automated tests pass, the software automatically goes live. Hold tight!

Exactly what these phases mean for your project, and how far you go with them, depends on your project.

What suits my embedded firmware probably won’t suit your cloud app or that other person’s desktop app.
The lines between the phases are blurry. For example, it may or may not make sense to build and integrate everything in one go.

❔ Why CI

If deployed appropriately, CI can save time, reduce costs and improve quality. Even on a hobby project, there is often value in saving your time.

1. Automating stuff, so the humans don’t have to

You could use your engineers to do the repetitive drudge work of creating a release across multiple platforms. You could have them run a full barrage of tests before committing a code change… but should you? Engineers are expensive and generally dislike boring stuff, so the smart business move is usually to automate away the repetitive parts and have them focus where they can deliver most value.

If you’re not sure, consider this: how much time does your team spend per release cycle on the repetitive parts? Consider your expected frequency of release cycles, that should lead you to the answer.

2. Automatic analysis and status reporting

One place I worked had a release process which relied on an engineer reading multiple megabytes of log file to see if things had been successful. Many things could go wrong and leave the final output in a plausible but half-broken state. Worse, it wasn’t as simple as running the script in stop-on-error mode, because some of the steps were prone to false alarms.

You may be ahead of me here, but I didn’t think much of that setup.

Compilation failed? Show me the compiler output from the file that failed.

A test failed? I want to see the result of that test (expected & observed results).

Everything passed? Great, but don’t spend megabytes to convey one single bit of information.

At its simplest, a small project will have a single main branch, and the operational information you need can be boiled down to a small number of states:



Something is broken	Non-critical warning (not all projects use this)	Everything is working

In a non-remote workplace it might make sense to set up some sort of status annunciator.

Some people use coloured lava lamps or similar.
At one place I worked the machinery in the factory had physical traffic light (andon) lamp sets. We set one of these up, driven by a Raspberry Pi wired in to the build server.
Some projects build more elaborate virtual dashboards that suit their needs. Multiple branches, multiple build configurations, whatever makes sense.

3. Improved quality

This one might be self-evident, but I’ll spell it out anyway.

A good CI system will let you incorporate tests of many different types, with variable pass/fail criteria. Think beyond unit and integration testing:

Regression (check that your bugs stay fixed)
Code quality (code/test coverage analysis; static analysis; dynamic memory leak analysis; automated code style checks)
Security analysis (are there any known issues in your dependencies?)
License/SBOM compliance
Fuzz testing (how does it handle randomised, unexpected inputs?)
Performance requirements
“Early warning” performance canaries
Standards compliance
System data migration
On-device testing (might be real, emulated or simulated hardware)

Performance canaries

Particularly where physical devices are involved, you might have a performance margin built in to your hardware spec. As the project evolves, inevitably new features will erode this margin. When you run out things are going to go wrong, so you want to take action before you get there.

An early warning canary is some sort of metric with a threshold. Examples might include free memory, CPU/MCU consumption, or task processing time. When the threshold is passed, that's a sign that things are getting tight and it's time to take pre-emptive action. You might plan to spend some time on algorithmic optimisations, or to kick off a new hardware spin.

If you can automate a really robust set of tests, you can have a lot of confidence in the state of your code at any given time. This gives incredible agility: you can release at any time, if the tests pass. This is the key to moving quickly, and is how a number of tech companies operate.

For a success story involving physical devices, check out the HP LaserJet team’s DevOps transformation.

4. Reduced time to resolve issues

If there’s one thing I’ve learned in the software business, it’s that it’s cheaper to find bugs closer to development - by orders of magnitude.

In other words, reduce the feedback cycle to reduce your costs. This is where automated tests and checks have great value.

If there is something wrong in code I modified a minute ago, I’m still in the right headspace and can usually fix it pretty quickly.
If it takes a few days to get a test result, I won’t remember all the detail and will have to refresh my memory.
If it takes several months to hear that something’s wrong, I may be working on a totally different part of the system and it will take longer to context switch.
If a bug report comes in from the field a year or two later, I might as well be starting again from scratch.

But - as ever - engineering is a trade-off. You can’t write a test to catch a bug you haven’t foreseen. It may be prohibitively expensive to test all possible combinations before release.

❌ Why not CI

CI is not suitable for all software projects.

If you’re writing a scratch throw-away project that won’t live for very long, even simple CI may not be worth it.

If you have a legacy codebase that was written without testing in mind, it might be prohibitively expensive to refactor to set these up. Nevertheless, in such projects there is often still some value to be found in a continuous build.

Let’s be pragmatic.

Tests aren’t everything

On the face of it, more testing means greater quality, right? Well… maybe?

Keep the end goal in sight. It’s up to you to decide what makes sense for your situation; I recommend taking a whole-of-organisation view.

You need to balance test runtime against overall feedback cycles. If the tests take too long to run, you’re slowing people down.
Some tests are expensive in terms of time or consuming resources, so you might not want to run them daily.
Tests involving physical devices can be difficult to automate, and risk creating a process bottleneck. (Consider emulation and/or simulation where appropriate.)
Beware of over-testing; you may not need to exhaustively check all the combinations. Statistical techniques might help you out here.
Beware of making your black-box tests too strict; this can lead to brittle tests that are more hassle to maintain than they are worth.

Costs and maintenance

It will take time and effort to set up CI. How much time and effort, I can’t say.

In times past, CI was quite the bespoke effort.

These days there is good tooling support for many environments, so it is usually pretty quick to get something going. From there you can decide how far to go.

It might be too big for your platform

CI platforms are designed for small, lightweight processes. Think seconds to minutes, not hours.

If you need to build a large application or a full Yocto firmware image, it’s going to be tough to make that fit within the limits of a cloud-hosted CI platform. Don’t despair! There are ways out, but you need to be smart. Alternative options include:

self-hosting CI runners that are take part in a cloud source repository;
self-hosting the CI environment (e.g. Gitlab, Jenkins, CircleCI), noting that most source code hosting platforms have integrations;
split up the task into multiple smaller CI jobs making good use of artefacts between stages;
reconsidering what is truly worth automating anyway.

👷 Steps you can take

1. Build your units

In most projects you already had to set up a buildsystem. Automating this is usually pretty cheap though you will need to get the tooling right.

Tooling on cloud platforms

On-cloud CI (as provided by Github, Gitlab, Bitbucket and others) is generally containerised. What this means is that your project has to know how to install its own tooling, starting with a minimal (usually Linux) container image.

This is really good practice! Doing so means your required tools are themselves expressed in source code under revision control.

Where this might get tricky is if you have multiple build configurations (platforms or builds with different features). Don’t be surprised if automating reveals shortcomings in your setup.

If you have autogenerated documentation, consider running that too. (In Rust, for example, it could be as easy as adding a cargo doc step.)

2. Test your units

Adding unit tests to CI is usually pretty cheap though it will depend on the language and available test frameworks.

If you want to include language-intrinsic checks (e.g. code style, static analysis) this is a good time to build them in. Some analyses can be quite expensive so it may not make sense to run all the checks at the same frequency.

3. Integrate it

If you’re pulling multiple component parts (microservices, standalone executables) together to make an end result, that’s the next step. Do they play nicely? Do you want to run any isolated tests among them before you move to delivery-level tests?

4. Add more checks

I spoke about these above.

This is where things stop being cheap and you have to start thinking about building out supporting infrastructure.

5. Deliver it

Now we’re getting quite situation-specific. Think about what it means to deliver your project.

Are you building a package for an ecosystem (Rust crate / Python pypi / npm.js / …) ? You might be able to automate the packaging steps and that might be pretty cheap.

Are you building an application? Perhaps you can automate the process of building the installer / container / whatever shape it takes. If you have multiple build configurations or platforms, it could get very tedious to build them all by hand and there is often a win for automation.

Where there's code signing involved, you'll need to decide whether it makes sense to automate that or leave it as a manual release step. Never put private keys or other code signing secrets directly into source! Some platforms have secrets mechanisms that may be of use, but it pays to be cautious. If your secrets leak, how will you repair the situation?

Closing thoughts

Most projects will benefit from a little CI. You don’t need to have unit tests, though they are a good idea.
You’re going to have to maintain your CI, so build it for maintainability like you do your software.
Apply agile to your CI as you do to your deliverables. Perfect is the enemy of good enough. Build something, get feedback, iterate!
CI vendors want to lock you in to their platform. Keep your eyes open.
Don’t let CI become an all-consuming monster that prevents you from delivering in the first place!

Announcing qcp

Sun, 03 Nov 2024 10:23:43 +1300

The QUIC Copier (qcp) is an experimental high-performance remote file copy utility for long-distance internet connections.

Source repository: https://github.com/crazyscot/qcp

📋 Features

🔧 Drop-in replacement for scp
🛡️ Similar security to scp, using existing, well-known mechanisms
🚀 Better throughput on congested networks

📖 About qcp

qcp is a hybrid protocol combining ssh and QUIC.

We use ssh to establish a control channel to the target machine, then spin up the QUIC protocol to transfer data.

This has the following useful properties:

User authentication is handled entirely by ssh
Data is transmitted over UDP, avoiding known issues with TCP over “long, fat pipe” connections
Data in transit is protected by TLS using ephemeral keys
The security mechanisms all use existing, well-known cryptographic algorithms

For full documentation refer to qcp on docs.rs.

Motivation

I needed to copy multiple large (3+ GB) files from a server in Europe to my home in New Zealand.

I’ve got nothing against ssh or scp. They’re brilliant. I’ve been using them since the 1990s. However they run on top of TCP, which does not perform very well when the network is congested. With a fast fibre internet connection, a long round-trip time and noticeable packet loss, I was right in the sour spot. TCP did its thing and slowed down, but when the congestion cleared it was very slow to get back up to speed.

If you’ve ever been frustrated by download performance from distant websites, you might have been experiencing this same issue. Friends with satellite (pre-Starlink) internet connections seem to be particularly badly affected.

💻 Getting qcp

The project is a Rust binary crate.

You can install it:

as a Debian package or pre-compiled binary from the latest qcp release page (N.B. the Linux builds are static musl binaries);
with cargo install qcp (you will need to have a rust toolchain ~~and capnpc~~ installed) (the capnp dependency was removed later);
by cloning and building the source repository.

You will need to install qcp on both machines. Please refer to the README for more.

Broadcast graphics for fencing

Thu, 15 Aug 2024 21:16:22 +1200

I created a TV graphics package for fencing tournaments.

Earlier this year, Christchurch played host to the Commonwealth Junior & Cadet fencing tournament.

Selected parts of the tournament were livestreamed, with a package broadcast on Sky TV (NZ). The broadcast and finals streams had a live graphics package fed from the scoreboard.

The programmes were produced using a broadcast-spec OB truck supplied by Kiwi Outside Broadcast. The truck graphics PC used Captivate to generate its graphics, which output as key+fill SDI signals. These were fed to the vision mixer and keyed onto the picture in the usual way.

The package can be seen in action on the Commonwealth Junior & Cadet 2024 programmes.

The details read from the scoreboard cover the “hit” lamps, scores, clock (including fractional seconds in the last 10s), period, red/yellow cards, and the priority indicator. On top of that, the package provides a place to enter the fencer names and nationalities, and set colours for them.

This is all made possible by the scoreboard, a Favero FA-07, offering a data feed over an RS-422 interface. I wrote a Python script to parse the data feed, turn it into a JSON dictionary and pass on to Captivate.

Fault-finding at the ends of the earth

Mon, 29 Apr 2024 13:29:43 +1200

This is a tale from many months ago, working on an embedded ARM target.

In my private journal I wrote:

Today I feel like I saddled up and rode my horse to the literal ends of the earth. I was fault-finding in the setting-up-of-the-universe that happens before your program starts up, and in the tearing-it-down-again that happens after you declare you’re finished.

If you know C++, you might guess that this was a story about static object allocation and deallocation. You’d be right. So, destructors belonging to static-allocated objects. You’d never think they’d run on a bare-metal embedded target.

Well, they can. If your target supports exit() - e.g. if you are running with newlib - then an atexit handler is set up for you, and that will be set up to run the static destructors. If your program then calls exit() (as, say, your on-silicon unit tests might, at the end of a test run) then things are at risk of turning to custard.

You might have enabled an interrupt for some peripheral on the silicon. In order to do anything really useful, the ISR might reference a static object. If you do this, you’d damn well better make sure the object has a static destructor that disables the interrupt, or hilarity is one day going to ensue. You know, the sort of hilarity that involves being savaged by a horde of angry rampaging badgers, or your socks catching fire.

But wait, I hear you say, it called exit! The program no longer exists! Well, sure it doesn’t; but what happens on exit? On this particular ARM target, running tests via a debugger as part of a CI chain, when the atexit handlers have run the process signals final completion with a semihosting call, which is a special flavour of debug breakpoint. It is… not fast. If your interrupt happens regularly, the goblins are going to get you before the pseudo-system-call completes. Your test framework will fail the test executable for hanging, despite somehow passing all of its tests.

There was an actual bug in there, and it was mine. Class X, which contained an RTOS queue and enabled an interrupt, only had a default destructor. On exit, somewhen between static destructors and completion of the semihosting exit call, the ISR fired. It duly failed to insert an item into the now-destroyed queue, so jumped to the internal panic routine. That routine contained a breakpoint and then went nowhere fast, waiting for a debugger command that was never going to arrive — hence the time-out. Maybe it would have been useful to have a library option to skip the static destructors, but I probably wouldn’t have been aware of it ahead of time anyway.

The static destructor ordering fiasco can also be yours for the taking, but thankfully that hadn’t bitten me. Nevertheless, it was a rough day.

Cover image: Cyber Bug Search, by juicy_fish on Freepik

The Road Less Travelled

Thu, 04 Jan 2024 20:03:08 +1200

I’ve started an occasional YouTube series about quirky, off-the-beaten-track places.

At the time of writing there are 5 episodes online. I’m still experimenting with style and techniques. My inspiration is a combination of The Tim Traveller and Tom Scott.

So far it consists of a few random places around South Island. There’s no particular timescale for publishing new episodes; I have many demands on my time, and this project isn’t yielding an income at the moment. (It may never; I’m doing it for fun.) Presenting is taking some getting used to!

Here’s the playlist:

About Me

Mon, 01 Jan 2024 00:00:00 +0000

I grew up in Scotland, lived for some years in Cambridge in England, and now live with my wife in Christchurch, New Zealand.

A child of the 1980s, I’ve been a heavy computer (and later Internet) user since my first home micro at the age of 8. I think of myself as a generalist, capable of deep understanding of systems and subjects, yet also capable of turning my hand to new challenges, new workflows, new environments. I’m not too proud to be told when I’m wrong, and I believe in finding the right solution.

My first career was as a software engineer, teaching machines to communicate. From 2000 to 2011 – after my first degree – I lived in Cambridge, working mostly as a software engineer with a focus on embedded and embedded-adjacent systems.

In software, my professional interests include cryptography, reliability, and leading teams to greater performance through careful application of DevOps principles.

My wife and I moved to Christchurch in 2011, several months after the devastating earthquake that struck the region.

In 2013 I realised that it was time for a new challenge and enrolled at the New Zealand Broadcasting School, to study Digital Film and TV Production for the Bachelor of Broadcasting Communications degree. I completed the course in 2016. I particularly enjoy camera operation, vision switching and editing and I love the buzz of the team environment, both on the set and in the studio.

For a few years I was a freelance videographer and video technician, but as of 2019 my primary focus is back on software. (I keep my hand in with the occasional freelance job. Since 2023 I’ve been part of a team covering NZ domestic basketball, broadcast live-to-air in NZ and available across the world via ESPN.)

I’m a fencer; I’ve been fencing on and off for most of my life. As of 2022 I am the president of United Fencing Club. I have served as both secretary and treasurer of the regional body Fencing Mid South, and I advise them and the national body Fencing NZ about video replay systems. I used to say that the épée was my preferred weapon but in 2023 I was tempted to try the dark side that is sabre …

Contact information & GPG key

If you’re not already in contact with me, webcontact at crazyscot.com is a good place to email in the first instance. (Mar 2026: Apologies if you’ve tried to email that mailbox within the past year or so and found it bouncing. Now fixed.)

I have a GPG key. (You might find a more up-to-date version on the standard keyservers.) If you know what to do with one of those, you’ll find other email addresses in it. The address which starts with the digit zero does not have any spam filtering, so is a good one to try if you have problems reaching me on other addresses.

2023 in pictures

Sun, 31 Dec 2023 00:00:00 +0000

It has been a somewhat lumpy year of photography, so I don’t have a favourite for each month. But I do have some favourites, more than 12 in fact.

Willowbank Wildlife Reserve. These ducks look like they’re about to drop a sick album.

A cheeky selfie before the Davis Cup tennis tie. It was 34C in the shade, and I had no shade.

Kakariki with malice aforethought

Aurora (airglow?) and Magellanic Clouds

Belatedly celebrating C’s PhD atop the Sky Tower

Blue-and-gold macaw in flight, at Willowbank

Rose macro

Roto Kohatu Reserve. The Pūteketeke (Australasian crested grebe)

Behind the scenes at basketball

Busy bee

Waiuta. Two hats and a carful of sandflies.

Willowbank. Scarlet macaw in flight.

Willowbank. A pair of love birds (more blue-and-gold macaws)

Rose macro in our garden

Willowbank. Kea in flight.

Faster C++ code style checks: detail

Thu, 17 Aug 2023 22:04:01 +1200

xargs, nproc and shell arrays, oh my!

This is a companion article to Developer tooling and boiled frogs.

Before: 4.7 seconds 💩

In this codebase, style checks were performed by clang-format with an in-house style file. They ran in a pre-commit hook (as well as in CI). However, the pre-commit hook is critical for a good DX; 5 seconds really sucks.

The entry point to the check was a shell script which invoked clang-format, amongst other things. Like all such scripts, it was not truly original, being assembled from snippets and ideas found around the internet. Here’s the gist of it:

check_files() {
    # arguments are filenames. if any failed, returns 1; if all passed, returns 0
    err=
    for file in $@; do
        ext=${file##*.}
        case "$ext" in
            c|cpp|h|hpp)
                if ! clang-format -Werror -style=file -n "$file" ; then
                    err=1
                    echo "code style check failed for $file"
                fi
                ;;
            *)
                # unsupported file type; ignore
                ;;
        esac
    done
    return $err
}

# ...

check_files "$(git ls-files)"

Do you see the problems?

It’s single-threaded! Almost all desktop and laptop CPUs have had multiple cores for 15 or more years, so this is truly Dark Ages style processing.
We’re invoking clang-format far more times than we need to. Every invocation has to parse the config file, which is wasteful.

After: 0.21 seconds 🚀

check_files() {
    # arguments are filenames. if any failed, returns 1; if all passed, returns 0
    FILES=()
    for file in $@; do
      ext=${file##*.}
      case "$ext" in
        c|cpp|h|hpp)
          FILES+=("$file")
          ;;
        *)
          # unsupported file type; ignore
          ;;
      esac
    done

    FILES_COUNT="${#FILES[@]}"
    CORES=$(nproc)
    # CORES+3 gave best performance for me. YMMV.
    JOB_SIZE=$((FILES_COUNT / (CORES+3)))
    if [ ${JOB_SIZE} == 0 ]; then JOB_SIZE=1; fi
    echo "${FILES[@]}" | xargs -P 0 -n ${JOB_SIZE} clang-format -Werror -style=file -n || return 1
    return 0
}

There’s a bit to unpack here:

We’ve moved from sh to bash to take advantage of array variables.
Rather than immediately running clang-format on every file that passes the filter, add them to the FILES array so we can use xargs. (Yes, I know this could be better structured. This was but a single step of evolution; first, make it work.)
We use xargs in parallel mode.

We’ve naively assumed that checking each file takes roughly the same amount of time. This isn’t quite true (I found about a factor of 3 variation amongst them) but the performance gains were good enough that it doesn’t seem worth worrying about at the present time.

The key here is combining -n $JOB_SIZE with -P 0 (unlimited parallelisation) and doing some shell arithmetic to figure out what that job size should be.

xargs with neither option runs a single clang-format with all the input files in a single invocation.
- This is good (about a 6x speedup) but we can do better! It is still single-threaded, iterating through the files in order.
xargs -P $(nproc) sounds interesting but is no better on its own. The man page says it all:

Use the -n option or the -L option with -P; otherwise chances are that only one exec will be done.
I’ve counted the files, divided by (slightly more than) the number of processor cores available, and told xargs to parcel the files out in jobs of that size. Of course, if you’re doing this yourself, you’ll want to repeat the experiment to figure out the sweet spot for your hardware and your codebase.

🚅 Final speedup for this activity: 22.3x 🚀 (timings as measured on my laptop for that codebase on that day; your mileage may vary, may contain nuts, etc.)

Developer tooling and boiled frogs

Thu, 17 Aug 2023 22:04:00 +1200

This is an article about software tooling and the developer experience.

Background

Right now I am spending most of my work time on an embedded project. It’s C++ running on bare-metal target hardware. I’ve got all of 768K of RAM to play with, but that’s still a tad more than my first home micro had.

Our tooling and infrastructure grew mostly organically. I was the one who initialised the git repo.

CI chain 🔗

We have a CI chain, of course. It has grown over time and does a few tricks:

build for the target hardware (various targets; RAM and flash builds)
build on Linux native port and run unit tests
perform various analyses (clang-tidy, sonarcloud, code coverage, valgrind)
code style enforcement (clang-format) 👮
prepare a signed upgrade image with the development signing key
assemble a whole-of-flash image for the factory 🏭
build and run some unit tests on a development board

The dev board is on-premises but the rest happens in the cloud, taking a matter of minutes on every push. It’s reasonably slick.

You can run all the build and analysis steps locally. We have a script that spins up a container emulating our CI provider so you don’t even need to have the dev tools installed, you only need to pull our in-house container image.

How much effort to spend on tooling? 💸

Many a savvy manager or product owner will tell you to not spend too much time on tooling. They have a point; time spent working on tooling and infra is not delivering features, after all. Many managers have been bitten by infrastructure blow-outs and would rather we focus on building the stuff they want us to build.

But here’s the thing. When they say that, they don’t mean “forget about tooling and never speak of it again”. You are allowed to bring the subject up again in future, and you are allowed to try to educate them as to the value.

The frog boils 📉

We now have over 300 source files that get compiled into the image, which includes some vendor and autogenerated code.

There’s the rub. We didn’t start out with anywhere near that many. The CI chain used to be really fast, under one minute. Local builds were fast, and even running the clang-tidy script wasn’t that awful (though I did build a cache for it, which helped).

The size and complexity grew over time, and here we are today.

It’s like the old tale about boiling frogs. (Disclaimer: I don’t even know if this is true, and I definitely don’t suggest you try it, but it’s an interesting thought experiment.)

What happens if you drop a frog into a pot of boiling water?

The frog doesn’t like it and jumps out.

But what happens if you drop a frog into a pot of cold water, then light the stove? The frog doesn’t realise anything is wrong. It boils to death before it thinks to jump out.

Builds with a cold cache were taking a couple of minutes locally, and about 8 minutes in the cloud. (This happens depressingly often when you change a low-level header that is included all over the place… there’s another observation, that maybe our architecture needs work, but that’s for another day.)

A full static analysis with cold cache was taking over 5 minutes locally, and 22 in the cloud. (The difference is because our company-provided laptops have 16 cores, while our CI provider runs our container with 4 vCPU cores and tightly limited RAM.)

Goodness me, 300 compile units now? This water sure is getting warm! 😰

It’s a poor developer experience. Even running the code style checker script takes five whole seconds. That’s 5 seconds in which you can get distracted, pop over to Slack, and whoops! ten minutes later, you’re trying to remember what it was you were doing.

I’ve been in this business long enough to know that keeping myself and my team focussed is vitally important to getting things done and keeping the stakeholders happy. An efficient DX is part of that puzzle.

Like the mythical frog, I don’t remember noticing these tasks growing longer.

A concrete walk-through 🔧

OK, so you’ve decided you need to jump out of the pan and spend some time sharpening your tools. Great.

The most important advice I can offer is to understand where the pain points are. 🔍

If this is stuff you work with every day, you maybe already know where the pain points are but are blind to them because you haven’t noticed how poor the DX has become. Open your eyes, and you may find it is in plain sight. Don’t optimise for its own sake; optimise where it will help.

Here’s what I noted about our tooling:

The compilation time itself is already pretty well optimised and parallelised thanks to meson, ninja and ccache.
The code style checker is something we run very frequently as part of our workflow (it’s in our pre-commit script). At 5 seconds, that’s like sand in your gears.
The static analyser is a different type of pain point. We run it interactively but we don’t like to because it’s a slow ol’ beast. On the cloud, it is the slowest part of the build; it is probably the reason why we’re regularly incurring overage on our CI build minutes. (Now, that’s not a big cost compared to an engineer’s salary, but it’s not zero and it has been noticed, so maybe worth spending a little time on.)
The unit test run on a dev board is a bottleneck, practically by definition, as it uses a self-hosted CI runner plugged into our pipeline.

Code style checks: Parallelise!

We use clang-format with an in-house style file.

The heart of our checker script is a shell function.

The tl;dr version is:

We were invoking clang-format in a tight loop, once per input file. 🤦
Now, we count the files, divide by (slightly more than) the number of CPU cores available, and have xargs parcel out the work into that many chunks. 🚅

Static analysis: Reconsider the load, cache, and parallelise where it will help!

The learnings from this one are:

Do the heavy lifting in parallel (we were already)
Cacheing positive analysis results helps (we were doing that too)
Don’t analyse unit tests or test harness/mocks/fakes unless you have a good reason
Don’t analyse target-specific files for obsolete hardware targets unless you have a good reason
Where you have a string-and-glue shell phase to prepare the analysis run, it can be worth parallelising that as well.

Results 🚀

(Timed on my work laptop, which has 16 cores. It’s running Ubuntu 22.04 on WSL on Windows 11.)

Action/Phase	Time before	Time after	Speed-up
Code style check	4.7 seconds	210 ms	22.3x
Static analysis setup	14 seconds	3.5s	4x

My next steps

The on-target testing phase is a bottleneck, so we’re going to add more hardware.
There might be some mileage in using using pre-compiled headers to speed up cold-cache compilation.
- This isn’t something the places I have worked have traditionally done, but I know it’s popular when you’re dealing with the Win32 C++ api.
- Our codebase has a “platform” layer; this smells like a good starting point for precompilation.
Coverage analysis. We already do this, but there’s another pain point while genhtml crunches the data and generates the HTML report. This one could be trickier; it’s a single-threaded perl script, invoked by meson.
Stack usage analysis. We’ve got some code paths that can be provoked to overflow on debug builds. This is a different sort of pain point; while we’ve configured the silicon to detect and hard-fault on overflow, I’d like to set up some analysis to automatically find these situations ahead of time.

Closing thoughts

How long to spend on infrastructure? 💸

Circling back to a question I asked earlier, how long is a piece of string?

It depends where you are and what your current DX is like.

10-20% of your developer time seems a reasonable benchmark; it’s enough that you can get a handful of DX tickets or story points through every iteration without becoming too much of a drag on progress. After all, if you’re working agile (and who isn’t, these days?) constant improvements are the key.

How’s your developer experience looking?

Time, projects and the sparking of joy

Mon, 28 Nov 2022 06:27:00 +1300

I had a big realisation the other day. The most important currency in my life, right now, is time.

The value of time

We had already reached the point in this household where it was worthwhile for us to pay a cleaner, because we both really hate cleaning. Now we are thinking about a gardener. It is worth paying $6 worth of courier fee to mail order something even when it is 10 or 15 minutes drive away, because the time cost of getting in the car and going there is worth more than $6 (before you consider petrol or parking). Provided, of course, the item can be readily found and confirmed on the supplier’s website (insert rant about crap online shopping sites here).

Chatting with a colleague the other day he mentioned feeling similarly, but there are chores he gets value from doing. He really loves mowing his lawn… OK, his wife said, then we won’t pay someone to do it, but that’s your project to keep on top of.

There are not enough hours in the day (week) (month) (year) to achieve everything I want to. Now I must qualify that statement; I do lose a noticeable chunk of time by not having the energy to achieve more, and some more by procrastination. Sometimes after work I have zero oomph to do, well, very much at all. Maybe there’s some neurodivergence effect in there that confounds.

I have too many personal projects I want to do (let alone spending money on tools or toys I’ll hardly use, which is a side grumble in of itself). So it becomes imperative to choose which projects live, and which must suffocate. But is it a death of a project, or merely parking it indefinitely? Practically speaking there isn’t much difference from the project’s point of view. From my POV it is better to not feel overwhelmed by my to-don’t list, lest it become a source of doom and make-work, so I prefer to park things sufficiently far out of sight & mind that they might as well be dead.

I used to try to prioritise things in terms of an Eisenhower matrix. Maybe I should again, but it somehow seems lacking. The urgent and important stuff gets done; the hard part is figuring out how to rank the importance of the non-urgent projects. The value, perhaps; but it’s tough to quantify.

The value points scale

Somewhat idly, I propose the value points scale. These are akin to story points, for those who practice various flavours of agile. These aren’t absolute dollar values; they are relative. Start with Marie Kondo-style thinking of whether completing this project will bring joy, or some other emotion that you find valuable. If you find the term value points confusing, maybe call them Kondo points? There is no direct correlation between value and the effort required, though there will likely be some.

Fixing a light switch or tidying the garden is mostly a chore but brings me a little joy at the feeling of a job well done, so perhaps call that one kondo.
Making something out of wood - say, a table - is a big project that will give me some pride, perhaps quite a lot of pride if it involved new techniques or tools. I could maybe see that being 8 or 13 kondos.
Doing something worthwhile for a good cause you really care about might fill you with warm fuzzies. How many?
For those of us who love to write software, a personal passion project might come with a raft of kondos.
Art? Writing? Videography? These won’t all spark joy for everyone, so see if you can calibrate your own relative scale.

Like story points, use a modified Fibonacci scale for the allowable numbers of points. 0, 1, 2, 3, 5, 8, 13, 20, 40, 100. Maybe a ½ point for the really small stuff. But don’t overthink it. This is supposed to be a lightweight gut-feel metric. As you complete projects you may find your calibration improves.

Agile practitioners talk of a definition of done; with these projects you will have your own idea of what done means, how good a job you need to do to satisfy yourself and cash in the joy. But beware of creeping perfectionism… don’t let that 40 kondo project expand to take infinite time.

Then, you have to decide how to spend your available time. If you have only 1 spare hour in a week that may not be enough for big projects, so consider the quick wins: projects that won’t take long but which will bring some joy. If you are fortunate to have lots of time and energy in a day or a week, you might carve up your time to balance a reward from completing short projects, and progressing longer ones.

With larger projects the joy may not all come at once. You may find some joy at the milestone points. Only you can figure out whether this is part of your own calculus; you are your own project-joy-owner.

Cover image by vectorjuice on Freepik

Tensegrity table

Sun, 15 Nov 2020 17:07:00 +1300

Tensegri-what?

These intriguing creations appear to defy gravify.

On first encountering the concept, most people’s response is either “hey, that’s cool!” or “burn the witch!”

Most of the weight is taken by the short central chain while the others serve to balance the top and keep it (mostly) flat. There’s a little play in the system, just enough to be intriguing.

Mine is made from some kwila and white oak off-cuts, coated in Danish oil. It’s reversible, and the table top is acrylic.

Patio table

Tue, 06 Oct 2020 16:34:00 +1300

I wanted a decent patio table. I know how to work wood. There are plenty of plans for projects like this online. What could possibly go wrong…?

Well, in short: a lot. This project is truly a testament to the old adage that there are few truly fatal mistakes in woodworking. No, I won’t tell you where the issues are; you can probably find them in person if you care to look closely.

It’s made from kwila. This is commonly used for decking here in NZ. However, as timbers go it exudes some very staining run-off (tannins) if you get it wet, so you have to be careful what you do with it. In my case, I treated it with a chemical wash that draws out the surface tannins, then a decking product that effectively restores some of its lovely rich colour.

Now, if anybody asks me if I can make one for them too, my honest answer will be that it would be quicker and cheaper for them to go down to Mitre 10. Seriously. My project cost about $700 on timber alone. By the time you factor in several days’ work, consumables and wear on my tools – and that you can get a very similar (albeit mass-produced) table for $1000-1500 – who’d ever buy artisan woodwork? Nobody, unless they wanted a unique piece or wanted the journey of making it themselves.

The full photo story

Precisely mitred edge pieces

The top takes shape

Leg assembly

Lightroom plugin for Tangent control surfaces

Sun, 15 Dec 2019 21:31:00 +1300

It’s all about speed. If you have a lot of interactive colour correction to do, it’s so much faster to use a hardware panel than it is to use the mouse.

Professional film and video colour graders use hardware control surfaces all the time. Broadcast (video) engineers do something very similar with their CCUs.

This is a plugin for Adobe Lightroom Classic, which draws heavily on a similar plugin called MIDI2LR but for MIDI controllers.

~~You can get it from tangent-lightroom on Github.~~

Update: This project is obsolete! On 1 Apr 2021, Tangent released their own official plugin for Lightroom. It’s included in version 1.7.0 of the Tangent bundle, whish you can get from Tangent support.

The repository lives on, as a place for me to keep my mapping for the Wave:

Ross Younger

Lignum

Features

Getting started

A web toy: Forced ranking assistant

Getting value from CI

Introduction

Terminology: What is CI?

❔ Why CI

1. Automating stuff, so the humans don’t have to

2. Automatic analysis and status reporting

3. Improved quality

4. Reduced time to resolve issues

❌ Why not CI

Tests aren’t everything

Costs and maintenance

It might be too big for your platform

👷 Steps you can take

1. Build your units

2. Test your units

3. Integrate it

4. Add more checks

5. Deliver it

Closing thoughts

Announcing qcp

📋 Features

📖 About qcp

Motivation

💻 Getting qcp

See also

Broadcast graphics for fencing

Fault-finding at the ends of the earth

The Road Less Travelled

About Me

Contact information & GPG key

2023 in pictures

Faster C++ code style checks: detail

Before: 4.7 seconds 💩

After: 0.21 seconds 🚀

Developer tooling and boiled frogs

Background

CI chain 🔗

How much effort to spend on tooling? 💸

The frog boils 📉

A concrete walk-through 🔧

Code style checks: Parallelise!

Static analysis: Reconsider the load, cache, and parallelise where it will help!

Results 🚀

My next steps

Closing thoughts

Time, projects and the sparking of joy

The value of time

The value points scale

Tensegrity table

Patio table

Precisely mitred edge pieces

The top takes shape

Leg assembly

Lightroom plugin for Tangent control surfaces