Software on Ross Younger

Lignum

Tue, 14 Jul 2026 16:24:56 +1200

A source code repository visualiser with a 1,000’ view.

I wasn’t satisfied with the state of Git repository visualisers. I found they love to show you every last commit. When your repo is more than trivial, this is not useful for getting the big-picture.

Mine is different: it focusses on the macroscopic structure. Which are the active branches, where are the tags, where are the divergence and merge points?

In my first software job out of college – back when dinosaurs roamed the earth and CVS was the height of version control – we often ended up drawing what we termed railway diagrams on the whiteboard to visualise the structure. This is my attempt to automate creating these.

Features

Output in GraphViz dot format
Revision of interest filtering, with wildcards
Options to include all local branches (default), remotes, tags and root (parentless) commits
Edges are annotated with their size in commits (broadly the number on the direct line from A to B, though merge commits give different results)

Getting started

This is a Rust command-line application.

cargo install --locked lignum, or get it from the source repository.

Run lignum in a directory that contains a git repository, or give it --repo <path>. If you have xdot installed, the -x option will pipe the output directly to xdot.

Check the --help message for more.

A web toy: Forced ranking assistant

Mon, 21 Jul 2025 18:21:31 +1300

I wanted to arrange some items in a strongly ranked order of preference, but found it quite hard. So I made a little web tool that tries to help.

The idea? A sort of battle royale.

Consider two of the items chosen at random. Which do you prefer? Give it one point.
Repeat for every possible pairing of the items, and add up the scores.

The tool: https://crazyscot.github.io/forced-rank/

It’s a Javascript single-page application written in VueJS. Source code

Getting value from CI

Sun, 01 Dec 2024 10:53:36 +1300

Many of my peers within the software world know the value of Continuous Integration and don’t need convincing. This article is for everybody else.

Introduction

In my first job out of college we had what you’d recognise as CI, though the term wasn’t so popular then. It was powerful, very useful, but a source of Byzantine complexity.

I’ve also worked for people who didn’t think CI was worth doing because it was too expensive to set up and maintain. This is not totally unreasonable; the real question is to figure out where the value for your project might lie.

Recently, a friend wrote:

I don't really know very much about CI. I would be interested in knowing more and might even use some of the quick wins (...) I do not want to become completely reliant upon GitHub for anything.

So let’s start with a primer.

Terminology: What is CI?

Unfortunately the term “CI” is sometimes misused and/or confused.

The short answer is that it’s automation that regularly (continuously) does something useful with your codebase. These actions might take place on every commit, nightly, or be activated by some external trigger.

CI usually refers to a spectrum of practices, each step building on the last:

Continuous…	Typical activities
Build	Builds your code, usually to the unit or module level. Runs unit tests.
Integration	Assembles modules to a “finished application”, whatever that means. Runs integration tests.
Test	A full suite of automated tests. May include regression, performance, deployability and data migration.
Delivery	When the test suite passes, the latest version of the system is automatically released to a staging environment. This might involve building packages and putting them in a download area.
Deployment	When the automated tests pass, the software automatically goes live. Hold tight!

Exactly what these phases mean for your project, and how far you go with them, depends on your project.

What suits my embedded firmware probably won’t suit your cloud app or that other person’s desktop app.
The lines between the phases are blurry. For example, it may or may not make sense to build and integrate everything in one go.

❔ Why CI

If deployed appropriately, CI can save time, reduce costs and improve quality. Even on a hobby project, there is often value in saving your time.

1. Automating stuff, so the humans don’t have to

You could use your engineers to do the repetitive drudge work of creating a release across multiple platforms. You could have them run a full barrage of tests before committing a code change… but should you? Engineers are expensive and generally dislike boring stuff, so the smart business move is usually to automate away the repetitive parts and have them focus where they can deliver most value.

If you’re not sure, consider this: how much time does your team spend per release cycle on the repetitive parts? Consider your expected frequency of release cycles, that should lead you to the answer.

2. Automatic analysis and status reporting

One place I worked had a release process which relied on an engineer reading multiple megabytes of log file to see if things had been successful. Many things could go wrong and leave the final output in a plausible but half-broken state. Worse, it wasn’t as simple as running the script in stop-on-error mode, because some of the steps were prone to false alarms.

You may be ahead of me here, but I didn’t think much of that setup.

Compilation failed? Show me the compiler output from the file that failed.

A test failed? I want to see the result of that test (expected & observed results).

Everything passed? Great, but don’t spend megabytes to convey one single bit of information.

At its simplest, a small project will have a single main branch, and the operational information you need can be boiled down to a small number of states:



Something is broken	Non-critical warning (not all projects use this)	Everything is working

In a non-remote workplace it might make sense to set up some sort of status annunciator.

Some people use coloured lava lamps or similar.
At one place I worked the machinery in the factory had physical traffic light (andon) lamp sets. We set one of these up, driven by a Raspberry Pi wired in to the build server.
Some projects build more elaborate virtual dashboards that suit their needs. Multiple branches, multiple build configurations, whatever makes sense.

3. Improved quality

This one might be self-evident, but I’ll spell it out anyway.

A good CI system will let you incorporate tests of many different types, with variable pass/fail criteria. Think beyond unit and integration testing:

Regression (check that your bugs stay fixed)
Code quality (code/test coverage analysis; static analysis; dynamic memory leak analysis; automated code style checks)
Security analysis (are there any known issues in your dependencies?)
License/SBOM compliance
Fuzz testing (how does it handle randomised, unexpected inputs?)
Performance requirements
“Early warning” performance canaries
Standards compliance
System data migration
On-device testing (might be real, emulated or simulated hardware)

Performance canaries

Particularly where physical devices are involved, you might have a performance margin built in to your hardware spec. As the project evolves, inevitably new features will erode this margin. When you run out things are going to go wrong, so you want to take action before you get there.

An early warning canary is some sort of metric with a threshold. Examples might include free memory, CPU/MCU consumption, or task processing time. When the threshold is passed, that's a sign that things are getting tight and it's time to take pre-emptive action. You might plan to spend some time on algorithmic optimisations, or to kick off a new hardware spin.

If you can automate a really robust set of tests, you can have a lot of confidence in the state of your code at any given time. This gives incredible agility: you can release at any time, if the tests pass. This is the key to moving quickly, and is how a number of tech companies operate.

For a success story involving physical devices, check out the HP LaserJet team’s DevOps transformation.

4. Reduced time to resolve issues

If there’s one thing I’ve learned in the software business, it’s that it’s cheaper to find bugs closer to development - by orders of magnitude.

In other words, reduce the feedback cycle to reduce your costs. This is where automated tests and checks have great value.

If there is something wrong in code I modified a minute ago, I’m still in the right headspace and can usually fix it pretty quickly.
If it takes a few days to get a test result, I won’t remember all the detail and will have to refresh my memory.
If it takes several months to hear that something’s wrong, I may be working on a totally different part of the system and it will take longer to context switch.
If a bug report comes in from the field a year or two later, I might as well be starting again from scratch.

But - as ever - engineering is a trade-off. You can’t write a test to catch a bug you haven’t foreseen. It may be prohibitively expensive to test all possible combinations before release.

❌ Why not CI

CI is not suitable for all software projects.

If you’re writing a scratch throw-away project that won’t live for very long, even simple CI may not be worth it.

If you have a legacy codebase that was written without testing in mind, it might be prohibitively expensive to refactor to set these up. Nevertheless, in such projects there is often still some value to be found in a continuous build.

Let’s be pragmatic.

Tests aren’t everything

On the face of it, more testing means greater quality, right? Well… maybe?

Keep the end goal in sight. It’s up to you to decide what makes sense for your situation; I recommend taking a whole-of-organisation view.

You need to balance test runtime against overall feedback cycles. If the tests take too long to run, you’re slowing people down.
Some tests are expensive in terms of time or consuming resources, so you might not want to run them daily.
Tests involving physical devices can be difficult to automate, and risk creating a process bottleneck. (Consider emulation and/or simulation where appropriate.)
Beware of over-testing; you may not need to exhaustively check all the combinations. Statistical techniques might help you out here.
Beware of making your black-box tests too strict; this can lead to brittle tests that are more hassle to maintain than they are worth.

Costs and maintenance

It will take time and effort to set up CI. How much time and effort, I can’t say.

In times past, CI was quite the bespoke effort.

These days there is good tooling support for many environments, so it is usually pretty quick to get something going. From there you can decide how far to go.

It might be too big for your platform

CI platforms are designed for small, lightweight processes. Think seconds to minutes, not hours.

If you need to build a large application or a full Yocto firmware image, it’s going to be tough to make that fit within the limits of a cloud-hosted CI platform. Don’t despair! There are ways out, but you need to be smart. Alternative options include:

self-hosting CI runners that are take part in a cloud source repository;
self-hosting the CI environment (e.g. Gitlab, Jenkins, CircleCI), noting that most source code hosting platforms have integrations;
split up the task into multiple smaller CI jobs making good use of artefacts between stages;
reconsidering what is truly worth automating anyway.

👷 Steps you can take

1. Build your units

In most projects you already had to set up a buildsystem. Automating this is usually pretty cheap though you will need to get the tooling right.

Tooling on cloud platforms

On-cloud CI (as provided by Github, Gitlab, Bitbucket and others) is generally containerised. What this means is that your project has to know how to install its own tooling, starting with a minimal (usually Linux) container image.

This is really good practice! Doing so means your required tools are themselves expressed in source code under revision control.

Where this might get tricky is if you have multiple build configurations (platforms or builds with different features). Don’t be surprised if automating reveals shortcomings in your setup.

If you have autogenerated documentation, consider running that too. (In Rust, for example, it could be as easy as adding a cargo doc step.)

2. Test your units

Adding unit tests to CI is usually pretty cheap though it will depend on the language and available test frameworks.

If you want to include language-intrinsic checks (e.g. code style, static analysis) this is a good time to build them in. Some analyses can be quite expensive so it may not make sense to run all the checks at the same frequency.

3. Integrate it

If you’re pulling multiple component parts (microservices, standalone executables) together to make an end result, that’s the next step. Do they play nicely? Do you want to run any isolated tests among them before you move to delivery-level tests?

4. Add more checks

I spoke about these above.

This is where things stop being cheap and you have to start thinking about building out supporting infrastructure.

5. Deliver it

Now we’re getting quite situation-specific. Think about what it means to deliver your project.

Are you building a package for an ecosystem (Rust crate / Python pypi / npm.js / …) ? You might be able to automate the packaging steps and that might be pretty cheap.

Are you building an application? Perhaps you can automate the process of building the installer / container / whatever shape it takes. If you have multiple build configurations or platforms, it could get very tedious to build them all by hand and there is often a win for automation.

Where there's code signing involved, you'll need to decide whether it makes sense to automate that or leave it as a manual release step. Never put private keys or other code signing secrets directly into source! Some platforms have secrets mechanisms that may be of use, but it pays to be cautious. If your secrets leak, how will you repair the situation?

Closing thoughts

Most projects will benefit from a little CI. You don’t need to have unit tests, though they are a good idea.
You’re going to have to maintain your CI, so build it for maintainability like you do your software.
Apply agile to your CI as you do to your deliverables. Perfect is the enemy of good enough. Build something, get feedback, iterate!
CI vendors want to lock you in to their platform. Keep your eyes open.
Don’t let CI become an all-consuming monster that prevents you from delivering in the first place!

Announcing qcp

Sun, 03 Nov 2024 10:23:43 +1300

The QUIC Copier (qcp) is an experimental high-performance remote file copy utility for long-distance internet connections.

Source repository: https://github.com/crazyscot/qcp

📋 Features

🔧 Drop-in replacement for scp
🛡️ Similar security to scp, using existing, well-known mechanisms
🚀 Better throughput on congested networks

📖 About qcp

qcp is a hybrid protocol combining ssh and QUIC.

We use ssh to establish a control channel to the target machine, then spin up the QUIC protocol to transfer data.

This has the following useful properties:

User authentication is handled entirely by ssh
Data is transmitted over UDP, avoiding known issues with TCP over “long, fat pipe” connections
Data in transit is protected by TLS using ephemeral keys
The security mechanisms all use existing, well-known cryptographic algorithms

For full documentation refer to qcp on docs.rs.

Motivation

I needed to copy multiple large (3+ GB) files from a server in Europe to my home in New Zealand.

I’ve got nothing against ssh or scp. They’re brilliant. I’ve been using them since the 1990s. However they run on top of TCP, which does not perform very well when the network is congested. With a fast fibre internet connection, a long round-trip time and noticeable packet loss, I was right in the sour spot. TCP did its thing and slowed down, but when the congestion cleared it was very slow to get back up to speed.

If you’ve ever been frustrated by download performance from distant websites, you might have been experiencing this same issue. Friends with satellite (pre-Starlink) internet connections seem to be particularly badly affected.

💻 Getting qcp

The project is a Rust binary crate.

You can install it:

as a Debian package or pre-compiled binary from the latest qcp release page (N.B. the Linux builds are static musl binaries);
with cargo install qcp (you will need to have a rust toolchain ~~and capnpc~~ installed) (the capnp dependency was removed later);
by cloning and building the source repository.

You will need to install qcp on both machines. Please refer to the README for more.

Broadcast graphics for fencing

Thu, 15 Aug 2024 21:16:22 +1200

I created a TV graphics package for fencing tournaments.

Earlier this year, Christchurch played host to the Commonwealth Junior & Cadet fencing tournament.

Selected parts of the tournament were livestreamed, with a package broadcast on Sky TV (NZ). The broadcast and finals streams had a live graphics package fed from the scoreboard.

The programmes were produced using a broadcast-spec OB truck supplied by Kiwi Outside Broadcast. The truck graphics PC used Captivate to generate its graphics, which output as key+fill SDI signals. These were fed to the vision mixer and keyed onto the picture in the usual way.

The package can be seen in action on the Commonwealth Junior & Cadet 2024 programmes.

The details read from the scoreboard cover the “hit” lamps, scores, clock (including fractional seconds in the last 10s), period, red/yellow cards, and the priority indicator. On top of that, the package provides a place to enter the fencer names and nationalities, and set colours for them.

This is all made possible by the scoreboard, a Favero FA-07, offering a data feed over an RS-422 interface. I wrote a Python script to parse the data feed, turn it into a JSON dictionary and pass on to Captivate.

Fault-finding at the ends of the earth

Mon, 29 Apr 2024 13:29:43 +1200

This is a tale from many months ago, working on an embedded ARM target.

In my private journal I wrote:

Today I feel like I saddled up and rode my horse to the literal ends of the earth. I was fault-finding in the setting-up-of-the-universe that happens before your program starts up, and in the tearing-it-down-again that happens after you declare you’re finished.

If you know C++, you might guess that this was a story about static object allocation and deallocation. You’d be right. So, destructors belonging to static-allocated objects. You’d never think they’d run on a bare-metal embedded target.

Well, they can. If your target supports exit() - e.g. if you are running with newlib - then an atexit handler is set up for you, and that will be set up to run the static destructors. If your program then calls exit() (as, say, your on-silicon unit tests might, at the end of a test run) then things are at risk of turning to custard.

You might have enabled an interrupt for some peripheral on the silicon. In order to do anything really useful, the ISR might reference a static object. If you do this, you’d damn well better make sure the object has a static destructor that disables the interrupt, or hilarity is one day going to ensue. You know, the sort of hilarity that involves being savaged by a horde of angry rampaging badgers, or your socks catching fire.

But wait, I hear you say, it called exit! The program no longer exists! Well, sure it doesn’t; but what happens on exit? On this particular ARM target, running tests via a debugger as part of a CI chain, when the atexit handlers have run the process signals final completion with a semihosting call, which is a special flavour of debug breakpoint. It is… not fast. If your interrupt happens regularly, the goblins are going to get you before the pseudo-system-call completes. Your test framework will fail the test executable for hanging, despite somehow passing all of its tests.

There was an actual bug in there, and it was mine. Class X, which contained an RTOS queue and enabled an interrupt, only had a default destructor. On exit, somewhen between static destructors and completion of the semihosting exit call, the ISR fired. It duly failed to insert an item into the now-destroyed queue, so jumped to the internal panic routine. That routine contained a breakpoint and then went nowhere fast, waiting for a debugger command that was never going to arrive — hence the time-out. Maybe it would have been useful to have a library option to skip the static destructors, but I probably wouldn’t have been aware of it ahead of time anyway.

The static destructor ordering fiasco can also be yours for the taking, but thankfully that hadn’t bitten me. Nevertheless, it was a rough day.

Cover image: Cyber Bug Search, by juicy_fish on Freepik

Faster C++ code style checks: detail

Thu, 17 Aug 2023 22:04:01 +1200

xargs, nproc and shell arrays, oh my!

This is a companion article to Developer tooling and boiled frogs.

Before: 4.7 seconds 💩

In this codebase, style checks were performed by clang-format with an in-house style file. They ran in a pre-commit hook (as well as in CI). However, the pre-commit hook is critical for a good DX; 5 seconds really sucks.

The entry point to the check was a shell script which invoked clang-format, amongst other things. Like all such scripts, it was not truly original, being assembled from snippets and ideas found around the internet. Here’s the gist of it:

check_files() {
    # arguments are filenames. if any failed, returns 1; if all passed, returns 0
    err=
    for file in $@; do
        ext=${file##*.}
        case "$ext" in
            c|cpp|h|hpp)
                if ! clang-format -Werror -style=file -n "$file" ; then
                    err=1
                    echo "code style check failed for $file"
                fi
                ;;
            *)
                # unsupported file type; ignore
                ;;
        esac
    done
    return $err
}

# ...

check_files "$(git ls-files)"

Do you see the problems?

It’s single-threaded! Almost all desktop and laptop CPUs have had multiple cores for 15 or more years, so this is truly Dark Ages style processing.
We’re invoking clang-format far more times than we need to. Every invocation has to parse the config file, which is wasteful.

After: 0.21 seconds 🚀

check_files() {
    # arguments are filenames. if any failed, returns 1; if all passed, returns 0
    FILES=()
    for file in $@; do
      ext=${file##*.}
      case "$ext" in
        c|cpp|h|hpp)
          FILES+=("$file")
          ;;
        *)
          # unsupported file type; ignore
          ;;
      esac
    done

    FILES_COUNT="${#FILES[@]}"
    CORES=$(nproc)
    # CORES+3 gave best performance for me. YMMV.
    JOB_SIZE=$((FILES_COUNT / (CORES+3)))
    if [ ${JOB_SIZE} == 0 ]; then JOB_SIZE=1; fi
    echo "${FILES[@]}" | xargs -P 0 -n ${JOB_SIZE} clang-format -Werror -style=file -n || return 1
    return 0
}

There’s a bit to unpack here:

We’ve moved from sh to bash to take advantage of array variables.
Rather than immediately running clang-format on every file that passes the filter, add them to the FILES array so we can use xargs. (Yes, I know this could be better structured. This was but a single step of evolution; first, make it work.)
We use xargs in parallel mode.

We’ve naively assumed that checking each file takes roughly the same amount of time. This isn’t quite true (I found about a factor of 3 variation amongst them) but the performance gains were good enough that it doesn’t seem worth worrying about at the present time.

The key here is combining -n $JOB_SIZE with -P 0 (unlimited parallelisation) and doing some shell arithmetic to figure out what that job size should be.

xargs with neither option runs a single clang-format with all the input files in a single invocation.
- This is good (about a 6x speedup) but we can do better! It is still single-threaded, iterating through the files in order.
xargs -P $(nproc) sounds interesting but is no better on its own. The man page says it all:

Use the -n option or the -L option with -P; otherwise chances are that only one exec will be done.
I’ve counted the files, divided by (slightly more than) the number of processor cores available, and told xargs to parcel the files out in jobs of that size. Of course, if you’re doing this yourself, you’ll want to repeat the experiment to figure out the sweet spot for your hardware and your codebase.

🚅 Final speedup for this activity: 22.3x 🚀 (timings as measured on my laptop for that codebase on that day; your mileage may vary, may contain nuts, etc.)

Developer tooling and boiled frogs

Thu, 17 Aug 2023 22:04:00 +1200

This is an article about software tooling and the developer experience.

Background

Right now I am spending most of my work time on an embedded project. It’s C++ running on bare-metal target hardware. I’ve got all of 768K of RAM to play with, but that’s still a tad more than my first home micro had.

Our tooling and infrastructure grew mostly organically. I was the one who initialised the git repo.

CI chain 🔗

We have a CI chain, of course. It has grown over time and does a few tricks:

build for the target hardware (various targets; RAM and flash builds)
build on Linux native port and run unit tests
perform various analyses (clang-tidy, sonarcloud, code coverage, valgrind)
code style enforcement (clang-format) 👮
prepare a signed upgrade image with the development signing key
assemble a whole-of-flash image for the factory 🏭
build and run some unit tests on a development board

The dev board is on-premises but the rest happens in the cloud, taking a matter of minutes on every push. It’s reasonably slick.

You can run all the build and analysis steps locally. We have a script that spins up a container emulating our CI provider so you don’t even need to have the dev tools installed, you only need to pull our in-house container image.

How much effort to spend on tooling? 💸

Many a savvy manager or product owner will tell you to not spend too much time on tooling. They have a point; time spent working on tooling and infra is not delivering features, after all. Many managers have been bitten by infrastructure blow-outs and would rather we focus on building the stuff they want us to build.

But here’s the thing. When they say that, they don’t mean “forget about tooling and never speak of it again”. You are allowed to bring the subject up again in future, and you are allowed to try to educate them as to the value.

The frog boils 📉

We now have over 300 source files that get compiled into the image, which includes some vendor and autogenerated code.

There’s the rub. We didn’t start out with anywhere near that many. The CI chain used to be really fast, under one minute. Local builds were fast, and even running the clang-tidy script wasn’t that awful (though I did build a cache for it, which helped).

The size and complexity grew over time, and here we are today.

It’s like the old tale about boiling frogs. (Disclaimer: I don’t even know if this is true, and I definitely don’t suggest you try it, but it’s an interesting thought experiment.)

What happens if you drop a frog into a pot of boiling water?

The frog doesn’t like it and jumps out.

But what happens if you drop a frog into a pot of cold water, then light the stove? The frog doesn’t realise anything is wrong. It boils to death before it thinks to jump out.

Builds with a cold cache were taking a couple of minutes locally, and about 8 minutes in the cloud. (This happens depressingly often when you change a low-level header that is included all over the place… there’s another observation, that maybe our architecture needs work, but that’s for another day.)

A full static analysis with cold cache was taking over 5 minutes locally, and 22 in the cloud. (The difference is because our company-provided laptops have 16 cores, while our CI provider runs our container with 4 vCPU cores and tightly limited RAM.)

Goodness me, 300 compile units now? This water sure is getting warm! 😰

It’s a poor developer experience. Even running the code style checker script takes five whole seconds. That’s 5 seconds in which you can get distracted, pop over to Slack, and whoops! ten minutes later, you’re trying to remember what it was you were doing.

I’ve been in this business long enough to know that keeping myself and my team focussed is vitally important to getting things done and keeping the stakeholders happy. An efficient DX is part of that puzzle.

Like the mythical frog, I don’t remember noticing these tasks growing longer.

A concrete walk-through 🔧

OK, so you’ve decided you need to jump out of the pan and spend some time sharpening your tools. Great.

The most important advice I can offer is to understand where the pain points are. 🔍

If this is stuff you work with every day, you maybe already know where the pain points are but are blind to them because you haven’t noticed how poor the DX has become. Open your eyes, and you may find it is in plain sight. Don’t optimise for its own sake; optimise where it will help.

Here’s what I noted about our tooling:

The compilation time itself is already pretty well optimised and parallelised thanks to meson, ninja and ccache.
The code style checker is something we run very frequently as part of our workflow (it’s in our pre-commit script). At 5 seconds, that’s like sand in your gears.
The static analyser is a different type of pain point. We run it interactively but we don’t like to because it’s a slow ol’ beast. On the cloud, it is the slowest part of the build; it is probably the reason why we’re regularly incurring overage on our CI build minutes. (Now, that’s not a big cost compared to an engineer’s salary, but it’s not zero and it has been noticed, so maybe worth spending a little time on.)
The unit test run on a dev board is a bottleneck, practically by definition, as it uses a self-hosted CI runner plugged into our pipeline.

Code style checks: Parallelise!

We use clang-format with an in-house style file.

The heart of our checker script is a shell function.

The tl;dr version is:

We were invoking clang-format in a tight loop, once per input file. 🤦
Now, we count the files, divide by (slightly more than) the number of CPU cores available, and have xargs parcel out the work into that many chunks. 🚅

Static analysis: Reconsider the load, cache, and parallelise where it will help!

The learnings from this one are:

Do the heavy lifting in parallel (we were already)
Cacheing positive analysis results helps (we were doing that too)
Don’t analyse unit tests or test harness/mocks/fakes unless you have a good reason
Don’t analyse target-specific files for obsolete hardware targets unless you have a good reason
Where you have a string-and-glue shell phase to prepare the analysis run, it can be worth parallelising that as well.

Results 🚀

(Timed on my work laptop, which has 16 cores. It’s running Ubuntu 22.04 on WSL on Windows 11.)

Action/Phase	Time before	Time after	Speed-up
Code style check	4.7 seconds	210 ms	22.3x
Static analysis setup	14 seconds	3.5s	4x

My next steps

The on-target testing phase is a bottleneck, so we’re going to add more hardware.
There might be some mileage in using using pre-compiled headers to speed up cold-cache compilation.
- This isn’t something the places I have worked have traditionally done, but I know it’s popular when you’re dealing with the Win32 C++ api.
- Our codebase has a “platform” layer; this smells like a good starting point for precompilation.
Coverage analysis. We already do this, but there’s another pain point while genhtml crunches the data and generates the HTML report. This one could be trickier; it’s a single-threaded perl script, invoked by meson.
Stack usage analysis. We’ve got some code paths that can be provoked to overflow on debug builds. This is a different sort of pain point; while we’ve configured the silicon to detect and hard-fault on overflow, I’d like to set up some analysis to automatically find these situations ahead of time.

Closing thoughts

How long to spend on infrastructure? 💸

Circling back to a question I asked earlier, how long is a piece of string?

It depends where you are and what your current DX is like.

10-20% of your developer time seems a reasonable benchmark; it’s enough that you can get a handful of DX tickets or story points through every iteration without becoming too much of a drag on progress. After all, if you’re working agile (and who isn’t, these days?) constant improvements are the key.

How’s your developer experience looking?

Lightroom plugin for Tangent control surfaces

Sun, 15 Dec 2019 21:31:00 +1300

It’s all about speed. If you have a lot of interactive colour correction to do, it’s so much faster to use a hardware panel than it is to use the mouse.

Professional film and video colour graders use hardware control surfaces all the time. Broadcast (video) engineers do something very similar with their CCUs.

This is a plugin for Adobe Lightroom Classic, which draws heavily on a similar plugin called MIDI2LR but for MIDI controllers.

~~You can get it from tangent-lightroom on Github.~~

Update: This project is obsolete! On 1 Apr 2021, Tangent released their own official plugin for Lightroom. It’s included in version 1.7.0 of the Tangent bundle, whish you can get from Tangent support.

The repository lives on, as a place for me to keep my mapping for the Wave:

Broadcast graphics package

Tue, 10 Apr 2018 10:25:03 +1200

This is a package for the CasparCG playout engine.

It comprises score bugs, a score lower third, generic lower thirds, and a count-up/count-down clock.

The templates are written in HTML, CSS and JavaScript.

The package comes with a a custom operator panel written in Python and wxWidgets.

Broadcast graphics product page

find-definition: a helper script for programmers

Fri, 01 Mar 2013 20:09:23 +1300

Opens up a new editor directly at the definition of a type, structure, function or anything else that ctags reports.

I use it with vim. With other editors you either need to have ctags support (contributions welcome!) or be prepared to use vi as an external source browser.

Browse on github

git clone https://github.com/crazyscot/find-definition.git

brot2: a fractal plotter

Thu, 15 Dec 2011 21:17:34 +1300

Revisiting the work of a great mathematician.

Banner image: Mandelbrot centred at (0.250648691259948, 2.6343326303079e-05), maxiter=16384, horizontal axis=3.176e-06

When Benoit Mandelbrot passed away in 2010, I was inspired to revisit his work (as, it turns out, were some of my friends). This is my second Mandelbrot plotter; the first was cryptically named ‘brot’, was pretty basic, and is lost to the mists of time; this one is an interactive GTK+ application.

Mandelbrot@(-0.67, 0.1) max=768 axis=(3,3)

Feature list

mouse-driven navigation (click to zoom in/out and recentre);
specific zooming by dragging out a rectangle;
multiple extensible palettes;
linear antialiasing;
save image as PNG;
draws a number of related fractals as well as the original Mandelbrot set and can be relatively easily extended to add more.

Getting brot2

brot2 is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

brot2 has a project page on LaunchPad. You can get it:

From the releases page;
from its its release builds PPA;
from the daily-build PPA; or
by cloning from the repositiory and building it yourself:
- git clone git://github.com/crazyscot/brot2.git
- browse

In order to build brot2, you will need to have various gtk and gtkmm developer libraries installed (amongst others). They are listed in the debian/control file (and taken care of if you use the standard package building tools).

Mandelbrot@(0.250650639435726,
2.7211535074015e-05) max=24576
axis=(3.176e-06,3.176e-06)

Using brot2

You can move around the plot using your mouse (or equivalent pointing device). The first time you run brot2, the Controls window appears; you can hide and show it via the Options menu. By default the controls are:

Left-click to recentre around the spot clicked on.
Drag with the right button to mark out a rectangle to zoom in to. (It may be adjusted to fit the window.)
Middle-click to zoom out around the spot clicked on.
Button 8 (the 4th button on my Kensington Expert Mouse) to zoom in, centred on the spot clicked on.
Scroll wheel up and down zoom in and out, centring around the pointer’s current location.
Keyboard ‘+’ and ‘-’ zoom in and out around the current centre of the plot.

There is a simple single-step undo mechanism. This is designed to let you recover from the “oh dear, I clicked somewhere silly, where was I?” scenario.

The current fractal position is shown in the heads-up display overlaid onto the window. You can change its size, colour, transparency and positioning via the Preferences dialog.

If you want to specify your plot numerically, you can do this via the Parameters dialog.

The plot and window sizes are always the same; if you resize the window, the plot is redrawn to fit.

You can save the current plot as a PNG file, with the option to save at a different size. (Changing size may take a while as the image has to be re-plotted.) The HUD is not drawn onto a saved image.

For better image quality, activate the antialiasing mode (Ctrl-A). This takes longer to draw as it has to plot four times as many points and average across them. (This is a pretty naive linear antialiasing; better ways remain to be found and implemented.)

brot2 has the notion of a plot being “good enough” to stop computing. This is expressed as a fraction (percentage) of the pixels which should have escaped (or be provably within the set), and a proportion of pixels continuing to escape in each pass. This is a tricky compromise between drawing enough detail and not taking forever, and it doesn’t always get it right; you can tune it via the Preferences dialog. You can always stop a plot with Ctrl-. or ask it to make another pass (“More iterations”) with Ctrl-M.

There are multiple colour palettes. At the moment these are hard-coded. Changing colour scheme is a quick and cheap operation; you can even cycle through them with Ctrl-3 and Ctrl-4.

Mandelbrot@(-0.761228719917374,
0.0847858796416676) max=3072
axis=(4.643e-10,4.643e-10)

What’s in a name?

This is my second Mandelbrotter.

The first was cryptically named brot; both the origins of its name, and the code, are lost in the mists of time. It was a bit pants, even by the standards of the time; it took its parameters on the command line and output as a TIFF, ASCII art or to a terminal which understood ANSI graphics codes. (I claim this as a dubiously legitimate reason for setting xterm to ‘unreadable’ :-) )

This one is a bit better featured, but to me it’s logically the heir to its now-departed ancestor, hence the name.

CurrentCost monitoring for RRDtool

Wed, 18 Aug 2010 12:11:52 +0100

Scripts to read data from a CurrentCost CC128 electricity consumption monitor and drop it into an RRDtool database.

They are based on Paul Mutton’s CurrentCost serial port script. My version adds:

minicom-compatible serial port locking
autodetection in case of multiple ttyUSB devices

Browse on github

git clone https://github.com/crazyscot/currentcost.git

chiark-cryptdisk: scripts for encrypted backups

Wed, 11 Aug 2010 11:39:19 +0000

This package is only available in source form for Debian systems.

It contains machinery to build a Debian package in the usual way. Be aware that it is not really polished and will require a certain amount of effort to integrate into a backup solution.

Browse on github

git clone https://github.com/crazyscot/chiark-cryptdisk.git

Using the ZTE MF626 on Ubuntu and T-Mobile

Wed, 10 Feb 2010 19:00:23 +0000

The ZTE MF626 is a 3G modem stick resold by T-Mobile in the UK.

They call it the “USB stick 120”; other networks likely call it something different. Rumour had it that it ought to work under Ubuntu; Google brought me to http://ubuntuforums.org/showthread.php?t=1147685 which contains some useful info for Jaunty (9.04) but turns out to be a bit out of date with regard to Karmic (9.10).

The USB ZeroCD oddity

The MF626 is not your average USB device. It's one of a growing trend of ZeroCD devices: when you plug them in, they report themselves as being of type USB mass storage (in this case, 19d2:2000 - check it with lsusb), with an autorun.inf which installs the drivers and other paraphernalia. When you have the driver, it issues a magic command to the device, which causes it to magically change its USB product ID to that of the real modem device (in this case, 19d2:0031).

ZeroCD on Ubuntu

It used to be the case that you had to install a program called usb_modeswitch which sent the magic command to the device. (It's in the Ubuntu archive.) Unfortunately, this didn't work reliably for me; on one machine, I eventually had it working most of the time, but there was no such luck on my laptop.

Almost by accident, I tripped over some steps which seem to work every time:

Insert the USB stick
Watch /var/log/messages to get the device node for the SCSI CDROM it thinks you just inserted (sr1 usually). This takes a few seconds to come in as the device boots up and udev figures out what it appears to be.
sudo eject /dev/sr1
Wait a few more seconds - watch messages if you like - then run lsusb to confirm the device is now 19d2:0031.

Geeky side note: Once you have successfully mode-switched, the device appears as three ttyUSB devices. You can fire up minicom at this point on at least two of the three - 115200 8N1 with hardware flow control works - and send it AT commands :-)

Obsolete usb_modeswitch notes:

The package in karmic comes with /etc/udev/rules.d/usb_modeswitch.rules which have the ZTE MF620 enabled by default. You'll have to edit it and comment out that stanza; I did try the stanza for the MF626 but couldn't get it to work at all.
Editing /etc/usb_modeswitch.conf to uncomment the relevant lines for the ZTE MF626 allowed it to work from the command line (sudo usb_modeswitch [-W]) - most of the time, on one machine, but not at all on the other. I found I did need to be patient and wait several seconds between insertion of the device and invoking the program; this might be what stops the udev rule from working.
USB modeswitch home page

Dial-up settings

Reportedly the NetworkManager is a bit flakey with USB modems in jaunty, but I had no problem in karmic.

Ensure modemmanager is installed.
Create the file /usr/share/hal/fdi/information/20thirdparty/20-zte-mf626.fdi as described at step 10 of http://ubuntuforums.org/showthread.php?t=1147685.
(Remove and re)insert the device and "eject" it as described above
Be sure to wait for the modem device to settle down. I found this can take up to 15 seconds from issuing the eject.
Click on the NetworkManager icon in the menubar. If the device has been recognised, you will see menu items for Mobile Broadband, and the option to create a new connection. (If not, wait a few more seconds and try again.)
The wizard asks you pertinent questions; thankfully, under T-Mobile there is only the one price plan and no need to enter a username or password. You should now be ready to dial up.

Once the connection is set up, the steps boil down to:

Insert stick;
wait a bit;
"eject" it;
wait a bit;
select the connection name from the NetworkManager drop-down list;
wait until NetworkManager tells you you're connected, and a couple more seconds for resolv.conf to be updated (also see below).

Alternatives to NetworkManager

Anecdotally from the forums, wvdial is said to be more reliable than NetworkManager. There are also some people in the Ubuntu world who refuse to use NetworkManager, preferring to do things the old way; for them, wvdial is obviously the way. (I used to be one such person. I have to say, it has been ten years since I last used wvdial, and I found it rock solid back then, so if I had had problems with NetworkManager then wvdial would be my next port of call.)

I haven't tried it myself, but the ubuntu forums link above does mention wvdial as a viable alternative. The device appears to present itself as an AT-compatible modem; NetworkManager seems to have it dial `*99***1#` .

Getting online with T-Mobile

DNS servers

I've found that, once in a while (and in fact the very first time I connected), NetworkManager doesn't manage to set up /etc/resolv.conf correctly when it connects.

At the time of writing, T-Mobile's nameservers for mobile broadband users are 149.254.192.126 and 149.254.201.12. I also noted that 8.8.8.8 (Google's public DNS service) also worked just fine.

If this turns out to happen frequently, NetworkManager does let you set the connection to be PPP "addresses only" with manually-specified nameservers.

Escaping the PAYG walled garden (registering a payment card)

My USB stick is on a pay-as-you-go account. This means that when you first dial up, www requests are redirected to the T-Zones site. (You can also access t-mobile.co.uk, but that seems to be about the limit.) You have to use T-Zones to select how many days usage you want to pay for; naturally, the week and month bundles work out better if you're going to use them.

So far so good. Unfortunately, the initial credit on the account had expired so I needed to top-up. I wanted to top-up online as this was much more convenient than finding my nearest top-up retailer or ATM. To do this, you have to first register yourself with T-Mobile and register the device with your account; then you can add a payment card and make a top-up.

The fun came in registering the device to my online account: you enter the device phone number and it sends you an SMS with a 4-digit confirmation "PIN". Right. Yes. This is a lighthouse USB stick, and I don't have SMS software installed. (Aside: Hello? surely you can tell that I'm connected from a tmo line and skip that step?)

Side note: A package called wader exists and claims to be a more fully featured mobile broadband connection manager, doing SMS as well, though it conflicts with the modemmanager package; I haven't tried it.

There was nothing else for it; I reluctantly booted into Windows and let the device install its own driver. The connection manager it installs is functional, if obviously cheaply developed, and allows you to receive SMSes. Great. But the SMS containing the code was apparently lost into the ether, and the T-Mobile site wouldn't resend it ... Thankfully, I was able to work around this by removing the device from my online account and re-adding it. At last, I was able to register a payment card, put some credit on the account and actually spend my £2 to get online!

Once you have selected a package and debited your PAYG account appropriately, it seems you can disconnect and reconnect as many times as you like during the connection period you've paid for and not be troubled by the walled garden. Nevertheless you can go back to www.t-zones.co.uk at any time to see your remaining credit.

Disabling ContentLock

By accident, I happened to trip over ContentLock. This is T-Mobile's attempt to do their bit to prevent children from accessing content deemed unsuitable, including "social networking sites" and sites with "user-generated content", which includes Flickr, that dangerous bastion of ... er ... something. You'll know if you hit ContentLock, because you get redirected to a message which tells you that access is blocked and purports to tell you how you switch it off. Unfortunately, actually switching it off took some doing...

(If the next few paragraphs are tl;dr, you essentially have two options: to take proof of ID into a T-Mobile store, or to put the SIM into a regular mobile, dial 1818 and provide credit card details.)

The you-are-blocked message offers to allow you to remove the lock on the spot, if you have a credit card to "prove" you're 18: they propose to charge and immediately refund £2. Unfortunately, this didn't work; the site took my card details then claimed to be unable to determine what phone number I was accessing it from, which didn't exactly leave me brimming with confidence.
- (Err, hello? Have these people never heard of authorised user cards?)
- Side note: The ContentLock portion of their infrastructure is obviously a quickly-written afterthought - it's plain HTML with no CSS.
Apparently - going on ContentLock's say-so - I could switch off the lock via my T-Mobile account. So I went there and it wanted me to select my "phone" model before it could show me the options... but USB sticks were not in the list! To summarise my googling, this doesn't seem to work for PAYG accounts. Sigh.
Next, I tried ringing T-Mobile customer services. (The number is 150 from T-Mobile, or 07953966150 from other networks.) The call centre operator asked the number I was calling about, then apologetically said she couldn't access PAYG accounts and put me through to the PAYG people. The PAYG "people" turned out to be an automated menu system. There is an option for ContentLock... but it only provides information and tells you to ring 1818.
I tried ringing 1818 from my mobile, but (surprise surprise) that system only works if you're calling from the line you want to en/disable it for. How are you expected to dial 1818 from a USB stick?!
I noted at this point that I could take proof of ID to a T-Mobile store in person, but I didn't want the hassle of going into town.
On a whim, I removed the SIM card from the USB stick and put it into my mobile. It powered up and connected to the network happily; I dialled 1818; keyed in my credit card details and voilà, after a delay, the voice told me it had removed ContentLock from the line. Sure enough, on popping the SIM back into the stick, I could access Flickr.

Conclusion: ContentLock is a hastily-developed addition to their network, probably added to pay lip service to some regulatory requirements, to legal due diligence or to "think-of-the-children" pressure. It seems to go a bit far with the sites it blocks, but was at least relatively straightforward to switch off.

Wishlist

Be able to receive SMSes under Linux. This is important in case you get close to your fair-use quota / data cap / whatever your telco calls it. (Sending would be nice but is optional for me as I have a regular mobile, and T-Mobile users can also send texts via their website.) There might be some mileage in gnome-phone-manager.

Software on Ross Younger

Lignum

Features

Getting started

A web toy: Forced ranking assistant

Getting value from CI

Introduction

Terminology: What is CI?

❔ Why CI

1. Automating stuff, so the humans don’t have to

2. Automatic analysis and status reporting

3. Improved quality

4. Reduced time to resolve issues

❌ Why not CI

Tests aren’t everything

Costs and maintenance

It might be too big for your platform

👷 Steps you can take

1. Build your units

2. Test your units

3. Integrate it

4. Add more checks

5. Deliver it

Closing thoughts

Announcing qcp

📋 Features

📖 About qcp

Motivation

💻 Getting qcp

See also

Broadcast graphics for fencing

Fault-finding at the ends of the earth

Faster C++ code style checks: detail

Before: 4.7 seconds 💩

After: 0.21 seconds 🚀

Developer tooling and boiled frogs

Background

CI chain 🔗

How much effort to spend on tooling? 💸

The frog boils 📉

A concrete walk-through 🔧

Code style checks: Parallelise!

Static analysis: Reconsider the load, cache, and parallelise where it will help!

Results 🚀

My next steps

Closing thoughts

Lightroom plugin for Tangent control surfaces

Broadcast graphics package

find-definition: a helper script for programmers

brot2: a fractal plotter

Mandelbrot@(-0.67, 0.1) max=768 axis=(3,3)

See also

Feature list

Getting brot2

Mandelbrot@(0.250650639435726, 2.7211535074015e-05) max=24576 axis=(3.176e-06,3.176e-06)

Using brot2

Mandelbrot@(-0.761228719917374, 0.0847858796416676) max=3072 axis=(4.643e-10,4.643e-10)

What’s in a name?

CurrentCost monitoring for RRDtool

chiark-cryptdisk: scripts for encrypted backups

Using the ZTE MF626 on Ubuntu and T-Mobile

The USB ZeroCD oddity

ZeroCD on Ubuntu

Obsolete usb_modeswitch notes:

Dial-up settings

Alternatives to NetworkManager

Getting online with T-Mobile

DNS servers

Escaping the PAYG walled garden (registering a payment card)

Disabling ContentLock

Wishlist

Mandelbrot@(0.250650639435726,
2.7211535074015e-05) max=24576
axis=(3.176e-06,3.176e-06)

Mandelbrot@(-0.761228719917374,
0.0847858796416676) max=3072
axis=(4.643e-10,4.643e-10)