Launchpad now supports SSH Ed25519 keys and RSA SHA-2 signatures

As of 2022-02-16, Launchpad supports a couple of features on its SSH endpoints (git.launchpad.net, bazaar.launchpad.net, ppa.launchpad.net, and upload.ubuntu.com) that it previously didn’t: Ed25519 public keys (a well-regarded format, supported by OpenSSH since 6.5 in 2014) and signatures with existing RSA public keys using SHA-2 rather than SHA-1 (supported by OpenSSH since 7.2 in 2016).

I’m hesitant to call these features “new”, since they’ve been around for a long time elsewhere, and people might quite reasonably ask why it’s taken us so long. The problem has always been that Launchpad can’t really use a normal SSH server such as OpenSSH because it needs features that aren’t practical to implement that way, such as virtual filesystems and dynamic user key authorization against the Launchpad database. Instead, we use Twisted Conch, which is a very extensible Python SSH implementation that has generally served us well. The downside is that, because it’s an independent implementation and one that occupies a relatively small niche, it often lags behind in terms of newer protocol features.

Catching up to this point has been something we’ve been working on for around five years, although it’s taken a painfully long time for a variety of reasons which I thought some people might find interesting to go into, at least people who have the patience for details of the SSH protocol. Many of the delays were my own responsibility, although realistically we probably couldn’t have added Ed25519 support before OpenSSL/cryptography work that landed in 2019.

  • In 2015, we did some similar work on SHA-2 key exchange and MAC algorithms.
  • In 2016, various other contributors were working on ECDSA and Ed25519 support (e.g. #533 and #644). At the time, it seemed best to keep an eye on this but mainly leave them to it. I’m very glad that some people worked on this before me - studying their PRs helped a lot, even parts that didn’t end up being merged directly.
  • In 2017, it became clear that this was likely to need some more attention, but before we could do anything else we had to revamp Launchpad’s build system to use pip rather than buildout, since without that we couldn’t upgrade to any newer versions of Twisted. That proved to be a substantial piece of yak-shaving: first we had to upgrade Launchpad off Ubuntu 12.04, and then the actual build system rewrite was a complicated project of its own.
  • In 2018, I fixed an authentication hang that happened if a client even tried to offer ECDSA or Ed25519 public keys to Launchpad, and we got ECDSA support fully working in Launchpad. We also discovered as a result of automated interoperability tests run as part of the Debian OpenSSH packaging that Twisted needed to gain support for the new openssh-key-v1 private key format, which became a prerequisite for Ed25519 support since OpenSSH only ever writes those keys in the new format, and so I fixed that.
  • In 2019, Python’s cryptography package gained support for X25519 (the Diffie-Hellman key exchange function based on Curve25519) and Ed25519, and it became somewhat practical to add support to Twisted on top of that. However, it required OpenSSL 1.1.1b, and it seemed unlikely that we would be in a position to upgrade all the relevant bits of Launchpad’s infrastructure to use that in the near term. I at least managed to add curve25519-sha256 key exchange support to Twisted based on some previous work by another contributor, and I prepared support for Ed25519 keys in Twisted even though I knew we weren’t going to be able to use it yet.
  • 2020 was … well, everyone knows what 2020 was like, plus we had a new baby. I did some experimentation in spare moments, but I didn’t really have the focus to be able to move this sort of complex problem forward.
  • In 2021, I bit the bullet and started seriously working on fallback mechanisms to allow us to use Ed25519 even on systems lacking a sufficient version of OpenSSL, though found myself blocked on figuring out type-checking issues following a code review. It then became clear on the release of OpenSSH 8.8 that we were going to have to deal with RSA SHA-2 signatures as well, since otherwise OpenSSH in Ubuntu soon wouldn’t be able to authenticate to Launchpad by default (which also caused me to delay uploading 8.8 to Debian unstable for a while). To deal with that, I first had to add SSH extension negotiation to Twisted.
  • Finally, in 2022, I added RSA SHA-2 signature support to Twisted, finally unblocked myself on the type-checking issue with the Ed25519 fallback mechanism, quickly put together a similar fallback mechanism for X25519, backported the whole mess to Twisted 20.3.0 since we currently can’t use anything newer due to the somewhat old version of Python 3 that we’re running, promptly ran into and fixed a regression that affected SFTP uploads to ppa.launchpad.net and upload.ubuntu.com, and finally added Ed25519 as a permissible key type in Launchpad’s authserver.

Phew! Thanks to everyone who works on Twisted, cryptography, and OpenSSL - it’s been really useful to be able to build on solid lower-level cryptographic primitives - and to those who helped with code review.