Deploying Swift
Sometimes I want to deploy Swift, the OpenStack object storage system.
Well, no, that’s not true. I basically never actually want to deploy Swift as such. What I generally want to do is to debug some bit of production service deployment machinery that relies on Swift for getting build artifacts into the right place, or maybe the parts of the Launchpad librarian (our blob storage service) that use Swift. I could find an existing private or public cloud that offers the right API and test with that, but sometimes I need to test with particular versions, and in any case I have a terribly slow internet connection and shuffling large build artifacts back and forward over the relevant bit of wet string makes it painfully slow to test things.
For a while I’ve had an Ubuntu 12.04 VM lying around with an Icehouse-based Swift deployment that I put together by hand. It works, but I didn’t keep good notes and have no real idea how to reproduce it, not that I really want to keep limping along with manually-constructed VMs for this kind of thing anyway; and I don’t want to be dependent on obsolete releases forever. For the sorts of things I’m doing I need to make sure that authentication works broadly the same way as it does in a real production deployment, so I want to have Keystone too. At the same time, I definitely don’t want to do anything close to a full OpenStack deployment of my own: it’s much too big a sledgehammer for this particular nut, and I don’t really have the hardware for it.
Here’s my solution to this, which is compact enough that I can run it on my laptop, and while it isn’t completely automatic it’s close enough that I can spin it up for a test and discard it when I’m finished (so I haven’t worried very much about producing something that runs efficiently). It relies on Juju and LXD. I’ve only tested it on Ubuntu 18.04, using Queens; for anything else you’re on your own. In general, I probably can’t help you if you run into trouble with the directions here: this is provided “as is”, without warranty of any kind, and all that kind of thing.
First, install Juju and LXD if necessary, following the instructions
provided by those projects, and also install the python-openstackclient
package as you’ll need it later. You’ll want to set Juju up to use
LXD, and you should probably
make sure that the shells you’re working in don’t have http_proxy set as
it’s quite likely to confuse things unless you’ve arranged for your proxy to
be able to cope with your local LXD containers. Then add a
model:
juju add-model swift
At this point there’s a bit of complexity that you normally don’t have to worry about with Juju. The swift-storage charm wants to mount something to use for storage, which with the LXD provider in practice ends up being some kind of loopback mount. Unfortunately, being able to perform loopback mounts exposes too much kernel attack surface, so LXD doesn’t allow unprivileged containers to do it. (Ideally the swift-storage charm would just let you use directory storage instead.) To make the containers we’re about to create privileged enough for this to work, run:
lxc profile set juju-swift security.privileged true
lxc profile device add juju-swift loop-control unix-char \
major=10 minor=237 path=/dev/loop-control
for i in $(seq 0 255); do
lxc profile device add juju-swift loop$i unix-block \
major=7 minor=$i path=/dev/loop$i
done
Now we can start deploying things! Save this to a file, e.g.
swift.bundle:
series: bionic
description: "Swift in a box"
applications:
mysql:
charm: "cs:mysql-62"
channel: candidate
num_units: 1
options:
dataset-size: 512M
keystone:
charm: "cs:keystone"
num_units: 1
swift-storage:
charm: "cs:swift-storage"
num_units: 1
to: [keystone]
options:
block-device: "/etc/swift/storage.img|5G"
swift-proxy:
charm: "cs:swift-proxy"
num_units: 1
to: [mysql]
options:
zone-assignment: auto
replicas: 1
relations:
- ["keystone:shared-db", "mysql:shared-db"]
- ["swift-proxy:swift-storage", "swift-storage:swift-storage"]
- ["swift-proxy:identity-service", "keystone:identity-service"]
And run:
juju deploy ./swift.bundle
This will take a while. You can run juju status to see how it’s going in
general terms, or juju debug-log for detailed logs from the individual
containers as they’re putting themselves together. When it’s all done, it
should look something like this:
Model Controller Cloud/Region Version SLA
swift lxd localhost 2.3.1 unsupported
App Version Status Scale Charm Store Rev OS Notes
keystone 13.0.1 active 1 keystone jujucharms 290 ubuntu
mysql 5.7.24 active 1 mysql jujucharms 62 ubuntu
swift-proxy 2.17.0 active 1 swift-proxy jujucharms 75 ubuntu
swift-storage 2.17.0 active 1 swift-storage jujucharms 250 ubuntu
Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 0 10.36.63.133 5000/tcp Unit is ready
mysql/0* active idle 1 10.36.63.44 3306/tcp Ready
swift-proxy/0* active idle 1 10.36.63.44 8080/tcp Unit is ready
swift-storage/0* active idle 0 10.36.63.133 Unit is ready
Machine State DNS Inst id Series AZ Message
0 started 10.36.63.133 juju-d3e703-0 bionic Running
1 started 10.36.63.44 juju-d3e703-1 bionic Running
At this point you have what should be a working installation, but with only administrative privileges set up. Normally you want to create at least one normal user. To do this, start by creating a configuration file granting administrator privileges (this one comes verbatim from the openstack-base bundle, though with one change that isn’t yet in the charm store version at the time of writing):
_OS_PARAMS=$(env | awk 'BEGIN {FS="="} /^OS_/ {print $1;}' | paste -sd ' ')
for param in $_OS_PARAMS; do
if [ "$param" = "OS_AUTH_PROTOCOL" ]; then continue; fi
if [ "$param" = "OS_CACERT" ]; then continue; fi
unset $param
done
unset _OS_PARAMS
_keystone_unit=$(juju status keystone --format yaml | \
awk '/units:$/ {getline; gsub(/:$/, ""); print $1; exit}')
_keystone_ip=$(juju run --unit ${_keystone_unit} 'unit-get private-address')
_password=$(juju run --unit ${_keystone_unit} 'leader-get admin_passwd')
export OS_AUTH_URL=${OS_AUTH_PROTOCOL:-http}://${_keystone_ip}:5000/v3
export OS_USERNAME=admin
export OS_PASSWORD=${_password}
export OS_USER_DOMAIN_NAME=admin_domain
export OS_PROJECT_DOMAIN_NAME=admin_domain
export OS_PROJECT_NAME=admin
export OS_REGION_NAME=RegionOne
export OS_IDENTITY_API_VERSION=3
# Swift needs this:
export OS_AUTH_VERSION=3
# Gnocchi needs this
export OS_AUTH_TYPE=password
Source this into a shell: for instance, if you saved this to
~/.swiftrc.juju-admin, then run:
. ~/.swiftrc.juju-admin
You should now be able to run openstack endpoint list and see a table for
the various services exposed by your deployment. Then you can create a
dummy project and a user with enough privileges to use Swift:
USERNAME=your-username
PASSWORD=your-password
openstack domain create SwiftDomain
openstack project create --domain SwiftDomain --description Swift \
SwiftProject
openstack user create --domain SwiftDomain --project-domain SwiftDomain \
--project SwiftProject --password "$PASSWORD" "$USERNAME"
openstack role add --project SwiftProject --user-domain SwiftDomain \
--user "$USERNAME" Member
(This is intended for testing rather than for doing anything particularly
sensitive. If you cared about keeping the password secret then you’d use
the --password-prompt option to openstack user create instead of
supplying the password on the command line.)
Now create a configuration file granting privileges for the user you just created. I felt like automating this to at least some degree:
touch ~/.swiftrc.juju
chmod 600 ~/.swiftrc.juju
sed '/^_password=/d;
s/\( OS_PROJECT_DOMAIN_NAME=\).*/\1SwiftDomain/;
s/\( OS_PROJECT_NAME=\).*/\1SwiftProject/;
s/\( OS_USER_DOMAIN_NAME=\).*/\1SwiftDomain/;
s/\( OS_USERNAME=\).*/\1'"$USERNAME"'/;
s/\( OS_PASSWORD=\).*/\1'"$PASSWORD"'/' \
<~/.swiftrc.juju-admin >~/.swiftrc.juju
Source this into a shell. For example:
. ~/.swiftrc.juju
You should now find that swift list works. Success! Now you can swift
upload files, or just start testing whatever it was that you were actually
trying to test in the first place.
This is not a setup I expect to leave running for a long time, so to tear it down again:
juju destroy-model swift
This will probably get stuck trying to remove the swift-storage unit,
since nothing deals with detaching the loop device. If that happens, find
the relevant device in losetup -a from another window and use losetup -d
to detach it; juju destroy-model should then be able to proceed.
Credit to the Juju and LXD teams and to the maintainers of the various charms used here, as well as of course to the OpenStack folks: their work made it very much easier to put this together.
2019-01-18: Edited to deploy to two containers rather than four, and to
incorporate a ~/.swiftrc.juju-admin change to cope with that.