Another pretty short post today, still in Firecracker land.
decided to build my VMs with Docker instead of cloud-init
I’ve been trying to figure out how to build my VMs for a while. Previously my
plan was to just run
cloud-init when the VM started, because I already had
cloud-init.yaml files to launch VMs that I’d written previously.
It turned out that for some reason I couldn’t get
cloud-init to start, and
also it felt like
cloud-init was going to be really slow to run – even when
it was failing, it was already taking more than 10 seconds. I really wanted the
VM to boot in less than 2 seconds.
So I decided to instead build my containers with Docker, convert the Docker filesystem to an ext4 image, and then start that image in Firecracker. Here’s what creating the image looks like in a bash script.
IMG_ID=$(docker build -q .) CONTAINER_ID=$(docker run -td $IMG_ID /bin/bash) MOUNTDIR=mnt IMAGE=ubuntu.ext4 mount $IMAGE $MOUNTDIR qemu-img create -f raw $IMAGE 800M mkfs.ext4 $IMAGE docker cp $CONTAINER_ID:/ $MOUNTDIR
It seems to work fine, and actually building my VMs with Docker feels a lot
simpler than doing it with
cloud-init.yaml, I think they might be easier to
develop this way.
setting up Docker-Compose’s bridges
I kind of want to run my VM management software with
docker-compose, but it
needs to make changes to the host network to set up the bridges / tap
I learned that Docker Compose by default creates a new bridge for every
docker-compose file with a random name. I didn’t think this was going to
work for me, because I want to put my VMs on the same bridge as the
container that needs to SSH into the VMs. So I needed to know the name of the
It turns out the Docker Compose is AMAZING and lets you explicitly set the name of the bridge for a network
So I set up a network in my
docker-compose.yml file that looks like this, and
gotty container in the
version: "3.3" networks: firenet: driver: bridge ipam: driver: default config: - subnet: 18.104.22.168/16 driver_opts: com.docker.network.bridge.name: firecracker0
I still haven’t tested this out because there are some other legos I need to put together, but I think it should work.
trying out fly.io
I got a bit frustrated with writing my own Firecracker VM service so later in the day, so I asked on Twitter if anyone knew of a cloud service where I could run a Firecracker VM. I was pretty sure the answer was no, but I’m happy I asked because I was wrong!
It turns out that https://fly.io supports a Docker container interface, but they actually just unpack the Docker container’s files, convert it into an ext4 filesystem, and boot a Firecracker VM with it. They don’t support you picking the init system or choosing a kernel and presumably the VMs are kind of locked down, but it’s not a container!
I’d initially assumed it was a container because Fargate and Kata Containers both run containers images in VMs by putting a container runtime inside the Firecracker VM and running the container that way.
I got some VMs to run on fly.io using their Go API after a few hours, so we’ll
see how that goes! I’ll do a few more experiments today. It was pretty
straightforward except that their API isn’t documented yet. But their command
line management tool is open source so I could just read the source for
flyctl to figure out how it worked.