Immutable Alpine Linux in bhyve 11 feb 2023 | Last updated: 11 feb 2023 16:18

Many things don't have proper installation docs any longer and are only provided as Docker (or podman) images. I set out to run Docker on FreeBSD with a minimal Linux using bhyve.

Immutable Alpine Linux in bhyve

Part 1 of a multip-part series where I hope to end up with some running Docker containers.

Setting up FreeBSD for bhyve

We'll use the sysutils/vm-bhyve port to simplify bhyve invocation.

Enable serial console

kldload nmdm
echo 'nmdm_load="YES"' >> /boot/loader.conf

Create storage for bhyve

zfs create -o mountpoint=/vm zroot/bhyve

Install required packages

pkg install bhyve-firmware vm-bhyve`

Enable bhyve-vm, set storage location an initialize

sysrc vm_enable YES
sysrc vm_dir='zfs:zroot/bhyve'

vm init
vm switch create public
vm switch add public em0

cp /usr/local/share/examples/vm-bhyve/* /vm/.templates

Create the VM, storage and configuration

Create the VM we'll use

vm create -t alpine Alpine

This will create zroot/bhyve/Alpine dataset

Create the persistent storage for the Alpine overlay.

cd /vm/Alpine
truncate -s 33555456 apkovl.img
mddev=$(mdconfig apkovl.img)
gpart create -s MBR /dev/${mddev}
gpart add -t linux-data ${mddev}
newfs_msdos -L apkovl /dev/${mddev}s1
mdconfig -d -u ${mddev}

The FAT32 partition will be /dev/vda1 in Alpine. Using partition type fat32 failed for me.

Modify your /vm/Alpine/Alpine.conf to something like

```:::conf loader="grub" graphics="no" cpu=4 memory=4096M network0_type="virtio-net" network0_switch="public" disk0_type="ahci-cd" disk0_name="alpine-virt-latest-x86_64.iso" disk1_type="virtio-blk" disk1_name="apkovl.img" grub_run0="set root=(hd0) grub_run1="linux /boot/vmlinuz-virt modules=loop,squashfs,sd-mod,usb-storage quiet console=tty0 console=ttyS0,115200" grub_run2="initrd /boot/initramfs-virt" uuid="random" network0_mac="random"

Note: The disk0 iso is mapped to the VM as hd0. disk1 will be /dev/vda in Alpine.

**NOTE**: We're not installing, we'll run Alpine from memory with storage on the FreeBSD host.

## Interlude: Grab latest image from Alpine

Python scriptlet to get latest URL from Alpine's download CDN, you'll need to have the `devel/py-yaml` port installed.

#!/usr/bin/env python3

import urllib.request
import yaml


with urllib.request.urlopen(f"{BASE_URL}/latest-releases.yaml") as response:
    data = yaml.safe_load(

iso = [flavor["iso"] for flavor in data if flavor["flavor"] == "alpine-virt"][0]


Download and link the latest iso

vm iso
(cd /vm/Alpine
ln -sf ../.iso/alpine-virt-3.17.2-x86_64.iso alpine-virt-latest-x86_64.iso

First boot

Start Alpine and enter using serial console

vm start Alpine
vm console alpine

This will show Connected, if you hit Enter you end up at the login prompt. User root has no password (yet). Mount the persistent storage and use the setup-alpine command to configure the system.

mkdir -p /media/vda1
echo "/dev/vda1 /media/vda1 vfat rw 0 0" >> /etc/fstab
mount -a

These are the answers I provided, removed some output for brevity. We're only creating the root user, no other users.

Enter system hostname (fully qualified form, e.g. '') [localhost] docker
Available interfaces are: eth0.
Enter '?' for help on bridges, bonding and vlans.
Which one do you want to initialize? (or '?' or 'done') [eth0] eth0
Ip address for eth0? (or 'dhcp', 'none', '?') [dhcp] dhcp
Do you want to do any manual network configuration? (y/n) [n] n
udhcpc: lease of obtained from, lease time 86400
Changing password for root
New password: <your password>
Retype password: <your password>
passwd: password for root changed by root
Which timezone are you in? ('?' for list) [UTC] UTC
HTTP/FTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] none
Which NTP client to run? ('busybox', 'openntpd', 'chrony' or 'none') [chrony] none
Enter mirror number (1-71) or URL to add (or r/f/e/done) [1]
Added mirror
Updating repository indexes... done.
Setup a user? (enter a lower-case loginname, or 'no') [no] no
Which ssh server? ('openssh', 'dropbear' or 'none') [openssh] openssh
Allow root ssh login? ('?' for help) [prohibit-password] prohibit-password
Enter ssh key or URL for root (or 'none') [none] none
No disks available. Try boot media /media/cdrom? (y/n) [n]
Enter where to store configs ('floppy', 'usb', 'vda1' or 'none') [vda1] vda1
Enter apk cache directory (or '?' or 'none') [/media/vda1/cache] /media/vda1/cache
WARNING: Ignoring No such file or directory

The sshd setup didn't do what it advertises it does, we'll get to that later. Now we persist our changes:

lbu commit

this generates a file /media/vda1/docker.apkovl.tar.gz. If you'd restart the VM now, you get the settings as stored in the apkovl tarball.

Powerdown the VM.


Not sure if this is how it should work, but my SSH session locks up for a couple seconds when I poweroff a VM. Type ~~. to exit the Serial Console.

Modify the Alpine overlay

Mount the Alpine overlay in your host

mkdir /mnt/apkovl
mddev=$(mdconfig /vm/Alpine/apkovl.img)
mount_msdosfs /dev/${mddev}s1 /mnt/apkovl

Unpack the transferred docker.apkovl.tar.gz in a working dir

cd ~
mkdir docker.apkovl
cd docker.apkovl
tar xf /mnt/apkovl/docker.apkovl.tar.gz

You end up with the overlay in docker.apkovl/etc. Anything you do in that will be in your Alpine host.

We'll be loging in as root with key authentication and fetch the key from the overlay in stead of from the home directory. No need for legacy crypto on FreeBSD.

HostKey /etc/ssh/ssh_host_ed25519_key

KexAlgorithms     curve25519-sha256,
Ciphers ,

PermitRootLogin prohibit-password

AuthorizedKeysFile      /etc/ssh/authorized_keys/%u .ssh/authorized_keys

PasswordAuthentication no

AllowTcpForwarding no
GatewayPorts no
X11Forwarding no

Subsystem       sftp    internal-sftp

Put the public key you'll use to access the Docker host in etc/ssh/authorized_keys/root

cp ~/.ssh/ etc/ssh/authorized_keys/root

Package the overlay again

tar zcf /mnt/apkovl/docker.apkovl.tar.gz --strip-components 1 .

Don't forget to tear down the mount

umount /mnt/apkovl
mdconfig -d -u ${mddev}

Second boot

We can now start the Alpine VM and connect to it with SSH and our key.

ssh -i ~/.ssh/id_docker docker

Welcome to Alpine!

The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <>.

You can setup the system with the command: setup-alpine

You may change this message by editing /etc/motd.