This is the first post in a new series regarding my new virtualization setup using the Xen Hypervisor. This post describes the setup of an Alpine Linux Dom0 domain.

Alpine will be installed on an LVM pool that can later be used to store VM disks. Another good option would be the Data Disk Mode, but that loads all modifications from the base ISO in a tmpfs at boot, which increases the memory consumption of dom0.

Preparation

• Download the Xen installer ISO alpine-xen-X.XX.X-x86_64.iso from the Alpine download site (replace X.XX.X with the current version).
• Write the ISO to a flash drive using dd if=alpine-xen-X.XX.X-x86_64.iso of=/dev/sdx (Ventoy doesn't work, the installer doesn't find the installation medium).
• Boot the installer

Installation

Base Installation

The base installation can be done by the setup-alpine script. If dom0 should handle the network (the other option would be a network driver domain) at least one bridge network is required. For LVM the setup-disk script needs additional arguments that setup-alpine currently cannot provide, so all the disk setup is skipped by answering none to all disk related questions.

The following listing shows an example invocation of setup-alpine.

$ setup-alpine
Select keyboard layout: [none] de
Select variant (or 'abort'): de
Enter system hostname (fully qualified form, e.g. 'foo.example.org') [localhost] hv1
Which one do you want to initialize? (or '?' or 'done') [eth0] br0
Which port(s) do you want add to bridge br0? (or 'done') [eth0] eth0
Ip address for br0? (or 'dhcp', 'none', '?') [dhcp] dhcp
Do you want to do any manual network configuration? (y/n) [n] n
New password: ********
Retype password: ********
Which timezone are you in? [UTC] Europe/Berlin
HTTP/FTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] none
Which NTP client to run? ('busybox', 'openntpd', 'chrony' or 'none') [chrony] chrony
Enter mirror number or URL: [1] 1
Setup a user? (enter  lower-case loginname, or 'no') [no] admin
Full name for user admin [admin] admin
New password: ********
Retype password: ********
Enter ssh key or URL for paulz (or 'none') [none] https://example.com/my-ssh-key
Which ssh server? ('openssh', 'dropbear' or 'none') [openssh] openssh
Which disk(s) would you like to use? (or '?' for help or 'none' [none] none
Enter where to store configs ('floppy', 'usb' or 'none') [none] none
Enter apk cache directory (or '?' or 'none') [/media/usb/cache] none

Disk setup

The disk is configured using setup-disk with the -L flag (-L means LVM). The setup-disk script creates a volume that fills the entire LVM pool. dom0 doesn't need that much storage, so it is reduced with lvreduce.

$ BOOTLOADER=grub SWAP_SIZE=2048 setup-disk -L
Which disk(s) would you like to use? (or '?' for help or 'none') [sda] sda
How would you like to use it? ('sys', 'data', or '?' for help) [?] sys
WARNING: Erase the above disk(s) and continue? (y/n) [n] y

# install resize2fs
$ apk add e2fsprogs-extra
# setup-disks uses the remaining space on the LVM volume group.
$ lvreduce -L 8G --resizefs vg0/lv_root

Reboot.

GRUB modifications

Somehow GRUB doesn't detect Xen currently, so it needs to be added manually to /etc/grub.d/40_custom:

menuentry "Xen/Linux lts" $menuentry_id_option 'xen'{
        multiboot2      /boot/xen.gz dom0_mem=512M,max:512M
        module2         /boot/vmlinuz-lts root=/dev/mapper/vg0-lv_root modules=sd-mod,usb-storage,ext4,loop quiet rootfstype=ext4 max_loop=32
        module2         /boot/initramfs-lts
}

The GRUB default target needs to be set to Xen by setting GRUB_DEFAULT="xen" in /etc/default/grub
After that GRUB can be updated by calling

$ update-grub

Impermanence

First create a backup of /boot/initramfs-lts at /boot/initramfs-lts.bck. Impermanence is implemented by cloning the rootfs logical volume on boot. A new OpenRC service /etc/init.d/clone-rootfs is used on runlevel boot to clone the rootfs on boot.

#!/sbin/openrc-run

description="Create and switch root filesystem to LVM snapshot on boot"

depend() {
    before localmount
    need lvm
}

start() {
    ebegin "Creating and switching root filesystem to LVM snapshot"

    # Define the original and snapshot LVM volumes
    ORIGINAL_VOLUME="/dev/vg0/lv_root"
    SNAPSHOT_VOLUME="/dev/vg0/lv_root-current"
    SNAPSHOT_SIZE="10G"
    SNAPSHOT_MOUNT="/mnt/snapshot"

    # this script gets called again on pivot_root and we can exit early.
    if [ -d "/new_root" ]; then
        eend 0
        return 0
    fi

    # Ensure the original volume is active
    lvchange -ay $ORIGINAL_VOLUME

    # Create the snapshot if it doesn't exist
    if lvdisplay $SNAPSHOT_VOLUME > /dev/null 2>&1; then
        lvremove -f $SNAPSHOT_VOLUME
    fi
    lvcreate --size $SNAPSHOT_SIZE --snapshot --name $SNAPSHOT_VOLUME $ORIGINAL_VOLUME

    # Ensure the snapshot volume is active
    lvchange -ay $SNAPSHOT_VOLUME

    # Mount the snapshot
    mkdir -p $SNAPSHOT_MOUNT
    mount $SNAPSHOT_VOLUME $SNAPSHOT_MOUNT

    # Prepare for switch_root
    mkdir -p $SNAPSHOT_MOUNT/new_root
    mount --bind / $SNAPSHOT_MOUNT/new_root

    # Switch root
    pivot_root $SNAPSHOT_MOUNT $SNAPSHOT_MOUNT/new_root

    # Move necessary mount points
    mount --move /new_root/proc /proc
    mount --move /new_root/sys /sys
    mount --move /new_root/dev /dev

    # Unmount old root
    umount -l /new_root

    eend 0
}

dom0$ chmod +x /etc/init.d/clone-rootfs
dom0$ rc-default add clone-rootfs boot

Replace /dev/vg0/lv_root with /dev/vg0/lv_root-current in /etc/fstab. Add /dev/vg0/lv_root /media/lv_root ext4 noauto,rw 0 0 to /etc/fstab.

Reboot.

• Install all needed packages for setting up the disk:

  iso$ apk add wipefs cfdisk lvm2
  

• Load the dm-mod kernel module.

  iso$ modprobe dm-mod
  

• Remove the existing disk label

  iso$ wipefs --all /dev/sdx
  

• Initialize the disk with GPT and format the disk (called /dev/sdx in this document) (with cfdisk) to create the following partitions:

  • EFI System (~ 512M)
  • Linux LVM

• Initialize the partitions

  iso$ mkfs.vfat /dev/sdx1
  iso$ pvcreate /dev/sdx2
  iso$ vgcreate vg0 /dev/sdx2
  

• Create logical volumes

  iso$ lvcreate -n dom0_swap -L 1G vg0 
  iso$ lvcreate -n dom0_root -L 8G vg0
  

Immutability

Resources

• Alpine Linux Wiki

Installation

Diskless mode has the benefit of resetting the filesystem on every reboot to the last commited state. This allows the operator to maintain a organized and clean installation.

Note: The inputs and outputs of commands are not fully included in the following listings (only the relevant parts).

Warning: Replace /dev/sda with the drive you want to use.

1. Load Xen Drivers, Services and Kernel Modules with the setup-xen-dom0 script.

iso$ setup-xen-dom0

2. Complete the base installation using the setup-alpine script. The important parts are the network bridge br0 and none for the last three questions.

iso$ setup-alpine
Select keyboard layout: [none] de
Select variant (or 'abort'): de
Enter system hostname (fully qualified form, e.g. 'foo.example.org') [localhost] hv1
Which one do you want to initialize? (or '?' or 'done') [eth0] br0
Which port(s) do you want add to bridge br0? (or 'done') [eth0] eth0
Ip address for br0? (or 'dhcp', 'none', '?') [dhcp] dhcp
Do you want to do any manual network configuration? (y/n) [n] n
New password: ********
Retype password: ********
Which timezone are you in? [UTC] Europe/Berlin
HTTP/FTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] none
Which NTP client to run? ('busybox', 'openntpd', 'chrony' or 'none') [chrony] chrony
Enter mirror number or URL: [1] 1
Setup a user? (enter  lower-case loginname, or 'no') [no] admin
Full name for user paulz [admin] admin
New password: ********
Retype password: ********
Enter ssh key or URL for paulz (or 'none') [none] none
Which ssh server? ('openssh', 'dropbear' or 'none') [openssh] openssh
Which disk(s) would you like to use? (or '?' for help or 'none' [none] none
Enter where to store configs ('floppy', 'usb' or 'none') [none] none
Enter apk cache directory (or '?' or 'none') [/media/usb/cache] none

3. Install cfdisk and wipefs.

apk add cfdisk wipefs

4. Wipe and Partition using cfdisk.

iso$ wipefs --all /dev/sda
iso$ cfdisk /dev/sda
# Choose GPT label
# Create a `EFI System` partition (2 G, must be bigger than ISO size)
# Create a `Linux swap` partition (2 G)
# Create a `Linux filesystem` partition (12G)
# Create a `Linux LVM` partition

5. Setup Disk

iso$ mkfs.vfat /dev/sda1
iso$ setup-bootable -v /media/usb /dev/sda1

6. Reboot and remove installation medium

7. repeat steps 1 and 2

8. Setting up swap

dom0$ mkswap /dev/sda2
dom0$ printf "%s\tnone\tswap\tsw 0 0\n" /dev/sda2 >> /etc/fstab
dom0$ rc-update add swap
dom0$ rc-service swap start

9. Setting up data partition (taken from setup-disks but that script doesn't support pre-formatted disks)

dom0$ apk add e2fsprogs
dom0$ mkfs.ext4 /dev/sda3
dom0$ printf "%s\t/var\t\t%s\tdefaults 1 2\n" /dev/sda3 ext4 >> /etc/fstab
dom0$ printf "%s\t/media/sdc3\t\t%s\tnoauto,rw 0 0\n" /dev/sda3 ext4 >> /etc/fstab
dom0$ mv /var /.var
dom0$ mkdir /var
dom0$ mount /var
dom0$ mv /.var/* /var/
dom0$ rmdir /.var
dom0$ service syslog condrestart

10. Setup LVM

dom0$ apk add lvm2
dom0$ rc-service lvm start
dom0$ rc-update add lvm boot
dom0$ pvcreate /dev/sda2
dom0$ vgcreate vg0 /dev/sda2

11. Setup LBU

dom0$ mkdir /media/sda3
dom0$ mount /dev/sda3 /media/sda3
dom0$ setup-lbu
Enter where to store configs ('floppy', 'sda1', 'sda3', 'usb' or 'none') [sda3] sda3

12. Setup apkcache

dom0$ setup-akcache
Enter apk cache direcory (or '?' or 'none') [/media/sda3/cache] /var/cache/apk

13. Remove first line (something like /media/sda/apks) from /etc/apk/repositories

14. Install Xen Packages

apk add xen-hypervisor xen-qemu grub-xenhost bridge

16. Commit Config

dom0$ lbu commit

17. Reboot

Testing

Now we are going to test the installation by installing a alpine-virt guest domain.

1. Download the alpine-virt-X.XX.X-x86_64.iso to /var/isos/.

2. Create a guest disk

dom0$ lvcreate -n testdomain -L 1G vg0

3. Create a file testdomain.cfg with the following content:

name      = "testdomain"
vcpus     = "1"
memory    = "1024"
kernel    = "/usr/lib/grub-xen/grub-x86_64-xen.bin"
disk      = [ "/var/isos/alpine-virt-3.20.3-x86_64.iso,,hdc,cdrom", "/dev/vg0/testdomain,,sda" ]
vif       = [ "mac=FD:57:37:0B:F3:DB,bridge=br0" ]

4. Start domain and attach console

dom0$ xl create -c testdomain.cfg

5. setup testdomain

testdomain$ BOOTLOADER=grub setup-alpine
testdomain$ poweroff

6. remove iso from /etc/xen/domains/testdomain.cfg

7. repeat step 4

8. remove testvm

dom0$ testdomain.cfg
dom0$ lvremove testdomain vg0