When tried to install offical Xen package apt-get install xen-hypervisor-4.16-arm64, it failed to boot Xen hypervisor on AVA board. This article records the details for how to build GRUB and replace Xen hypervisor with TRS binary.

Build GRUB binary

I cross built GRUB binary for Arm64 target on my x86_64 machine.

Step 1: we need to download git source code:

$ git clone https://git.savannah.gnu.org/git/grub.git
$ cd grub

Step 2: execute bootstrap for downloading dependent package:

$ ./bootstrap

Step 3: configure the package:

$ ./configure --build=x86_64 --host=x86_64 --target=aarch64 --with-platform=efi \
  --disable-werror TARGET_CC=aarch64-linux-gnu-gcc \
  TARGET_OBJCOPY=aarch64-linux-gnu-objcopy TARGET_NM=aarch64-linux-gnu-nm \
  TARGET_RANLIB=aarch64-linux-gnu-ranlib TARGET_STRIP=aarch64-linux-gnu-strip \
  --prefix=$PWD --exec-prefix=$PWD

Step 4: prepare the grub-initial.cfg file.
Firstly, you need to know the UUID for the Ubuntu root file system:

root@ava:~# blkid 
/dev/nvme0n1p3: UUID="3e651572-426e-4280-86b6-752ce01bbe2f" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="f55eceef-7657-4af7-8e99-6301238e7ae0"
/dev/nvme0n1p1: UUID="E5E9-73BC" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="808434d6-d314-4ffe-92bd-f30037f1f832"
/dev/nvme0n1p4: UUID="5836e2e5-bdde-4355-bcab-d08d74a8f6ee" TYPE="swap" PARTUUID="a2d5fa9f-49a8-40af-9558-415fbefaff7d"
/dev/nvme0n1p2: UUID="7b2d7994-68ad-4a0b-bdca-db30c6f377ce" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="0baf2b82-0eb3-4782-b8a2-e8ae487040da"
/dev/nvme1n1p2: LABEL="rootfs" UUID="6091b3a4-ce08-3020-93a6-f755a22ef03b" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="rootfs" PARTUUID="f3374295-b635-44af-90b6-3f65ded2e2e4"
/dev/nvme1n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="bootfs" LABEL="bootfs" UUID="7819-74F8" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="ESP" PARTUUID="00112233-1234-1111-2222-000123456789"

From above log we can know the Ubuntu’s root file system UUID is 7b2d7994-68ad-4a0b-bdca-db30c6f377ce. Then we can prepare grub-initial.cfg file as below. The main purpose for this file is to initialize root for GRUB and then it loads Ubunut’s grug config file $prefix/grub.cfg.

# First partition on first disk, most likely EFI system partition. Set it here
# as fallback in case the search doesn't find the given UUID.
search.fs_uuid 7b2d7994-68ad-4a0b-bdca-db30c6f377ce root
set prefix=($root)'/boot/grub'
configfile $prefix/grub.cfg

Step 5: build GRUB efi.
I used below command to build GRUB efi file, which can be loaded by UEFI.

#!/usr/bin/bash

set -eu

# GRUB doesn't allow loading new modules from disk when secure boot is in
# effect, therefore pre-load the required modules.
MODULES=
MODULES="$MODULES part_gpt fat ext2"           # partition and file systems for EFI
MODULES="$MODULES configfile"                  # source command
MODULES="$MODULES password_pbkdf2"             # hashed password
MODULES="$MODULES echo normal linux"           # boot linux
MODULES="$MODULES all_video"                   # video output
MODULES="$MODULES search search_fs_uuid"       # search --fs-uuid
MODULES="$MODULES reboot sleep"                # sleep, reboot

TMP_GRUB_CFG='grub-initial.cfg'
TMP_GRUB_EFI='grubaa64.efi'

rm -f $TMP_GRUB_EFI

make clean
make && make install

./bin/grub-mkstandalone \
        --disable-shim-lock \
        --format=arm64-efi \
        --modules="$MODULES" \
        --locale-directory=//usr/share/locale/ \
        --directory=/home/leoy/Dev1/arm_blueprint/grub/lib/grub/arm64-efi \
        --output="$TMP_GRUB_EFI" \
        "boot/grub/grub.cfg=$TMP_GRUB_CFG"

Step 6: Install GRUB and the associated modules.

Copy GRUB files from x86_64 host to AVA board:

$ cd /path/to/grub/
$ scp grubaa64.efi root@ubuntu_ip:/tmp/
$ scp -r lib/grub/arm64-efi/ root@ubuntu_ip:/tmp/

The grub EFI is stored in the boot partition, the folder is efi/EFI/ubuntu. Use the built Grub EFI to replace the old one (and backup the old file):

$ mount /dev/nvme0n1p1 /mnt/
$ cp /mnt/EFI/ubuntu/grubaa64.efi /mnt/EFI/ubuntu/grubaa64.efi.bak # Backup original file
$ cp /tmp/grubaa64.efi /mnt/EFI/ubuntu/

Use below commands to copy Grub modules into :

$ mv /boot/grub/arm64-efi /boot/grub/arm64-efi.bak
$ mv /tmp/arm64-efi /boot/grub

Update Xen hypvervisor binary

The build Xen hypervisor has been uploaded to xen.efi, you can download and copy it into the Ubuntu’s folder /boot.

Add GRUB menu entry

Add a new GRUB entry for booting Xen hypervisor:

menuentry 'Ubuntu GNU/Linux, with Xen hypervisor' --class ubuntu --class gnu-linux --class gnu --class os --class xen $menuentry_id_option 'xen-gnulinux-simple-7b2d7994-68ad-4a0b-bdca-db30c6f377ce' {
        insmod part_gpt
        insmod ext2
        search --no-floppy --fs-uuid --set=root 7b2d7994-68ad-4a0b-bdca-db30c6f377ce
        echo    'Loading Xen xen.efi ...'
        if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then
            xen_rm_opts=
        else
            xen_rm_opts="noreboot dom0_mem=8192M bootscrub=0 iommu=on loglvl=all guest_loglvl=all"
        fi
        xen_hypervisor  /boot/xen.efi placeholder   ${xen_rm_opts}
        echo    'Loading Linux 6.7.0-rc8-00119-g1f874787ed9a ...'
        xen_module      /boot/vmlinuz-6.7.0-rc8-00119-g1f874787ed9a placeholder root=UUID=7b2d7994-68ad-4a0b-bdca-db30c6f377ce ro  ignore_loglevel sysrq_always_enabled nr_cpus=32 mitigations=off nmi_watchdog=0 nowatchdog nosoftlockup
        echo    'Loading initial ramdisk ...'
        xen_module      --nounzip   /boot/initrd.img-6.7.0-rc8-00119-g1f874787ed9a
}

Here three thing you need to change on your working AVA board:

Firstly, you need to replace the root file system UUID 7b2d7994-68ad-4a0b-bdca-db30c6f377c.
Then, you need to update the Linux kernel image file name and initrd file name respectively.

Build QEMU

We need to build QEMU, this is because Xen toolkit is dependent on QEMU (qemu-system-aarch64).

Step 1: install dependent packages on Ubuntu / Debian:

sudo apt-get install git-email
sudo apt-get install libaio-dev libbluetooth-dev libcapstone-dev libbrlapi-dev libbz2-dev
sudo apt-get install libcap-ng-dev libcurl4-gnutls-dev libgtk-3-dev
sudo apt-get install libibverbs-dev libjpeg-dev libncurses5-dev libnuma-dev
sudo apt-get install librbd-dev librdmacm-dev
sudo apt-get install libsasl2-dev libsdl2-dev libseccomp-dev libsnappy-dev libssh-dev
sudo apt-get install libvde-dev libvdeplug-dev libvte-2.91-dev libxen-dev liblzo2-dev
sudo apt-get install valgrind xfslibs-dev

Step 2: Download QEMU source code. Now we maintain the QEMU repo in: linaro / Blueprints / Automotive / xen-aosp / Qemu · GitLab . Akashi-san maintains his QEMU repo for enablig virtual GPU, but the building for QEMU is dependent on other libs, e.g. virglrenderer, mesa libs. I think the better way is to land the changes in a central place. So this document use the previous QEMU repo for building:

 $ mkdir /home/leo.yan/local  # Create a folder to install QEMU binaries
 
 $ https://gitlab.com/Linaro/blueprints/automotive/xen-aosp/qemu
 $ cd qemu
 $ mkdir build
 $ cd qemu
 $ ../configure --prefix=/home/leo.yan/local --target-list=aarch64-softmmu,i386-softmmu   --enable-kvm   --enable-xen   --disable-werror   --enable-slirp   --enable-opengl   --enable-virglrenderer   --enable-gtk   --enable-sdl
 $ make
 $ make install

As result, you can see QEMU libs and binaries are installed in the folder /home/leo.yan/local/:

leo.yan@ampere-bullseye:~$ ls /home/leo.yan/local/
bin  include  libexec  share  var

Build Xen toolkit

Step 1: Download Xen repo:

$ git clone https://gitlab.com/Linaro/blueprints/automotive/xen-aosp/xen

Step 2: Configure:

$ ./configure --prefix=/usr --libdir=/usr/lib64 --disable-docs \
  --disable-golang --disable-ocamltools --enable-ioreq-server \
  --with-system-qemu=/home/leo.yan/local/bin/qemu-system-aarch64

Step 3: Build Xen. In this step, it’s suggested to build Debian package:

$ make debball

After the building, the Debian package and built files are located in dist folder:

leo.yan@ampere-bullseye:~/xen$ cd dist/
leo.yan@ampere-bullseye:~/xen/dist$ ls
COPYING  README  install  install.sh  xen-upstream-4.18.0.deb

Step 4: Install Xen in system.

There have two ways to install Xen, one is to install Debian package:

$ dpkg -i xen-upstream-4.18.0.deb

Or, you can use the script install.sh for the installation:

$ sudo ./install.sh

As the result, you can see Xen toolkit binary xl is installed in the folder /usr/local/sbin/xl and the service script is placed in /etc/init.d/xencommon.

Setting up after system booting up

On the AVA board in the Cambridge lab, the grub has an entry DEBUG: boot with Xen hypervisor. You need to select this entry to boot Xen hypervisor. Then wait for Xen dom0 booting, you can use the serial to setup dhcp for the networking port:

$ sudo dhclient enP3p2s0f0

Afterwards, you can connect the board with ssh.

You also need to initialize the Xen script with the command:

$ sudo /etc/init.d/xencommons start
Setting domain 0 name, domid and JSON config...
Dom0 is already set up
Starting xenconsoled...
Starting QEMU as disk backend for dom0
qemu-system-aarch64: unsupported machine type
Use -machine help to list supported machines

Issue: here it reports qemu-system-aarch64: unsupported machine type, now we can ignore this error as I don’t see issue for Xen dom0 and domU. Please see below testing result.

Testing

The command sudo xl list can be used to do a smoke test:

leo.yan@ampere-bullseye:~/xen/dist$ sudo xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0 32768    80     r-----      45.0

A Xen configuration file, a kernel image and ramdisk image are placed in the folder /home/leo.yan/test_xen_vm :

leo.yan@ampere-bullseye:~$ cd test_xen_vm/
leo.yan@ampere-bullseye:~/test_xen_vm$ ls
Image.gz  startvm.cfg  xen_guest_image.cpio.gz

You can launch the Xen VM with commands:

sudo xl -vvv create startvm.cfg -c

As a result, the xl list command can show the new created virtual machine:

$ sudo xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0 32768    80     r-----      47.1
guest                                        1   512     2     -b----       0.6