When tried to install offical Xen package apt-get install xen-hypervisor-4.16-arm64, it failed to boot Xen hypervisor on AVA board. This article records the details for how to build GRUB and replace Xen hypervisor with TRS binary.

Build GRUB binary

I cross built GRUB binary for Arm64 target on my x86_64 machine.

Step 1: we need to download git source code:

Step 2: execute bootstrap for downloading dependent package:

Step 3: configure the package:

Step 4: prepare the grub-initial.cfg file.
Firstly, you need to know the UUID for the Ubuntu root file system:

From above log we can know the Ubuntu’s root file system UUID is 7b2d7994-68ad-4a0b-bdca-db30c6f377ce. Then we can prepare grub-initial.cfg file as below. The main purpose for this file is to initialize root for GRUB and then it loads Ubunut’s grug config file $prefix/grub.cfg.

Step 5: build GRUB efi.
I used below command to build GRUB efi file, which can be loaded by UEFI.

Step 6: Install GRUB and the associated modules.

Copy GRUB files from x86_64 host to AVA board:

The grub EFI is stored in the boot partition, the folder is efi/EFI/ubuntu. Use the built Grub EFI to replace the old one (and backup the old file):

Use below commands to copy Grub modules into :

Update Xen hypvervisor binary

The build Xen hypervisor has been uploaded to xen.efi, you can download and copy it into the Ubuntu’s folder /boot.

Add GRUB menu entry

Add a new GRUB entry for booting Xen hypervisor:

Here three thing you need to change on your working AVA board:

Firstly, you need to replace the root file system UUID 7b2d7994-68ad-4a0b-bdca-db30c6f377c.
Then, you need to update the Linux kernel image file name and initrd file name respectively.

Build QEMU

We need to build QEMU, this is because Xen toolkit is dependent on QEMU (qemu-system-aarch64).

Step 1: install dependent packages on Ubuntu / Debian:

Step 2: Download QEMU source code. Now we maintain the QEMU repo in: https://gitlab.com/Linaro/blueprints/automotive/xen-aosp/qemu . Akashi-san maintains his QEMU repo for enablig virtual GPU, but the building for QEMU is dependent on other libs, e.g. virglrenderer, mesa libs. I think the better way is to land the changes in a central place. So this document use the previous QEMU repo for building:

As result, you can see QEMU libs and binaries are installed in the folder /home/leo.yan/local/:

Build Xen toolkit

Step 1: Download Xen repo:

Step 2: Configure:

Step 3: Build Xen. In this step, it’s suggested to build Debian package:

After the building, the Debian package and built files are located in dist folder:

Step 4: Install Xen in system.

There have two ways to install Xen, one is to install Debian package:

Or, you can use the script install.sh for the installation:

As the result, you can see Xen toolkit binary xl is installed in the folder /usr/local/sbin/xl and the service script is placed in /etc/init.d/xencommon.

Setting up after system booting up

On the AVA board in the Cambridge lab, the grub has an entry DEBUG: boot with Xen hypervisor. You need to select this entry to boot Xen hypervisor. Then wait for Xen dom0 booting, you can use the serial to setup dhcp for the networking port:

Afterwards, you can connect the board with ssh.

You also need to initialize the Xen script with the command:

Issue: here it reports qemu-system-aarch64: unsupported machine type, now we can ignore this error as I don’t see issue for Xen dom0 and domU. Please see below testing result.

Testing

The command sudo xl list can be used to do a smoke test:

A Xen configuration file, a kernel image and ramdisk image are placed in the folder /home/leo.yan/test_xen_vm :

You can launch the Xen VM with commands:

As a result, the xl list command can show the new created virtual machine: