When tried to install offical Xen package apt-get install xen-hypervisor-4.16-arm64, it failed to boot Xen hypervisor on AVA board. This article records the details for how to build GRUB and replace Xen hypervisor with TRS binary.
Build GRUB binary
I cross built GRUB binary for Arm64 target on my x86_64 machine.
Step 1: we need to download git source code:
$ git clone https://git.savannah.gnu.org/git/grub.git $ cd grub
Step 2: execute bootstrap for downloading dependent package:
$ ./bootstrap
Step 3: configure the package:
$ ./configure --build=x86_64 --host=x86_64 --target=aarch64 --with-platform=efi \ --disable-werror TARGET_CC=aarch64-linux-gnu-gcc \ TARGET_OBJCOPY=aarch64-linux-gnu-objcopy TARGET_NM=aarch64-linux-gnu-nm \ TARGET_RANLIB=aarch64-linux-gnu-ranlib TARGET_STRIP=aarch64-linux-gnu-strip \ --prefix=$PWD --exec-prefix=$PWD
Step 4: prepare the grub-initial.cfg file.
Firstly, you need to know the UUID for the Ubuntu root file system:
root@ava:~# blkid /dev/nvme0n1p3: UUID="3e651572-426e-4280-86b6-752ce01bbe2f" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="f55eceef-7657-4af7-8e99-6301238e7ae0" /dev/nvme0n1p1: UUID="E5E9-73BC" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="808434d6-d314-4ffe-92bd-f30037f1f832" /dev/nvme0n1p4: UUID="5836e2e5-bdde-4355-bcab-d08d74a8f6ee" TYPE="swap" PARTUUID="a2d5fa9f-49a8-40af-9558-415fbefaff7d" /dev/nvme0n1p2: UUID="7b2d7994-68ad-4a0b-bdca-db30c6f377ce" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="0baf2b82-0eb3-4782-b8a2-e8ae487040da" /dev/nvme1n1p2: LABEL="rootfs" UUID="6091b3a4-ce08-3020-93a6-f755a22ef03b" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="rootfs" PARTUUID="f3374295-b635-44af-90b6-3f65ded2e2e4" /dev/nvme1n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="bootfs" LABEL="bootfs" UUID="7819-74F8" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="ESP" PARTUUID="00112233-1234-1111-2222-000123456789"
From above log we can know the Ubuntu’s root file system UUID is 7b2d7994-68ad-4a0b-bdca-db30c6f377ce. Then we can prepare grub-initial.cfg file as below. The main purpose for this file is to initialize root for GRUB and then it loads Ubunut’s grug config file $prefix/grub.cfg.
# First partition on first disk, most likely EFI system partition. Set it here # as fallback in case the search doesn't find the given UUID. search.fs_uuid 7b2d7994-68ad-4a0b-bdca-db30c6f377ce root set prefix=($root)'/boot/grub' configfile $prefix/grub.cfg
Step 5: build GRUB efi.
I used below command to build GRUB efi file, which can be loaded by UEFI.
#!/usr/bin/bash
set -eu
# GRUB doesn't allow loading new modules from disk when secure boot is in
# effect, therefore pre-load the required modules.
MODULES=
MODULES="$MODULES part_gpt fat ext2" # partition and file systems for EFI
MODULES="$MODULES configfile" # source command
MODULES="$MODULES password_pbkdf2" # hashed password
MODULES="$MODULES echo normal linux" # boot linux
MODULES="$MODULES all_video" # video output
MODULES="$MODULES search search_fs_uuid" # search --fs-uuid
MODULES="$MODULES reboot sleep" # sleep, reboot
TMP_GRUB_CFG='grub-initial.cfg'
TMP_GRUB_EFI='grubaa64.efi'
rm -f $TMP_GRUB_EFI
make clean
make && make install
./bin/grub-mkstandalone \
--disable-shim-lock \
--format=arm64-efi \
--modules="$MODULES" \
--locale-directory=//usr/share/locale/ \
--directory=/home/leoy/Dev1/arm_blueprint/grub/lib/grub/arm64-efi \
--output="$TMP_GRUB_EFI" \
"boot/grub/grub.cfg=$TMP_GRUB_CFG"
Step 6: Install GRUB and the associated modules.
Copy GRUB files from x86_64 host to AVA board:
$ cd /path/to/grub/ $ scp grubaa64.efi root@ubuntu_ip:/tmp/ $ scp -r lib/grub/arm64-efi/ root@ubuntu_ip:/tmp/
The grub EFI is stored in the boot partition, the folder is efi/EFI/ubuntu. Use the built Grub EFI to replace the old one (and backup the old file):
$ mount /dev/nvme0n1p1 /mnt/ $ cp /mnt/EFI/ubuntu/grubaa64.efi /mnt/EFI/ubuntu/grubaa64.efi.bak # Backup original file $ cp /tmp/grubaa64.efi /mnt/EFI/ubuntu/
Use below commands to copy Grub modules into :
$ mv /boot/grub/arm64-efi /boot/grub/arm64-efi.bak $ mv /tmp/arm64-efi /boot/grub
Update Xen hypvervisor binary
The build Xen hypervisor has been uploaded to xen.efi, you can download and copy it into the Ubuntu’s folder /boot.
Add GRUB menu entry
Add a new GRUB entry for booting Xen hypervisor:
menuentry 'Ubuntu GNU/Linux, with Xen hypervisor' --class ubuntu --class gnu-linux --class gnu --class os --class xen $menuentry_id_option 'xen-gnulinux-simple-7b2d7994-68ad-4a0b-bdca-db30c6f377ce' {
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 7b2d7994-68ad-4a0b-bdca-db30c6f377ce
echo 'Loading Xen xen.efi ...'
if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then
xen_rm_opts=
else
xen_rm_opts="noreboot dom0_mem=8192M bootscrub=0 iommu=on loglvl=all guest_loglvl=all"
fi
xen_hypervisor /boot/xen.efi placeholder ${xen_rm_opts}
echo 'Loading Linux 6.7.0-rc8-00119-g1f874787ed9a ...'
xen_module /boot/vmlinuz-6.7.0-rc8-00119-g1f874787ed9a placeholder root=UUID=7b2d7994-68ad-4a0b-bdca-db30c6f377ce ro ignore_loglevel sysrq_always_enabled nr_cpus=32 mitigations=off nmi_watchdog=0 nowatchdog nosoftlockup
echo 'Loading initial ramdisk ...'
xen_module --nounzip /boot/initrd.img-6.7.0-rc8-00119-g1f874787ed9a
}
Here three thing you need to change on your working AVA board:
Firstly, you need to replace the root file system UUID 7b2d7994-68ad-4a0b-bdca-db30c6f377c.
Then, you need to update the Linux kernel image file name and initrd file name respectively.
Build QEMU
We need to build QEMU, this is because Xen toolkit is dependent on QEMU (qemu-system-aarch64).
Step 1: install dependent packages on Ubuntu / Debian:
sudo apt-get install git-email sudo apt-get install libaio-dev libbluetooth-dev libcapstone-dev libbrlapi-dev libbz2-dev sudo apt-get install libcap-ng-dev libcurl4-gnutls-dev libgtk-3-dev sudo apt-get install libibverbs-dev libjpeg-dev libncurses5-dev libnuma-dev sudo apt-get install librbd-dev librdmacm-dev sudo apt-get install libsasl2-dev libsdl2-dev libseccomp-dev libsnappy-dev libssh-dev sudo apt-get install libvde-dev libvdeplug-dev libvte-2.91-dev libxen-dev liblzo2-dev sudo apt-get install valgrind xfslibs-dev