...
| Code Block |
|---|
./qemu-system-aarch64 \
-machine type=virt,virtualization=on,pflash0=rom,pflash1=efivars \
-cpu max,pauth-impdef=on \
-smp 8 \
-accel tcg \
-device virtio-net-pci,netdev=unet \
-device virtio-scsi-pci \
-device scsi-hd,drive=hd \
-netdev user,id=unet,hostfwd=tcp::2222-:22 \
-blockdev driver=raw,node-name=hd,file.driver=host_device,file.filename=/dev/zen-ssd2/bookworm-arm64,discard=unmap \
-serial mon:stdio \
-blockdev node-name=rom,driver=file,filename=(pwd)/pc-bios/edk2-aarch64-code.fd,read-only=true \
-blockdev node-name=efivars,driver=file,filename=$HOME/images/qemu-arm64-efivars \
-m 8192 \
-object memory-backend-memfd,id=mem,size=8G,share=on \
-device virtio-gpu-pci \
-device qemu-xhci -device usb-kbd -device usb-tablet \
-display gtk,gl=on |
Booting a custom kernel
One of the main reasons to have a test system is to test kernels. This is done by applying the -kernel and -initrd options. If no UEFI rom is supplied then you will boot direct, otherwise the UEFI firmware will pick up the kernel via the fw_cfg interface and pass it on instead of loading grub
| Code Block |
|---|
./qemu-system-aarch64 \
-machine type=virt,virtualization=on,pflash0=rom,pflash1=efivars \
-cpu max,pauth-impdef=on \
-smp 8 \
-accel tcg \
-device virtio-net-pci,netdev=unet \
-device virtio-scsi-pci \
-device scsi-hd,drive=hd \
-netdev user,id=unet,hostfwd=tcp::2222-:22 \
-blockdev driver=raw,node-name=hd,file.driver=host_device,file.filename=/dev/zen-ssd2/bookworm-arm64,discard=unmap \
-serial mon:stdio \
-blockdev node-name=rom,driver=file,filename=(pwd)/pc-bios/edk2-aarch64-code.fd,read-only=true \
-blockdev node-name=efivars,driver=file,filename=$HOME/images/qemu-arm64-efivars \
-m 8192 \
-object memory-backend-memfd,id=mem,size=8G,share=on \
-display none \
-kernel /home/alex/lsrc/linux.git/builds/arm64/arch/arm64/boot/Image \
-append "root=/dev/sda2" |
Installing Xen
The best way is to build the current Xen you are interested on your host with a crossbuild and then:
| Code Block |
|---|
make debball CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64 |
and you can copy the resulting image to your emulated system and install it. Make sure you haven’t installed any of the disto bits. You may need to run update-grub manually to add the Xen entries.
While you probably don’t want to touch most of the tooling stuff you may well want to build your own QEMU so you can have the latest Xen enabling bits. We shall strip down the config as building inside QEMU will be slower than native:
| Code Block |
|---|
git clone https://gitlab.com/qemu-project/qemu.git qemu.git
cd qemu.git
mkdir -p builds/xen
cd builds/xen
../../configure --disable-docs --disable-tools --disable-user --disable-tcg --disable-kvm
ninja |
And finally tweak /etc/default/xencommons to point at it
| Code Block |
|---|
# qemu path
QEMU_XEN=/root/lsrc/qemu.git/builds/xen/qemu-system-i386 |
And you you can systemctl restart xencommons.service or reboot and you should be able to list Xen domains:
| Code Block |
|---|
18:43:39 [root@debian-arm64:~/l/q/b/xen] + systemctl restart xencommons.service
18:43:45 [root@debian-arm64:~/l/q/b/xen] + systemctl status xencommons.service
● xencommons.service - LSB: Start/stop xenstored and xenconsoled
Loaded: loaded (/etc/init.d/xencommons; generated)
Active: active (running) since Mon 2023-12-18 18:43:45 GMT; 6s ago
Docs: man:systemd-sysv-generator(8)
Process: 15117 ExecStart=/etc/init.d/xencommons start (code=exited, status=0/SUCCESS)
Tasks: 9 (limit: 4659)
Memory: 33.0M
CPU: 501ms
CGroup: /system.slice/xencommons.service
├─ 1135 /usr/local/sbin/xenstored --pid-file /var/run/xen/xenstored.pid
├─ 1141 /usr/local/sbin/xenconsoled --pid-file=/var/run/xen/xenconsoled.pid
├─15140 /usr/local/sbin/xenconsoled --pid-file=/var/run/xen/xenconsoled.pid
└─15146 /root/lsrc/qemu.git/builds/xen/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize -monitor /dev/null -serial /dev/null>
Dec 18 18:43:44 debian-arm64 systemd[1]: Starting xencommons.service - LSB: Start/stop xenstored and xenconsoled...
Dec 18 18:43:45 debian-arm64 xencommons[15117]: Setting domain 0 name, domid and JSON config...
Dec 18 18:43:45 debian-arm64 xencommons[15137]: Dom0 is already set up
Dec 18 18:43:45 debian-arm64 xencommons[15117]: Starting xenconsoled...
Dec 18 18:43:45 debian-arm64 xencommons[15117]: Starting QEMU as disk backend for dom0
Dec 18 18:43:45 debian-arm64 systemd[1]: Started xencommons.service - LSB: Start/stop xenstored and xenconsoled.
18:44:02 [root@debian-arm64:~/l/q/b/xen] 1 + xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 4096 8 r----- 6753.8 |
Booting Xen Directly
Once you have the user space tooling installed you can now boot the hypervisor directly and manually load the dom0 kernel. Please note you’ll want to skip the UEFI bios for this, also we downgrade the CPU as Xen doesn’t support SVE+ out of the box.
| Code Block |
|---|
./qemu-system-aarch64 \
-machine type=virt,virtualization=on \
-cpu cortex-a57 \
-smp 8 \
-accel tcg \
-device virtio-net-pci,netdev=unet \
-device virtio-scsi-pci \
-device scsi-hd,drive=hd \
-netdev user,id=unet,hostfwd=tcp::2222-:22 \
-blockdev driver=raw,node-name=hd,file.driver=host_device,file.filename=/dev/zen-ssd2/bookworm-arm64,discard=unmap \
-serial mon:stdio \
-m 8192 \
-object memory-backend-memfd,id=mem,size=8G,share=on \
-display none \
-kernel $HOME/lsrc/xen/xen.git/xen/xen.efi \
-append "dom0_mem=4G,max:4G loglvl=all guest_loglvl=all" \
-device guest-loader,addr=0x49000000,kernel=$HOME/lsrc/linux.git/builds/arm64/arch/arm64/boot/Image,bootargs="console=hvc0 earlyprintk=xen root=/dev/sda2" |