Content Comparison

Mathieu The whole software stack for CCA is in development, meaning instructions will change frequently and repositories are temporary.

Table of Contents

With the OP-TEE build environment

This method requires at least the following tools and libraries. The manual build described below also requires most of them.

...

Read on for the details of the software stack, or skip to the following section to boot a Realm guest.

Manual build

The following sections detail how to build and run all components of the CCA software stack. Two QEMU binaries are built. The system emulation QEMU implements a complete machine, emulating Armv9 CPUs with FEAT_RME and four security states: Root, Secure, Non-secure and Realm. The VMM (Virtual Machine Manager) QEMU is cross-built by buildroot, and launches the realm guest from Non-secure EL0.

Code Block

       |    REALM     |  NON-SECURE  |
-------+--------------+--------------+
  EL0  | Guest Rootfs |  Host Rootfs |
       |              |  QEMU VMM    |
-------+--------------+--------------+
  EL1  |        EDK2  |              |
       | Linux Guest  |              |
       |              |  EDK2        |
-------+--------------+  Linux Host  |
  EL2  |      TF-RMM  |    (KVM)     |
       |              |              |
-------+--------------+--------------+
 (ROOT)|                             |
  EL3  |            TF-A             |
-------+-----------------------------+
  HW   |            QEMU             |
-------+-----------------------------+

TF-RMM

The Realm Management Monitor (RMM) connects KVM and the Realm guest.

...

Code Block
git submodule update --init --recursive export CROSS_COMPILE=aarch64-none-elf- cmake -DCMAKE_BUILD_TYPE=Debug -DRMM_CONFIG=qemu_virt_defcfg -B build-qemu cmake --build build-qemu

Host EDK2

Edk2 is the firmware used in non-secure world. It works out of the box. However, we rely on edk2 not allocating memory from the DRAM area reserved for the RMM at the moment, which is fragile. Future work will add support for the reserved memory node provided by TF-A in the device-tree.

...

Code Block
git submodule update --init --recursive source edksetup.sh make -j -C BaseTools export GCC5_AARCH64_PREFIX=aarch64-linux-gnu- build -b RELEASE -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc

TF-A

TF-A loads the RMM as well as the Non-secure firmware, and bridges RMM and KVM. It also owns the Granule Protection Table (GPT).

...

Code Block

# Embed the RMM image and edk2 into the Final Image Package (FIP)
make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu ENABLE_RME=1 DEBUG=1 LOG_LEVEL=40 \
    QEMU_USE_GIC_DRIVER=QEMU_GICV3 RMM=../rmm/build-qemu/Debug/rmm.img \
    BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/RELEASE_GCC5/FV/QEMU_EFI.fd all fip
# Pack whole image into flash.bin
dd if=build/qemu/debug/bl1.bin of=flash.bin
dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096

Host and guest Linux

Both host and guest need extra patches.

...

Code Block

make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 defconfig
# Enable the configfs-tsm driver that provides the attestation interface
scripts/config -e VIRT_DRIVERS -e ARM_CCA_GUEST
make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 -j8

Guest edk2

The QEMU VMM can either launch the guest kernel itself, or launch edk2 which launches the kernel or an intermediate bootloader. That latter method is generally used to boot a Linux distribution. Edk2 needs modifications in order to run as a Realm guest.

...

Note that the DEBUG build is very verbose (even with a few patches that remove repetitive messages), which is extremely slow in a nesting environment with emulated UART. Change it to -b RELEASE to speed up the guest boot.

QEMU VMM

Both kvmtool and QEMU can be used to launch Realm guests. For details about kvmtool, see the cover letter for the Linux support above.

...

Code Block
# Although it is buildroot that builds the VMM from this source directory, # the following is needed to first download all the submodules ./configure --target-list=aarch64-softmmu

Root filesystem

Buildroot provides a convenient way to build lightweight root filesystems. It can also embed the VMM into the rootfs if you specify the path to kvmtool or QEMU source in a local.mk file in the build directory.

...

This creates the rootfs images in buildroot’s output/images/ when building in-tree, or images/ when building out of tree.

Guest disk image for edk2

To create a guest disk image that resembles more a Linux distribution, containing the grub2 bootloader and the kernel, have a look at buildroot’s configs/aarch64_efi_defconfig, which enables a few options to generate a disk with an EFI partition:

...

With these, after generating the root filesystem, buildroot packs it into another disk image images/disk.imgalong with an EFI FAT partition that contains grub and the kernel Image (layout is defined by board/aarch64-efi/genimage-efi.cfg).

QEMU system emulation

Repo: https://gitlab.com/qemu-project/qemu.git or the same repository as the VMM.

...

Code Block
mkdir -p ../build/qemu/ # outside of the source directory cd ../build/qemu/ ../../qemu/configure --target-list=aarch64-softmmu make -j

Running the system emulation

QEMU will connect to four TCP ports for the different consoles. Create the servers manually with socat -,rawer TCP-LISTEN:5432x (x = 0, 1, 2, 3) or use the script given at the end.

...

Note: The base system (started above) is currently set to 8GB of RAM, which we believe is enough memory to demonstrate how CCA works in a simulated environment. Modifications to the trusted firmware and RMM elements are needed if a different value is selected.

Launching a Realm guest

Once at the host command line prompt simply use root to log in.

...

Code Block
mount -t 9p shr0 /mnt

Launching a Realm guest using QEMU

The following script uses the QEMU VMM to launch a Realm guest with KVM.

...

Followed a few minutes later by the guest kernel starting in the Realm terminal.

Launching a Realm guest using the KVMTool

Code Block
lkvm run --realm -c 2 -m 2G -k /mnt/out/bin/Image -d /mnt/out-br/images/rootfs.ext4 --restricted_mem -p "console=hvc0 root=/dev/vda" < /dev/hvc1 > /dev/hvc1

Running edk2 as a guest

Enable USE_EDK2 to boot the Realm guest with the edk2 firmware. It can either load kernel and initrd through the FwCfg device provided by QEMU (DIRECT_KERNEL_BOOT=true), or launch a bootloader from a disk image (see grub2 above). When booting the kernel directly, edk2 measures the kernel, initrd and parameters provided on the QEMU command-line and adds them to the Realm Extended Measurement via RSI calls, so that they can be attested later.

...

Disable USE_VIRTCONSOLE in order to see all boot logs. Doing this enables the emulated PL011 serial and is much slower. Although edk2 does support virtio-console, it doesn’t display the debug output there (but you’ll still see RMM logs showing progress during boot).
When booting via grub2, the kernel parameters are stored in grub.cfg which is copied from board/aarch64-efi/grub.cfg by the buildroot script board/aarch64-efi/post-image.sh. Bu default the kernel parameters do not define a console, so Linux will determine the boot console from the device tree’s /chosen/stdout-path property, which QEMU initializes to the default serial console. So if you want to boot with virtconsole, add console=hvc0 to board/aarch64-efi/grub.cfg before making buildroot.

Tips

Automate some things in the host boot

You can add files to the buildroot images by providing an overlay directory. The BR2_ROOTFS_OVERLAY option points to the directory that will be added into the image. For example I use:

...

Code Block

# /etc/inittab
#
# Copyright (C) 2001 Erik Andersen <andersen@codepoet.org>
#
# Note: BusyBox init doesn't support runlevels.  The runlevels field is
# completely ignored by BusyBox init. If you want runlevels, use
# sysvinit.
#
# Format for each entry: <id>:<runlevels>:<action>:<process>
#
# id        == tty to run on, or empty for /dev/console
# runlevels == ignored
# action    == one of sysinit, respawn, askfirst, wait, and once
# process   == program to run

# Startup the system
::sysinit:/bin/mount -t proc proc /proc
::sysinit:/bin/mount -o remount,rw /
::sysinit:/bin/mkdir -p /dev/pts /dev/shm
::sysinit:/bin/mount -a
::sysinit:/bin/mount -t debugfs debugfs /sys/kernel/debug
::sysinit:/sbin/swapon -a
null::sysinit:/bin/ln -sf /proc/self/fd /dev/fd
null::sysinit:/bin/ln -sf /proc/self/fd/0 /dev/stdin
null::sysinit:/bin/ln -sf /proc/self/fd/1 /dev/stdout
null::sysinit:/bin/ln -sf /proc/self/fd/2 /dev/stderr
::sysinit:/bin/hostname -F /etc/hostname
# now run any rc scripts
::sysinit:/etc/init.d/rcS

# Put a getty on the serial port
#console::respawn:/sbin/getty -L  console 0 vt100 # GENERIC_SERIAL

::respawn:-/bin/sh

# Stuff to do for the 3-finger salute
#::ctrlaltdel:/sbin/reboot

# Stuff to do before rebooting
::shutdown:/etc/init.d/rcK
::shutdown:/sbin/swapoff -a
::shutdown:/bin/umount -a -r

Spawn four consoles with servers listening for QEMU system emulation:

This uses OP-TEE’s soc_term.py

...

Version	Old Version 2	New Version 3
Changes made by	Mathieu Poirier	Mathieu Poirier
Saved on	May 06, 2024	May 06, 2024

Versions Compared

Key

With the OP-TEE build environment

Manual build

TF-RMM

Host EDK2

TF-A

Host and guest Linux

Guest edk2

QEMU VMM

Root filesystem

Guest disk image for edk2

QEMU system emulation

Running the system emulation

Launching a Realm guest

Launching a Realm guest using QEMU

Launching a Realm guest using the KVMTool

Running edk2 as a guest

Tips

Automate some things in the host boot

Spawn four consoles with servers listening for QEMU system emulation: