Building an RME stack for QEMU

The whole software stack for CCA is in development, meaning instructions will change frequently and repositories are temporary. Instructions to compile the stack, both manually and from the OP-TEE build environment, have been written from a Ubuntu 22.04 LTS based system.

With the OP-TEE build environment

This method requires at least the following tools and libraries. The manual build described below also requires most of them.

repo
python3-pyelftools, python3-venv
acpica-tools
openssl (debian libssl-dev)
libglib2.0-dev, libpixman-1-dev
dtc (debian device-tree-compiler)
flex, bison
make, cmake, ninja (debian ninja-build), curl, rsync

The easiest way to build and run a complete stack is through OP-TEE. We support two system emulation QEMU machines, i.e Virt and SBSA. The amount of system RAM supported by QEMU-virt is set to at most 8GB and can not be increased. QEMU-sbsa also uses 8GB by default but can be configured between 2GB and 1TB.

The following commands will download all components and build them, in about thirty minutes on a fast machine. To avoid dealing with dependencies you can also build the latest stack in a container environment and run it in a tmux session with this set of scripts.

Virt machine:

mkdir cca-v4
cd cca-v4
repo init -u https://git.codelinaro.org/linaro/dcap/op-tee-4.2.0/manifest.git -b cca/v4 -m qemu_v8_cca.xml
repo sync -j8 --no-clone-bundle
cd build
make -j8 toolchains
make -j8

SBSA machine:

mkdir cca-v4
cd cca-v4
repo init -u https://git.codelinaro.org/linaro/dcap/op-tee-4.2.0/manifest.git -b cca/v4 -m sbsa_cca.xml
repo sync -j8 --no-clone-bundle
cd build
make -j8 toolchains
make -j8

Note:

If the build fails, try without -j. It will point out missing dependencies.
Add CLOUDHV=y to build cloud-hypervisor. This requires a rust toolchain >= 1.77.

Images can be found under cca-v4/out/ and cca-v4/out-br/. The following command launches system emulation QEMU with the RME feature enabled, running TF-A, RMM and the Linux host.

make run-only

This should launch 4 new terminals, i.e Firmware, Host, Secure and Realm. Output from the boot process will start flowing in the Firmware terminal followed by the Host terminal. The build environment automatically makes the cca-v4 directory available to the host VM via 9p.

Read on for the details of the software stack, or skip to the following section to boot a Realm guest.

Manual build

The following sections detail how to build and run all components of the CCA software stack. Two QEMU binaries are built. The system emulation QEMU implements a complete machine, emulating Armv9 CPUs with FEAT_RME and four security states: Root, Secure, Non-secure and Realm. The VMM (Virtual Machine Manager) QEMU is cross-built by buildroot, and launches the realm guest from Non-secure EL0.

Instructions to build the TF-RMM, TF-A and host EDK2 differ based on the QEMU machine selected for system emulation. All other components of the stack are common to both machines.

Manual build instructions for TF-RMM, TF-A and host EDK2 on QEMU-virt

Manual build instructions for TF-RMM, TF-A and host EDK2 on QEMU-sbsa

Host and guest Linux

Both host and guest need extra patches.

Status: Guest support is in Linux v6.13. Host support on the list

Repo: https://gitlab.arm.com/linux-arm/linux-cca cca-full/v5+v7

Build:

Guest edk2

The QEMU VMM can either launch the guest kernel itself, or launch edk2 which launches the kernel or an intermediate bootloader. That latter method is generally used to boot a Linux distribution. Edk2 needs modifications in order to run as a Realm guest.

Status: in development. Only the ArmVirtQemu firwmare supports booting in a Realm at the moment, not ArmVirtQemuKernel. Only disk boot works in the latest version, not direct kernel.

Repo: https://git.codelinaro.org/linaro/dcap/edk2 branch cca/v3

Build:

Note that the RELEASE build is a lot faster than the DEBUG build, but doesn’t provide a lot of information. If the boot doesn’t work, try building with -b DEBUG.

QEMU and kvmtool VMM

Both kvmtool and QEMU can be used to launch Realm guests.

Status: in development

Repo: https://git.codelinaro.org/linaro/dcap/qemu branch cca/v3
https://git.codelinaro.org/linaro/dcap/kvmtool branch cca/log

Build:

The buildroot recipe below already includes kvmtool and QEMU for CCA, so there is no need to download them separately. For development you can use your own source directory: add a local.mk file at the root of the buildroot build directory, containing:

You will need to run make qemu-rebuild in the buildroot build directory after making changes to the QEMU source.

Cloud-hypervisor

Status: in development. Not yet updated to latest version.

Repo: for now https://git.codelinaro.org/linaro/dcap/cloud-hypervisor branch cca/v3

Build:

Then copy target/aarch64-unknown-linux-gnu/debug/cloud-hypervisor into the Root filesystem or the shared folder.

Root filesystem

Buildroot provides a convenient way to build lightweight root filesystems.

Repo:

buildroot.org / buildroot · GitLab
Use the master branch to have up-to-date recipes for building QEMU.
https://git.codelinaro.org/linaro/dcap/buildroot-external-cca.git contains additional packages used for demonstrating attestation and useful scripts. It also provides qemu-cca and kvmtool-cca packages that replace buildroot’s qemu and kvmtool, in order to use the latest work-in-progress VMM support for CCA.

Build:

This creates the rootfs images in buildroot’s output/images/ when building in-tree, or images/ when building out of tree.

Guest disk image for edk2

To create a guest disk image that resembles more a Linux distribution, containing the grub2 bootloader and the kernel, have a look at buildroot’s configs/aarch64_efi_defconfig, which enables a few options to generate a disk with an EFI partition:

With these, after generating the root filesystem, buildroot packs it into another disk image images/disk.imgalong with an EFI FAT partition that contains grub and the kernel Image (layout is defined by board/aarch64-efi/genimage-efi.cfg).

Build the Ubuntu Rootfs

Below there is a scripts to automatically build the ubuntu 22.04 rootfs, it consists of several parts:

Generate a 4G image, make partitions and mount.
Download the ubuntu-base filesystem and abstract the files to the image
Configure the essential files for ubuntu for installing packages.
Set up the init process for Ubuntu Rootfs boot.
Install essential packages via Chroot, in case of failing enabling the /dev/hvc0 when realm boot.
Umount

NOTE: Please copy this content below to a ubuntu_fs.sh and run it with sudo, as the mount, chroot needs the root permissions.

The above command will generate a ubuntu 22 rootfs with the name ubuntu22.img. It is very easy to launch, just change the disk image to ubuntu22.img and you can enjoy. It can be either running as the Realm Host, or the Realm itself for daily development. Just tweak the scripts below to suit your user cases.

QEMU system emulation

Repo: QEMU / QEMU · GitLab or the same repository as the VMM.

Build: do not build in the same source directory as the VMM! Since buildroot copies the whole content of that source directory, binary files will conflict (the VMM is cross-built while the system emulation QEMU is native).

If you want to use the same source directory as the target QEMU, QEMU integrated in buildroot, do use a separate build directory as described here:

Running the system emulation

QEMU will connect to four TCP ports for the different consoles. Create the servers manually with socat -,rawer TCP-LISTEN:5432x (x = 0, 1, 2, 3) or use the script given at the end in section “Spawn four consoles with servers listening for QEMU system emulation”:

QEMU-virt startup script:

QEMU-sbsa startup script:

Crucially, the x-rme=on parameter enables the (experimental) FEAT_RME.

In the host kernel log, verify that KVM communicates with the RMM and is ready to launch Realm guests:

Note: The virt platform currently has at most 8GB of RAM, which we believe is enough memory to demonstrate how CCA works in a simulated environment. Modifications to the trusted firmware and RMM elements are needed if a different value is selected.

Launching a Realm guest

Once at the host command line prompt simply use root to log in.

Using buildroot-external-cca, the shared directory should be automatically mounted. Otherwise mount it with with:

Launching a Realm guest using QEMU

When using buildroot-external-cca, the filesystem should contain a script provided by cca-realm-measurements. Launch the VM with:

Using a tap network puts the guest on the same network as the host, and allows running the attestation demo below.

Or manually (with user networking rather than tap):

The -M confidential-guest-support=rme0 and -object rme-guest,id=rme0 parameters declare this as a Realm VM.

You should see RMM logs in the Firmware terminal:

Followed a few minutes later by the guest kernel starting in the Realm terminal.

Launching a Realm guest using Kvmtool

or manually:

Launching a Realm guest using cloud-hypervisor

This example uses a macvtap interface to connect the guest to the host network. CONFIG_MACVTAP needs to be 'y' in the host kernel config.

or manually:

Running edk2 as a guest

Enable USE_EDK2 to boot the Realm guest with the edk2 firmware. It can either load kernel and initrd through the FwCfg device provided by QEMU, or launch a bootloader from a disk image (gen-run-vmm.sh --edk2 --boot-disk). When booting the kernel directly, edk2 measures the kernel, initrd and parameters provided on the QEMU command-line and adds them to the Realm Extended Measurement via RSI calls, so that they can be attested later.

Notes:

When booting via grub2, the kernel parameters are stored in grub.cfg which is copied from board/aarch64-efi/grub.cfg by the buildroot script board/aarch64-efi/post-image.sh. By default the kernel parameters do not define a console, so Linux will determine the boot console from the device tree’s /chosen/stdout-path property, which QEMU initializes to the default serial console. So if you want to boot with virtconsole, add console=hvc0 to board/aarch64-efi/grub.cfg before making buildroot.

Attestation Proof of Concept

Two demonstration applications are available in the root file system.

cca-workload-attestation

From a Realm VM, you can query the RMM for a CCA attestation token that is printed to the console and saved to a file:

The tool can also demonstrate a typical interaction with an attestation service by communicating the CCA attestation token to an instance of the Veraison services:

Details on the cca-workload-attestation, the Veraison services and the endorser that populate the endorsement values can be found here.

keybroker-demo

The keybroker demo shows interaction between an attester, a relying party and a verifier. The keybroker-app, running in the Realm, requests a secret key from keybroker-server, run by the relying party.

Repo: GitHub - veraison/keybroker-demo: A simple key broker protocol

Build: the keybroker server

Run:

The server, on the build machine, listens on port 8088, and connects to the Linaro veraison instance for platform token verification. You can change these settings with command-line options.

The client, in the Realm, requests secret key “skywalker”:

You will normally get the following error:

This means that the platform token verification succeeded, but the realm token verification did not. The Realm Initial Measurement is not known by the server:

Retrying the key request after restarting the server with the suggested parameter should now succeed:

realm-measurements

Instead of running the server twice, you can also predict the Realm Initial Measurement, even before running the guest, using the cca-realm-measurements tool.

Repo: GitHub - veraison/cca-realm-measurements: Tool to compute Realm initial and extended measurements for Arm CCA

Build:

Run:

Since this is the same script that you use to launch the VM, it already knows the QEMU command-line arguments that you use, and the generated DTB loaded into the VM. Since you also give it the kernel and initrd images that will be loaded into the Realm, it will correctly predict the resulting Realm Initial Measurement (RIM). You can of course run the realm-measurements tool manually if you use a different VMM command-line.

Pass the resulting RIM to keybroker-server:

When running your own veraison instance (see the documentation and poc-endorser), you can also provision the verifier with reference values directly, so cca-workload-attestation passport can obtain an affirming appraisal of the realm token:

And in the guest:

Tips

Automate some things in the host boot

You can add files to the buildroot images by providing an overlay directory. The BR2_ROOTFS_OVERLAY option points to the directory that will be added into the image. For example I use:

S50-shr is an initscript that mounts the shared directory:

inittab is buildroot’s package/busybox/inittab, modified to automatically log into root (the respawn line). It could also mount the 9p filesystem.

Spawn four consoles with servers listening for QEMU system emulation:

This uses OP-TEE’s soc_term.py