We are using Qemu Emulator here to serve as Kata Container host, and then launch the Kata containers with RME/CCA support inside it.

Host: Qemu Emulated, machine model: max, root filesystem is ubuntu 24.04.

Guest: VMM is Qemu with CCA/RME support.

Our Target

Launch Kata/CoCo now with CCA support and do the remote attestation inside the containers.

Current Limitation:

Use nerdctl to launch the container manually, the k0s support with Kata-deploy is still WIP.
The guest-image pull with Nydus are not enabled, WIP.
No image encryption / decryption and sign support on in the CoCo boot process.

Prepare Kata container Host and Qemu

Kata containers request a bunch of kernel configurations in order to serve as a Host. The kata-dev reporsitory contains some scripts to build Host kernel, host firmware and Host filesystem(currently support ubuntu 24.04 and 22.04).

Host Kernel

git clone https://github.com/kevinzs2048/kata-dev
cd kata-dev
make kata_host_kernel

Host Filesystem

cd kata-dev
OS_IMAGE_SIZE=20 IMAGE_NAME=ubuntu_24.img make ubuntu_image

It will generate an 20GB image named ubuntu_24.img with ubuntu 24.04 installed in kata-dev directory. This image also autamatically configured an network with static IP 192.168.122.22, assume that your bridge network for libvirt is 192.168.122.1. If you install the libvirtd and virt-manager, it will automatically installed and get the bridge.

The login credential to this Ubuntu image is set to linaro/linaro.

Install essential kernel modules

cd kata-dev
make install_kernel_modules

This command need sudo.

Build Host Firmware

make host_firmware

This process will build tf-rmm, EDK2 and TF-A, then generated flash.bin command under the kata-dev directory.

Build the Qemu command to launch CCA Host

make qemu_cca_host

The Qemu version: https://git.codelinaro.org/linaro/dcap/qemu -b cca/2025-05-28

Launch the CCA Host

make run_host

You can also run this command locally:

$QEMU_WORKDIR/build/qemu-system-aarch64 \
-M virt,virtualization=on,secure=on,gic-version=3 \
-M acpi=off -cpu max,x-rme=on,sme=off,pauth-impdef=on \
-m 3G -smp 4 \
-nographic \
-bios ../flash.bin \
-kernel $KERNEL_WORKDIR/arch/arm64/boot/Image \
-drive format=raw,if=none,file=../$IMAGE_NAME,id=hd0 \
-device virtio-blk-pci,drive=hd0 \
-append root=/dev/vda \
-nodefaults \
-serial tcp:localhost:54320 \
-serial tcp:localhost:54321 \
-chardev socket,mux=on,id=hvc0,port=54322,host=localhost \
-device virtio-serial-device \
-device virtconsole,chardev=hvc0 \
-chardev socket,mux=on,id=hvc1,port=54323,host=localhost \
-device virtio-serial-device \
-device virtconsole,chardev=hvc1 \
-append "root=/dev/vda rw earlycon console=hvc0 nokaslr" \
-net nic,macaddr=52:54:30:12:34:63 \
-net tap,ifname=tap1,script=no,downscript=no \
-device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
-fsdev local,security_model=none,path=../../,id=shr0

The command above is taking reference from here: https://git.codelinaro.org/linaro/dcap/op-tee-4.2.0/build/-/blob/cca/v8/qemu_v8_cca.mk?ref_type=heads#L623

Connect to the Host via SSH

Assume that your environment already has a bridge to the NIC:

4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:51:7b:99 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

Add tap1 to virbr0 and enable it.

sudo brctl addif virbr0 tap1
sudo ifconfig tap1 0.0.0.0 promisc up

Connect to the Kata host:

ssh linaro@192.168.122.22

Install and Configure the Kata container

Build the Kata containers and CoCo components using Kata-deploy

cd kata-dev
make build_kata

After this components, you will see the generated file as below:

# In the directory: kata-containers/tools/packaging/kata-deploy/local-build/build
kata-static-kernel-cca-confidential.tar.xz
kata-static-qemu-cca-confidential.tar.xz
kata-static-rootfs-cca-confidential-image.tar.xz
kata-static-rootfs-cca-confidential-initrd.tar.xz
kata-static-shim-v2.tar.xz

If you don’t want to build the Kata releated components yourself, you can simply get a pre build files here:

https://people.linaro.org/~kevin.zhao/kata-cca/

Install Kata/CoCo from tarball

TODO: Use kata-deploy and K0s

Currently manually: On the build machine, copy the binaries stored at: kata-containers/tools/packaging/kata-deploy/local-build/build to Kata Host Machines which mentioned at the above part.

rm -rf /opt/kata
cd binaries/

# only install below packages
file_list=("kata-static-kernel-cca-confidential.tar.xz"
           "kata-static-qemu-cca-experimental.tar.xz"
           "kata-static-rootfs-cca-confidential-image.tar.xz"
           "kata-static-rootfs-cca-confidential-initrd.tar.xz"
           "kata-static-shim-v2.tar.xz")

for file in "${file_list[@]}" ; do
    if [ -f "$file" ]; then
        sudo tar -xJf "$file" -C /
    fi
done

After installation, the rootfs/initrd/Guest kernel are stored at /opt/kata/share, and containerd-shim-kata-v2/Kata-runtime/Kata-monitor are located at /opt/kata/bin

Configure and install the Kata-containers

Kata-containers configuration

Get the Kata container configuration from: /opt/kata/share/defaults/kata-containers/configuration-qemu-cca.toml, copy it to /etc/kata-containers/configuration.toml

Some essential items should be modified at /etc/kata-containers/configuration.toml. Kata containers can run with either an initrd image or a rootfs image. But they can just be set either rootfs or initrd, please do not set both. Here we test rootfs. Some essential configurations items should be changed as below.

[hypervisor.qemu]
path = "/opt/kata/bin/qemu-system-aarch64-cca-experimental"

shared_fs = "virtio-9p"

# Enlarge below in order to boot inside the emulator
dial_timeout = 5000
create_container_timeout = 1200

# Add the below parameters to kernel params
# agent.aa_kbc_params=cc_kbc::http://213.146.155.117:8080 agent.guest_components_rest_api=all
kernel_params = "cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 agent.aa_kbc_params=cc_kbc::http://213.146.155.117:8080 agent.guest_components_rest_api=all"

The Qemu path current is not setting properly for Qemu CCA, will fix soon.
Note: shared_fs here is currently used virtio-9p, sharing the container image from host to guest. WIP for set up guest-pull function and image-encryption and nydus snapshotter to enable the the full image function.
Timeout: The two timeout parameters are needed as the low performance at emulator spaces.
confidential_guest is the key variable to indicated if we launch the confidential VM or common vm, this is by default set to true in this configuration.

Configure the Containerd to use Kata-containers

Configure the Containerd to use Kata-container. Copy https://people.linaro.org/~kevin.zhao/config.toml.containerd to /etc/containerd/config.toml

sudo apt install containerd -y;
sudo systemctl daemon-reload
sudo systemctl restart containerd

More things for Kata with containerd, please refer here: https://github.com/kata-containers/kata-containers/blob/main/docs/install/container-manager/containerd/containerd-install.md

Install CNI and nerdctl to launch the container

CNI - Container network interface

CNI is essential for container network. CNI can be installed either binaries or build from source, check :

More things for Kata with containerd, please refer here: https://github.com/kata-containers/kata-containers/blob/main/docs/install/container-manager/containerd/containerd-install.md

nerdctl command line

Install nerdctl command line. nerdctl cmdline will automatically set up the CNI network with host-local, and it is compatible with docker command line. Just copy the binaries to the host and it is ready to go. See more for Nerdctl

Run command like below to launch Kata container:

nerdctl run --runtime io.containerd.kata.v2 -it docker.io/library/busybox:latest sh

nerdctl is the docker-compatible OCI cmd line, it can also automatically configuration the CNI network.

Configure and Launch the Trustee verification service

Please refer this document to deploy Trustee verification service, and using the Veraison verification as the 3rd party verifier: https://github.com/confidential-containers/trustee/tree/main/deps/verifier/src/cca

After deployment, we get the kbs endpoint:

http://213.146.155.117:8080

Launch Kata containers and running the Remote attestation

Launch the Kata-containers with nerdctl.

sudo nerdctl run --runtime io.containerd.kata.v2 -it curlimages/curl:latest sh

After about 1 mins, the Kata container is ready and we are now in the sh cmdline. This command includes the curl command so that we can test get_token inside the container.

Manually Get Token

in the container sh do get_evidence cmd

# in the container sh
curl http://127.0.0.1:8006/aa/token\?token_type\=kbs

This command above is the HTTP access to api-server-rest service which is the coco-guest-component binaries packaged inside the Kata container sandbox rootfs image. When Kata is started, it will automatically launched the api-server-rest for services inside container to use, and attestation-agent to get the real attestation token, and also the confidential-data-hub serves for the connection between the attester and key broker service running inside the CoCo Trustee.

Please also check the log for Trustee, you can get the logs.

More about the remote attestation, please also check the comment at https://linaro.atlassian.net/jira/software/c/projects/DCAP/issues/DCAP-94

Build and run CoCo/Kata Container with RME Support