We are using Qemu Emulator here to serve as Kata Container host, and then launch the Kata containers with RME/CCA support inside it.

Host: Qemu Emulated, machine model: max, root filesystem is ubuntu 24.04.

Guest: VMM is Qemu with CCA/RME support.

Our Target

Launch Kata/CoCo now with CCA support and do the remote attestation inside the containers.

Current Limitation:

Use nerdctl to launch the container manually, the k0s support with Kata-deploy is still WIP.
The guest-image pull function is now supported without any external snapshotter such as Nydus.
Boot the encrypted image, doing the remote attestation, get the decryption key and decrypted is supported for this working flow.

Prepare Kata container Host and Qemu

Kata containers request a bunch of kernel configurations in order to serve as a Host. The kata-dev repository contains some scripts to build Host kernel, host firmware and Host filesystem (currently support ubuntu 24.04 and 22.04).

Host Kernel

git clone https://github.com/kevinzs2048/kata-dev
cd kata-dev
make kata_host_kernel

Host Filesystem

cd kata-dev
OS_IMAGE_SIZE=20 IMAGE_NAME=ubuntu_24.img make ubuntu_image

It will generate an 20GB image named ubuntu_24.img with ubuntu 24.04 installed in kata-dev directory. This image also automatically configures a network with static IP 192.168.122.22, assumes that your bridge network for libvirt is 192.168.122.1. If you install the libvirtd and virt-manager, it will automatically setup the bridge.

The login credential to this Ubuntu image is set to linaro/linaro.

Install essential kernel modules

cd kata-dev
make install_kernel_modules

This command need sudo.

Build Host Firmware

make host_firmware

This process will build tf-rmm, EDK2 and TF-A, then generates flash.bin under the kata-dev directory.

Build the Qemu command to launch CCA Host

make qemu_cca_host

The Qemu version: https://git.codelinaro.org/linaro/dcap/qemu -b cca/2025-05-28

Launch the CCA Host

make run_host

You can also run this command locally:

$QEMU_WORKDIR/build/qemu-system-aarch64 \
-M virt,virtualization=on,secure=on,gic-version=3 \
-M acpi=off -cpu max,x-rme=on,sme=off,pauth-impdef=on \
-m 3G -smp 4 \
-nographic \
-bios ../flash.bin \
-kernel $KERNEL_WORKDIR/arch/arm64/boot/Image \
-drive format=raw,if=none,file=../$IMAGE_NAME,id=hd0 \
-device virtio-blk-pci,drive=hd0 \
-append root=/dev/vda \
-nodefaults \
-serial tcp:localhost:54320 \
-serial tcp:localhost:54321 \
-chardev socket,mux=on,id=hvc0,port=54322,host=localhost \
-device virtio-serial-device \
-device virtconsole,chardev=hvc0 \
-chardev socket,mux=on,id=hvc1,port=54323,host=localhost \
-device virtio-serial-device \
-device virtconsole,chardev=hvc1 \
-append "root=/dev/vda rw earlycon console=hvc0 nokaslr" \
-net nic,macaddr=52:54:30:12:34:63 \
-net tap,ifname=tap1,script=no,downscript=no \
-device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
-fsdev local,security_model=none,path=../../,id=shr0

The command above is taking reference from here: https://git.codelinaro.org/linaro/dcap/op-tee-4.2.0/build/-/blob/cca/v8/qemu_v8_cca.mk?ref_type=heads#L623

Connect to the Host via SSH

Assume that your environment already has a bridge to the NIC:

4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:51:7b:99 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

Add tap1 to virbr0 and enable it.

sudo brctl addif virbr0 tap1
sudo ifconfig tap1 0.0.0.0 promisc up

Connect to the Kata host:

ssh linaro@192.168.122.22

Mount the shared directry:

Once in the the host, you can mount the kata-dev directory shared with the build machine:

sudo mount -t 9p shr0 /mnt/

Install and Configure the Kata container

Build the Kata containers and CoCo components using Kata-deploy

cd kata-dev
make build_kata

After this components, you will see the generated file as below:

# In the directory: kata-containers/tools/packaging/kata-deploy/local-build/build
kata-static-kernel-cca-confidential.tar.xz
kata-static-qemu-cca-confidential.tar.xz
kata-static-rootfs-cca-confidential-image.tar.xz
kata-static-rootfs-cca-confidential-initrd.tar.xz
kata-static-shim-v2.tar.xz
kata-static-ovmf-cca.tar.xz

If you don’t want to build the Kata related components yourself, you can simply get pre build files here:

https://people.linaro.org/~kevin.zhao/kata-cca/

Install Kata/CoCo from tarball

TODO: Use kata-deploy and K0s

Currently manually: On the build machine, copy the binaries stored at: kata-containers/tools/packaging/kata-deploy/local-build/build to Kata Host Machines which mentioned at the above part.

rm -rf /opt/kata
cd binaries/

# only install below packages
file_list=("kata-static-kernel-cca-confidential.tar.xz"
           "kata-static-qemu-cca-experimental.tar.xz"
           "kata-static-rootfs-cca-confidential-image.tar.xz"
           "kata-static-rootfs-cca-confidential-initrd.tar.xz"
           "kata-static-shim-v2.tar.xz"
           "kata-static-ovmf-cca.tar.xz")

for file in "${file_list[@]}" ; do
    if [ -f "$file" ]; then
        sudo tar -xJf "$file" -C /
    fi
done

After installation, the rootfs/initrd/guest kernel are stored at /opt/kata/share, and containerd-shim-kata-v2/Kata-runtime/Kata-monitor are located at /opt/kata/bin

Configure Kata-containers and other container runtime configuration

Kata-containers configuration

Get the Kata container configuration from: /opt/kata/share/defaults/kata-containers/configuration-qemu-cca.toml, copy it to the path as below:

mkdir /etc/kata-containers/
cp /opt/kata/share/defaults/kata-containers/configuration-qemu-cca.toml /etc/kata-containers/configuration.toml

Modify some essential items in /etc/kata-containers/configuration.toml:

[hypervisor.qemu]
path = "/opt/kata/bin/qemu-system-aarch64-cca-experimental"
kernel = "/opt/kata/share/kata-containers/vmlinuz-confidential.container"
image = "/opt/kata/share/kata-containers/kata-containers-confidential.img"

Modify the timeout parameters for RME/CCA container

# Enlarge below in order to boot inside the emulator
dial_timeout = 500
create_container_timeout = 300

Add the remote attestation configurations to the kernel params:

# Add the below parameters to kernel params
# agent.aa_kbc_params=cc_kbc::http://213.146.155.117:8080 agent.guest_components_rest_api=all
kernel_params = "cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 agent.aa_kbc_params=cc_kbc::http://213.146.155.117:8080 agent.guest_components_rest_api=all"

You may need to load the vhost modules

sudo modprobe vhost_vsock
sudo modprobe vhost_net
echo -e "vhost-vsock\nvhost-net" | sudo tee /etc/modules-load.d/vhost.conf

Configure the Containerd to use Kata-containers

Configure the Containerd to use Kata-container. Copy https://people.linaro.org/~kevin.zhao/config.toml.containerd to /etc/containerd/config.toml. Modify it to use the Kata configuration created above:

         [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options]
            ConfigPath = "/etc/kata-containers/configuration.toml"

sudo apt install containerd -y
sudo systemctl daemon-reload
sudo systemctl restart containerd

More things for Kata with containerd, please refer here: https://github.com/kata-containers/kata-containers/blob/main/docs/install/container-manager/containerd/containerd-install.md

In particular you may need to make the binary accessible by containerd:

sudo ln -s /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin

Install CNI - Container network interface

CNI is essential for container network. CNI can be installed either binaries or build from source, check : https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/containerd-kata.md

Install nerdctl command line

Install nerdctl command line. nerdctl cmdline will automatically set up the CNI network with host-local, and it is compatible with docker command line. Just copy the binaries to the host and it is ready to go. See more for Nerdctl.

nerdctl is the docker-compatible OCI cmd line, it can also automatically configuration the CNI network.

Run command like below to launch Kata container:

nerdctl run --runtime io.containerd.kata.v2 -it docker.io/library/busybox:latest sh

Remote attestation: Configure and Launch the Trustee verification service

We need another node to running as the Trustee service node

Please refer this document to deploy Trustee verification service, and using the Veraison verification as the 3rd party verifier: https://github.com/confidential-containers/trustee/tree/main/deps/verifier/src/cca

After deployment, we get the kbs endpoint:

http://<Trustee Node IP>:8080

Use case 1: Launch Kata containers and execute the remote attestation

No need to set up nydus

Before start, modify the items in /etc/kata-containers/configuration.toml (The default configure for share_fs is none, which means we use guest-pull features and don’t use the file share). In this case, we just use virtio-9p to share the container image from host to guest. There is no confidential guest image protection in this case, we will try the remote attestation after the container start. NOTE: `virtio-fs` is currently not supported with RME support.

shared_fs = "virtio-9p"

Restart containerd service:

sudo systemctl restart containerd

Launch the Kata-containers with nerdctl.

sudo nerdctl run --runtime io.containerd.kata.v2 -it curlimages/curl:latest sh

After about 1 mins, the Kata container is ready and we are now in the sh cmdline. This command includes the curl command so that we can test get_token inside the container.

Manually Get Attestation Token

In the container sh do get_evidence cmd:

# in the container sh
curl http://127.0.0.1:8006/aa/token\?token_type\=kbs

This command above is the HTTP access to api-server-rest service which is the coco-guest-component binaries packaged inside the Kata container sandbox rootfs image. This http access will final call attestation-agent to get the evidence via linux-coco TSM api, and then send to the KBS service endpoint to get the remote attestation token.

Please also check the log for Trustee, you can get the logs.

Use Case 2: Launch the container with the encrypted image

This case will include 2 parts of work:

Image encryption and upload the decryption key to KBS
Kata container boot with guest-pull function, and then doing remote attestation to get the decryption
We will enable the force_guest_pull, and no image snapshotter service(like Nydus )is needed.

Before start, make sure that the two configurations items is enabled:

# Do not share between host and guest
shared_fs = "none"

# Force enable the guest_pull features which will pull image in the guest
experimental_force_guest_pull = true

Setup the local container registry

This step we need to setup a local container registry, please note that the registry node should be different with the Kata container host.

In the registry node, install docker.

Then use the self signed key, and set the node DNS name as registry.com

mkdir -p /home/linaro/certs
openssl req -newkey rsa:4096 \
-addext "subjectAltName = DNS:registry.com" \
-nodes -sha256 -keyout /home/linaro/certs/domain.key \
-x509 -days 365 -out /home/linaro/certs/domain.crt

NOTE when the command prompted with Common Name, please type into registry.com

Common Name (e.g. server FQDN or YOUR name) []:registry.com

Set the local registry host name IP:

vim /etc/hosts
127.0.0.1 registry.com

Add the domain.crt into registry node filesystem root cert, so that we can get access to the local container registry locally with https. In Ubuntu/Debian execute this:

cp certs/domain.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

In Centos:

cat /home/linaro/certs/domain.crt >>/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
cat /home/linaro/certs/domain.crt >>/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

Start the container registry

docker run -d -p 5000:5000 \
--restart=always \
--name registry.com \
-v /home/linaro/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
-v /home/linaro/data:/var/lib/registry \
registry:2

Write the domain.crt to the Kata Guest rootfs

Copy the domain.crt to the Kata Container Host node. This process either do like below or we can pre-set when we do the Kata rootfs build process.

mkdir mnt
qemu-nbd --connect=/dev/nbd0 /opt/kata/share/defaults/kata-containers-confidential.img
mount /dev/nbd0p1 mnt/
cat domain.crt >> mnt/etc/ssl/certs/ca-certificates.crt

Add the DNS info for the Registry Node. Assume the Registry Node IP is: 192.168.122.1:

vim mnt/ect/hosts
# Add below to the file
192.168.122.1 registry.com

Umount the image:

umount mnt/
qemu-nbd --disconnect /dev/nbd0

Use Skopeo to Prepare the encrypted image

# Prepare the encrypted image
KEY_FILE="image_key"
head -c 32 /dev/urandom | openssl enc > "$KEY_FILE"

# create the directory for the key
mkdir output

# Use the coco-keyprovider container image to encrpyted image
# Parameters: 
# -i : the Key storaged path in keybroker.
# -s : the image to be encrypted.
# -k : The key to encrypt the container image.
# -d : The path for the output images.
docker run -v "$PWD/output:/output" \
ghcr.io/confidential-containers/staged-images/coco-keyprovider:latest /encrypt.sh \
-k "$(base64 < image_key)" \
-i kbs:///default/image_key/ubuntu_enc \
-s docker://ubuntu:latest \
-d dir:/output

# Push the image to local registry
skopeo copy dir:output docker://registry.com:5000/ubuntu_enc:latest

# Use the command below to see if the image is encrypted successfully
skopeo inspect docker://registry.com:5000/ubuntu_enc:latest

If the container image manifest shows like "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted", that means the image is right encrypted.

The image_key generated above needs to copied to Trustee Service node, with the directory path:

/opt/confidential-containers/kbs/repository/default/image_key/ubuntu_enc

Launch the container with the following command:

sudo nerdctl run  --runtime io.containerd.kata.v2 --annotation io.kubernetes.cri.image-name=registry.com:5000/ubuntu-encrypt:noble --rm -it registry.com:5000/ubuntu-encrypt:noble

The --annotation io.kubernetes.cri.image-name indicates the image to be pulled in the guest.
The image name pass to annotation should be the same with the container image parameters in nerdctl command, otherwise the container can not start.

The totally working flow is:

Build and run CoCo/Kata Container with RME Support