/
Build and run CoCo/Kata Container with RME Support

Build and run CoCo/Kata Container with RME Support

Owned by Kevin Zhao
Last updated: Oct 28, 2024
5 min read

Limitation

Launch Kata/CoCo now with CCA support, and we can do:

  • Launch Kata-sandbox and container inside the sandbox

  • After launching the Kata container, manually jump into the Kata sandbox to verify the function for remote attestation with Trustee cluster.

  • A modified Kata-agent is needed to disable the attestation-agent/api-server-rest/Confidential Datahub component to start in the Kata-agent start process.

  • No image encryption / decryption and sign support on in the CoCo boot process.

Build Environment

Ubuntu 22.04 Arm64 Machine, Kernel 6.5.0, native build. The total software stack is based on cca-v3 patch series.

Build process

git clone https://git.codelinaro.org/linaro/dcap/cca-demos/kata-containers cd kata-containers git checkout origin/cca-v3-kata-3.7

Prepare the Kernel headers for building the Kata-runtime. The kernel headers are generated in Kernel building process and it will not covered in this document. You can easy get from make headers_install in the kernel building process, or just grab from here: https://people.linaro.org/~kevin.zhao/kernel-headers.tar.gz

cd kata-containers/tools/packaging/static-build/shim-v2 cp ~/kernel-headers.tar.gz kata-containers/tools/packaging/static-build/shim-v2/

Build Kata component:

make -C kata-containers/tools/packaging/kata-deploy/local-build TEE_PLATFORM=cca shim-v2-tarball rootfs-image-cca-tarball rootfs-initrd-cca-tarball

After run the command, it will automatically build the:

  • Kata Guest Kernel(cca-v3)

  • Kata rootfs image/Kata initrd image(Kata-agent and CoCo guest components will be also included)

  • Kata container kata-runtime/kata-monitor

Build Qemu

Refer to this link,Building an RME stack for QEMU | QEMU VMM , the process is the same. Assume that the binaries is located at /root/qemu/build/qemu-system-aarch64

Prepare Kata container Host

In order to launch the Kata-container, it needs the host kernel have different configurations. The demo kernel configuration is https://people.linaro.org/~kevin.zhao/config.cca-v3.host

Repo: https://gitlab.arm.com/linux-arm/linux-cca cca-full/v3

cd linux-cca cp config.cca-v3.host make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 -j8

Follow the instructions show here Building an RME stack for QEMU | Build the Ubuntu Rootfs , you will get a ubuntu rootfs. Assume it is named as ubuntu22.img, mount at /root/mnt, then make modules install to put the modules to the rootfs.

sudo mount /root/ubuntu22.img /root/mnt cd ../linux-cca sudo make modules_install ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=/root/mnt sudo umount /root/mnt

Install and Configure the Kata container

Launch the Kata container Host

Can refer to : https://linaro.atlassian.net/wiki/spaces/QEMU/pages/29051027459/Building+an+RME+stack+for+QEMU#With-the-OP-TEE-build-environment. Assume that you are using the OPTEE build environment to set up the pure software stack, and that will be also easy to launch the host.

cd cca/build

Then, Change the Makefile script, re-direct the kernel/rootfs you just get from Build and run CoCo/Kata Container with RME Support | Prepare Kata container Host .

make CCA_SUPPORT=yes run-only

Next step, copy the binaries to the Kata host.

Install Kata/CoCo from tarball

On the build machine, copy the binaries stored at: kata-containers/tools/packaging/kata-deploy/local-build/build

rm -rf /opt/kata cd binaries/ # only install below packages file_list=("kata-static-kernel-cca-experimental.tar.xz" "kata-static-rootfs-image-cca.tar.xz" "kata-static-rootfs-initrd-cca.tar.xz" "kata-static-shim-v2.tar.xz") for file in "${file_list[@]}" ; do if [ -f "$file" ]; then sudo tar -xJf "$file" -C / fi done

After installation, the rootfs/initrd/Guest kernel are stored at /opt/kata/share, and containerd-shim-kata-v2/Kata-runtime/Kata-monitor are located at /opt/kata/bin

Configure the Kata-containers

Kata-containers configuration

Get the Kata container configuration from: /opt/kata/share/defaults/kata-containers/configuration-qemu.toml, copy it to /etc/kata-containers/configuration.toml

Some essential items should be modified at /etc/kata-containers/configuration.toml. Kata containers can run with either an initrd image or a rootfs image. But they can just be set either rootfs or initrd, please do not set both. Here we test rootfs. Some essential configurations items should be changed as below.

[hypervisor.qemu] path = "/root/qemu/bin/qemu-system-aarch64" kernel = "/opt/kata/share/vmlinux-confidential.container" image = "/opt/kata/share/kata-containers/kata-containers-cca.img" # initrd = "/usr/share/kata-containers/kata-containers-initrd.img" machine_type = "virt" confidential_guest = true shared_fs = "virtio-9p" dial_timeout = 5000 create_container_timeout = 1200

  • shared_fsthe support for virtiofs with Qemu Arm CCA is not validated, so we use virtio-9p for host/guest sharing.

  • Timeout: The two timeout parameters are needed as the low performance at emulator spaces.

  • confidential_guest is the key variable to indicated if we launch the confidential VM or common vm.

Configure the Containerd to use Kata-containers

Configure the Containerd to use Kata-container. Copy https://people.linaro.org/~kevin.zhao/config.toml.containerd to /etc/containerd/config.toml

sudo apt install containerd -y; sudo systemctl daemon-reload sudo systemctl restart containerd

More things for Kata with containerd, please refer here: kata-containers/docs/install/container-manager/containerd/containerd-install.md at main ยท kata-containers/kata-containers

Install CNI and nerdctl to launch the container

CNI - Container network interface

CNI is essential for container network. CNI can be installed either binaries or build from source, check :

More things for Kata with containerd, please refer here: kata-containers/docs/install/container-manager/containerd/containerd-install.md at main ยท kata-containers/kata-containers

nerdctl command line

Install nerdctl command line. nerdctl cmdline will automatically set up the CNI network with host-local, and it is compatible with docker command line. Just copy the binaries to the host and it is ready to go. See more for Nerdctl

Run command like below to launch Kata container:

nerdctl run --runtime io.containerd.kata.v2 -it docker.io/library/busybox:latest sh

nerdctl is the docker-compatible OCI cmd line, it can also automatically configuration the CNI network.

Configure and Launch the Trustee verification service

The Trustee is the remote attestation service and components used for CoCo. The trustee can be deployed at a remote node which must have the network connection with the Realm(CoCo). This one is now configured very easy to deploy and management. Trustee is also automatically for HTTP based connection for demo use.

git clone https://git.codelinaro.org/linaro/dcap/cca-demos/build-ccs.git cd build-ccs/trustee docker compose up -d

Then docker-compose will launch several containers like below:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e3e4dad412fd trustee-keyprovider "coco_keyprovider --โ€ฆ" 7 hours ago Up 7 hours 0.0.0.0:50000->50000/tcp, :::50000->50000/tcp trustee-keyprovider-1 ca5322525a38 trustee-kbs "/usr/local/bin/kbs โ€ฆ" 7 hours ago Up 7 hours 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp trustee-kbs-1 d2be0edaa893 trustee-as "grpc-as --socket 0.โ€ฆ" 7 hours ago Up 7 hours 0.0.0.0:50004->50004/tcp, :::50004->50004/tcp trustee-as-1 aa924b6ea275 trustee-guest "bash" 7 hours ago Up 7 hours trustee-guest-1 4de0fa460325 trustee-rvps "rvps" 7 hours ago Up 7 hours 0.0.0.0:50003->50003/tcp, :::50003->50003/tcp trustee-rvps-1

The port is very important, those ones should be configured at the below get_token steps for remote attestation.

Launch Kata containers and running the Remote attestation

First, on the Realm Host, start the Kata-monitor, this is used to support kata-runtime exec into the sandbox.

# kata-monitor

Then, launch the Kata-containers with nerdctl.

nerdctl run --runtime io.containerd.kata.v2 -it docker.io/library/busybox:latest sh

After about 25mins, the Kata container is ready and we are now in the sh cmdline.

Manually do remote attestation

Use kata-runtime to jump into the sandbox, first pick up the sandbox id.

# ctr c ls CONTAINER IMAGE RUNTIME 9fa87a9bb1f13a0b8e8b14fcfed517616a218f0097e92f36b4d27625c14279ba docker.io/library/busybox:latest io.containerd.kata.v2

Use Kata-runtime to jump into the sandbox, open 3 terminals

kata-runtime exec 9fa87a9bb1f13a0b8e8b14fcfed517616a218f0097e92f36b4d27625c14279ba

Terminal 1, start attestation-agent. The configuration concludes the url for attestation service.

# cat > /tmp/attestation-agent.toml << EOF [token_configs] [token_configs.coco_as] url = "http://213.146.155.117:8000" [token_configs.kbs] url = "http://213.146.155.117:8080" EOF # RUST_LOG=debug attestation-agent --attestation_sock unix:///run/confidential-containers/attestation-agent/attestation-agent.sock -c /tmp/attestation-agent.toml [2024-09-11T03:21:35Z INFO attestation_agent] Using AA config file: /tmp/attestation-agent.toml [2024-09-11T03:21:38Z DEBUG ttrpc_aa] Attestation ttRPC service listening on: "unix:///run/confidential-containers/attestation-agent/attestation-agent.sock"

Terminal 2, start api-server-rest, which is used to export the interface for other services.

# RUST_LOG=debug api-server-rest -b 0.0.0.0:8006 -f attestation Starting API server on 0.0.0.0:8006 with features attestation API Server listening on http://0.0.0.0:8006

Terminal 3, do get_evidence cmd

# curl http://127.0.0.1:8006/aa/token\?token_type\=kbs {"token":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJjdXN0b21pemVkX2NsYWltcyI6eyJpbml0X2RhdGEiOm51bGwsInJ1bnRpbWVfZGF0YSI6eyJub25jZSI6IlhtRkw2Y0JRRHR3dEM5ZkxLSDErV3N6SWNNWFpzZ3djSHQ2bDFHWUpiSVE9IiwidGVlLXB1YmtleSI6eyJhbGciOiJSU0ExXzUiLCJlIjoiQVFBQiIsImt0eSI6IlJTQSIsIm4iOiIxbFkzWEY1dHBHQ0lOZmpwVTd3Y0xxVE1RQkVoVWxkeDU2blVMSExjYkIxcVhZVnJlckN6VWk0TTdvX3RVcFd5SVRmWGJnN3JTWUpYcFBWNmZha1VJVmFtWGdCNjZyQ1g5OXBWRnF4bkUzdUxLSU5kM3VEOF9ZczlnSjk0Q2NuQVBTSUlDcktqRHpwbnd5VDljYkpnV0FyUjVsLV93SlJzVGhGcmtFREpVZUFGcU1sTzlYa2ItZE5KTU9Id1Zlb2xtYjQ2OGFJSWtZYnFWYTZCMVl6S3M3Y09DbXlhOEw5d2RrOTdsVk91Q0VwRHJ0Y1dxWElJT0JkYk9wdlZhcWZxNFd6c1lfX2xMS3BLY3RURTBvNEFPV1R6ekMxaVNkVV8tRkJ0NWFyTkpOejdjajNFRFpfdUhRVGthVmZuT2pjbDhaNVgxQ2w3RjJTZktfSGhSekFEZncifX19LCJldmFsdWF0aW9uLXJlcG9ydHMiOlt7ImV2YWx1YXRpb24tcmVzdWx0Ijp7ImFsbG93Ijp0cnVlfSwicG9saWN5LWhhc2giOiJjMGU3OTI5NjcxZmI2NzgwMzg3ZjU0NzYwZDg0ZDY1ZDJjZTk2MDkzZGZiMzNlZmRhMjFmNWViMDVhZmNkYTc3YmJhNDQ0YzAyY2QxNzdiMjNhNWQzNTA3MTY3MjYxNTciLCJwb2xpY3ktaWQiOiJkZWZhdWx0In1dLCJleHAiOjE3MjI5NTQzNTQsImlhdCI6MTcyMjk1NDA1NCwiaXNzIjoiQ29Dby1BdHRlc3RhdGlvbi1TZXJ2aWNlIiwianRpIjoiSUx3QnlPbVJxNyIsImp3ayI6eyJhbGciOiJSUzM4NCIsImUiOiJBUUFCIiwia3R5IjoiUlNBIiwibiI6IjZBWFgwRDQ4T091THh5Rk02c1A4YWZOSnowZmFLMEU5bFFyTm91QnFzbnRjZEVjSmYzMVlxNFJRSXRxTXQ3S0Rkek1odG51R0NHWlVYc1p3UmNndjhLdVFFMUhvWmVXVk9wVXRybFkzVV9EQkdCeG9mVzJRVi1kUVQyQk0wX2RKRXBfd3B6Qm5DUG1EQTNjZERjS2piRjl5VzFUaHY5cTBDME82ZTBLSmxTakl5aVhGdzBHdURhV1VsNDNVazhKQTVaNVVJQWdGem1EZTBpNnVJU0ZJYVlVZTJCSVQzUXh1YzJqYkwtQTZ6RkxXUW02RzNaSGVoaGhfUng5endtNjNUd1puX25zYkltVHFaRFBZMUI3V2h3UzA2SW1TZE1RcnpxejIydXlGT0cxN1Y1eER2cWpUZWpKM0w0WEYwWnFtN0hncDJhejNQX0tEam9tR2xoSVBHdyJ9LCJuYmYiOjE3MjI5NTQwNTQsInJlZmVyZW5jZS1kYXRhIjp7fSwidGNiLXN0YXR1cyI6eyJjY2EucGxhdGZvcm0uY2NhLXBsYXRmb3JtLWNoYWxsZW5nZSI6bnVsbCwiY2NhLnBsYXRmb3JtLmNjYS1wbGF0Zm9ybS1zdy1jb21wb25lbnRzIjpudWxsLCJjY2EucmVhbG0uY2NhLXJlYWxtLWV4dGVuc2libGUtbWVhc3VyZW1lbnRzLjAiOiJBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBPSIsImNjYS5yZWFsbS5jY2EtcmVhbG0tZXh0ZW5zaWJsZS1tZWFzdXJlbWVudHMuMSI6IkFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUE9IiwiY2NhLnJlYWxtLmNjYS1yZWFsbS1leHRlbnNpYmxlLW1lYXN1cmVtZW50cy4yIjoiQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQT0iLCJjY2EucmVhbG0uY2NhLXJlYWxtLWV4dGVuc2libGUtbWVhc3VyZW1lbnRzLjMiOiJBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBPSIsImNjYS5yZWFsbS5jY2EtcmVhbG0taGFzaC1hbGdvLWlkIjoiIiwiY2NhLnJlYWxtLmNjYS1yZWFsbS1pbml0aWFsLW1lYXN1cmVtZW50IjoiQjdDZmR4Z1pHTmFzMnFEajJOTDVsTHZiNDdCUmRHNFFxZkpVVnpvOVExQT0iLCJjY2EucmVhbG0uY2NhLXJlYWxtLXBlcnNvbmFsaXphdGlvbi12YWx1ZSI6IkFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBPT0iLCJjY2EucmVhbG0uY2NhLXJlYWxtLXB1YmxpYy1rZXktaGFzaC1hbGdvLWlkIjoiIiwiaW5pdF9kYXRhIjoiIiwicmVwb3J0X2RhdGEiOiIifSwidGVlIjoiY2NhIn0.lmBPh0JLvrJXNYsUkcHwWH3MnxEJj-CyqyhYo9TIgpQaUUZDm3KZhAszEa6ZW6pBRZJQ9tCY4xeFOab-yIkTd9RnbcjWEvHyCAs5gSYbI7F0eqSbzF3NNJdZgnKSAhXMGsq2n0MP5voej47Kcs6KRURrUHtTaEQoRy43L-EwWj6WJ8gRRW8n7h9n5ZHVTqXHqjI5YYvgFYkStXhtLoOhop2DidwNGtxjd-s9bw9Lg6DiwZjyeDlHIM2oBt3V_TlMlDoB1cltdTVnSM8GFfGrEq_B7tAbOwZICycX7akUb7vhTFUa2m7uF2ZEkKMjdSD26SlDRR9QZErdT4mnq2S1xA","tee_keypair":"-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA1lY3XF5tpGCINfjpU7wcLqTMQBEhUldx56nULHLcbB1qXYVr\nerCzUi4M7o/tUpWyITfXbg7rSYJXpPV6fakUIVamXgB66rCX99pVFqxnE3uLKINd\n3uD8/Ys9gJ94CcnAPSIICrKjDzpnwyT9cbJgWArR5l+/wJRsThFrkEDJUeAFqMlO\n9Xkb+dNJMOHwVeolmb468aIIkYbqVa6B1YzKs7cOCmya8L9wdk97lVOuCEpDrtcW\nqXIIOBdbOpvVaqfq4WzsY//lLKpKctTE0o4AOWTzzC1iSdU/+FBt5arNJNz7cj3E\nDZ/uHQTkaVfnOjcl8Z5X1Cl7F2SfK/HhRzADfwIDAQABAoIBAHHfCDZ4rIOCDZuv\nDDdCXy6guyrmbu+SKZ/MaD7dHyIJVrMs6Pcb3YkxkglESe9wPFzNkBSWtIYvS2UG\nTfdo1psHsAeM9cCt5W72ffI30eNV5Y62jC6XFZGWYixtjJaWqx1PcTpNwbebbfkN\nM+EtxWXBCdb806D6ubeYJa9T3CE0jKchcZMDCs1tqV9BpvNmmOhfxyeRVYS7th2w\nFWpM2rnsrYkACIPF/NFY1+D/FPyLU0uCDMWA62ip6iCqUNiIxEt/IWFU148Kd71g\nwR2yjNHeIR5xkf3CW4AyXnH6y4ji0bF8ehTFZT3O6SvqHKkRNhU+xHGBpRV1wbj2\nEvGg6qECgYEA6nLePEtW0jjCBiWuh8+QkVHV6PfWj0WhpltisbB1qUI3L7lG8yZn\nvoDYBK1XZzjVMiSZPYD8eLEGv6xvTMDfqgVRSC26dqI6eWoP1aN0xEEAEC2I+/gP\nYVbpPEWEoA+sYcCaAmw3MMxvYkhoPnezUlXNFAFT3wpbq7E1hlg/3HkCgYEA6goR\nLuUYuy90aX2c+WfiaEa53tcSmBacNGURZYQNsnkx0WxLtzPkQZpA+Soi6CgkHlPH\naW61G+8yO3QkjXcrm/SgIJz7+T1SyHZcvthe5yYIFBGOlsNDi1oOIqsBzKZEY5tF\npbYPEW72qER4A8pAMI5MGTEQSwSeFLUhhsz4cbcCgYEAhVh+FDcZk4nauRpXkUE6\nTZ0JGG9WJrkI/xB9Kuldkr5pmwg7VXKjkRPBSa+5xajqgZGVYwjpEFBHsV8HEtWk\noRRTLcVsh6svrPzpMq68bB84RPtRLVbikI+ZKsGzdEJV3mdocTZRkbzFIVZjjBwS\n7yKD+PvkUziV+SpaciXhdKkCgYBgxuDsH1fmr2zX4lVy3H5G2FgryYuq0E6va8uR\nytVPI8Zx7vIwQ4jzh7GuVrowfcjRsZsYAetqLbL72r7ysICb8L/2PHQFUFJXCnq3\ngAleU9+rTMdFmGDp0sddlpEb6vUEq0RVjb8VWmksk4N9Lbj1WyIkmKFAJByqO2wt\nrbxeaQKBgQDbd5JXORLjVnDuONf6Fit86swV5z1khJJimxfxqCbqdJA0Z8zKm/4V\n9A0yEBH3ZCfqIzOkvrgmJ27rR7ek9j1F/olvu+W0Gx8R0qHgr9521AuX34i/XtI/\nB5ChUvwWBtFtYGzWXeflNHMvpLAjyemwKMfi3VbpVLWVZqOCb4ViLw==\n-----END RSA PRIVATE KEY-----\n"}

ย 

Please also check the log for Trustee, you can get the logs.

More about the remote attestation, please also check the comment at https://linaro.atlassian.net/jira/software/c/projects/DCAP/issues/DCAP-94

Related content