Booting a Android Cuttlefish guest directly with QEMU

Owned by Alex Bennée
Last updated: Jan 03, 2024 by Leo Yan (Deactivated)
7 min read

Abstract

This documentation describes how to use the primitive commands for generating Android images and running Android on virtual platform. In the end, we can see how to directly run Android Automotive OS on Xen virtual machine with QEMU command.

What’s Cuttlefish flow?

Cuttlefish allows user to use command line interface (CLI) to launch Android images on Cuttlefish virtual platform, it provides the command launch_cvd for this purpose.

The prerequisites for using the launch_cvd is Cuttlefish package. The package contains binaries and utilites. We can download the Cuttlefish package for Arm64 via http://ci.android.com/, then search the branch aosp-main-throttled and select the device target aosp_cf_arm64_only_phone-trunk_staging-userdebug .

After downloading and uncompress the Cuttlefish package, combining with Android images, we can execute the launch_cvd command to boot up Android on Cuttlefish:

$ HOME=$PWD ./bin/launch_cvd

The launch_cvd command rus two commands assemble_cvd and run_cvd in its child processes. If both commands work as expected, the Android can launch successfully on Cuttlefish machine. The assemble_cvd command generates root file system images with overlay / qcow2 format, then the run_cvd command invokes the virtual machine manager (either Crosvm or QEMU) to boot the virtual machine with the generated images. The diagram below shows the Cuttlefish flow.

Using assemble_cvd

Let’s assume we work on TRS and want to run Android Automotive OS on Xen virtual machine, in this case, we cannot directly use launch_cvd command for launching Xen virtual machine. Therefore, we need to use assumble_cvd command to help us to generate file system images, and afterwards we can execute QEMU command to launch Xen virtual machine. This is why we need to understand how to use assumble_cvd command.

If directly run assumble_cvd command, you will receive an error log and the program exits abnormaly:

leoy@leoy-yangtze:~/Dev2/aosp_cuttlefish_test/cf$ HOME=$PWD ./bin/assemble_cvd 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 assemble_cvd failed: 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 | device/google/cuttlefish/host/commands/assemble_cvd/assemble_cvd.cc:484 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 | Result<int> cuttlefish::AssembleCvdMain(int, char **) 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 v CF_EXPECT(tty == 0) 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 stdin was a tty, expected to be passed the output of a previous stage. Did you mean to run launch_cvd? Aborted

The log reminds user to invoke launch_cvd instead of the assemble_cvd command. But this doesn’t mean we really cannot run assemble_cvd command directly. Alternatively, we can save what’s the command and arguments into a shell script (we named it as test.sh):

HOME=$PWD ./bin/assemble_cvd --group_id=cvd_1 --report_anonymous_usage_stats=y --vm_manager=qemu_cli --webrtc_device_id=cvd_1-1 --gpu_mode=drm_virgl

Then we can launch the shell script with below command. This command closes stdin and directs the stdout to the pipe for workaround the failure mentioned above:

 

Note, as you have noticed that we can pass most launch_cvd’s parameters to assemble_cvd , but it is not always true that these two commands can support exact same parameters. If you want to check what’s the parameters passed to assemble_cvd , you can firstly run launch_cvd and after assumble_cvd is launched in the sub process, you can check the parameters for the assumble_cvd process:

The Android images

Using above command, the two prebuilt Android images are shared in:
cuttlefish_drm_virgl_venus.tgz : Android automotive images with GPU mode drm_virgl for venus
cuttlefish_gfxstream.tgz : Android automotive images with GPU mode gfxstream for vulkan.

Download the package and uncompress it in the folder /home/leoy/Dev2/aosp_cuttlefish_test/cf to ensure all linkages work well.

Using run_cvd

Same as the assemble_cvd command, we are prevent to directly run the run_cvd command. If we want to execute it, we can use the command:

run_cvd manages dozen of backend programs internally for emulating virtual devices (e.g. for WiFi, secure devices, etc). For the brief understanding the run_cvd’s internal, you could refer to below diagram:

QEMU command

After the configurations for the virtual device’s emulation, it launches VMM (QEMU or Crosvm) to kick off virtual machine instance. Below is the compelete command parameters for the QEMU command:

Let’s look into these parameters:
-name guest=cvd-1,debug-threads=on : Set the name of the guest as cvd-1 and enable debug-threads so individual threads have dedicated names.

-machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off : a standard PC architecture and enables NVDIMM support, enables KVM accelerator, doesn’t include guest memory in a core dump, disables USB support.

-m size=4096M,maxmem=4102M,slots=2 : Sets guest startup RAM size to 4096MB. Set amount of hotpluggable memory slots as 2 and maximum amount of memory 4102MB.

-overcommit mem-lock=off : Disable memory locking (so can be swapped to swap area).

-uuid 699acfc4-c8c4-11e7-882b-5065f31dc101 : Set virtual system UUID.

-no-user-config : Don’t load any of the user-provided config files on sysconfdir.

-nodefaults : Don’t create default devices.

-no-shutdown : Don’t exit QEMU on guest shutdown.

-smp 2,cores=2,threads=1 : 2 virtual CPUs, and each CPU is part of a 2-core socket with one thread per core.

-rtc base=utc : virtual RTC is initialized with the current UTC time.

-chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-/internal/qemu_monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control : establish a socket-based character device for monitoring and control QEMU instance.

-display egl-headless -vnc 127.0.0.1:544 : Use EGL in a headless mode for graphics rendering. Additionally, a VNC server is enabled on localhost and port 544, allowing remote access to the virtual machine's display. ('-display sdl` is used by Xen)

-device pcie-pci-bridge,id=hvc-bridge,addr=01.2 : Emulate a PCIe to PCI bus with PCI address "01.2". (Needed by Xen)

-device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280 : Enable virgl GPU mode (host rendering). (Needed by Xen)

-chardev file,id=serial0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -serial chardev:serial0 : Create virtual serial device and bind it to the local pipe kernel-log-pipe.

-chardev file,id=hvc0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01 -device virtconsole,bus=virtio-serial0.0,chardev=hvc0 : Create a hypervisor virtual console and attached to local pipe.

Cuttlefish creates HVC 0~15. Except HVC 14 and 15 are not used, other HVCs are bound with the pipes on the host and works as communication channel with the backend program on host. (HVC0 is needed by Xen)

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/overlay.img,if=none,id=drive-virtio-disk0,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 : Enable virtual block device for the Android overlay image, which contains boot images, meta images, super.img and userdata.img, etc. (Needed by Xen)

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/persistent_composite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1: Enable virtual block device for the Android persistent image, which contains uboot enviriable image, bootconfig (for Android configurations), etc. (Needed by Xen)

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2 : Enable virtual block device for the Android SD image. (Needed by Xen)

-object memory-backend-file,id=objpmem0,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/pstore,size=2097152 -device nvdimm,memdev=objpmem0,id=ramoops : Enable memory object and store the data into file, which is used for kernel ramoops. (Needed by Xen)

-object memory-backend-file,id=objpmem2,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/hwcomposer-pmem,size=2097152 -device virtio-pmem-pci,disable-legacy=on,memdev=objpmem2,id=pmem1 : Enable memory object for hwcomposer.

-object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000 : Enable random device (Needed by Xen)

-device virtio-mouse-pci,disable-legacy=on : Enable virtual mouse (Needed by Xen)
-device virtio-keyboard-pci,disable-legacy=on : Enable virtual keyboard (Needed by Xen)

-device virtio-balloon-pci-non-transitional,id=balloon0 : Enable balloon for dynamically allocation memory size.

-netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no -device virtio-net-pci-non-transitional,netdev=hostnet0,id=net0,mac=00:1a:11:e0:cf:00 -device virtio-net-pci-non-transitional,netdev=hostnet1,id=net1,mac=00:1a:11:e1:cf:00 : Enable virtual network cards (Needed by Xen)

-device AC97,audiodev=audio_none -audiodev driver=none,id=audio_none : Enable sound device.

-bios /home/leoy/Dev2/aosp_cuttlefish_test/cf/etc/bootloader_x86_64/bootloader.qemu : Enable bootloader (U-boot). (Needed by Xen)

Suggested QEMU command for Xen virtual machine

Below is the suggested command for running Android image on Xen virtual machine. It has not been tested yet. Once we have hardware is avaliable for the testing, we might update below arguments.