Abstract
This documentation describes how to use the primitive commands for generating Android images and running Android on virtual platform. In the end, we can see how to directly run Android Automotive OS on Xen virtual machine with QEMU command.
What’s Cuttlefish flow?
Cuttlefish allows user to use command line interface (CLI) to launch Android images on Cuttlefish virtual platform, it provides the command launch_cvd for this purpose.
The prerequisites for using the launch_cvd is Cuttlefish package. The package contains binaries and utilites. We can download the Cuttlefish package for Arm64 via http://ci.android.com/, then search the branch aosp-main-throttled and select the device target aosp_cf_arm64_only_phone-trunk_staging-userdebug .
After downloading and uncompress the Cuttlefish package, combining with Android images, we can execute the launch_cvd command to boot up Android on Cuttlefish:
| Code Block |
|---|
$ HOME=$PWD ./bin/launch_cvd |
The launch_cvd command rus two commands assemble_cvd and run_cvd in its child processes. If both commands work as expected, the Android can launch successfully on Cuttlefish machine. The assemble_cvd command generates root file system images with qcow2 overlay format, then the run_cvd command invokes the virtual machine manager (either Crosvm or QEMU) to boot the generated images on virtual machine. The diagram below shows the Cuttlefish flow.
Using assemble_cvd
Let’s we assume we work on TRS and want to run Android Automotive OS on Xen virtual machine, in this case, it’s likely we cannot directly use launch_cvd command for launching Xen virtual machine. Then we need to use assumble_cvd command to help us to generate file system images, and we can execute QEMU command to launch Xen virtual machine. This is why we need to understand how to use assumble_cvd command.
If directly run assumble_cvd command, you will receive an error log and the program exits abnormaly:
| Code Block |
|---|
leoy@leoy-yangtze:~/Dev2/aosp_cuttlefish_test/cf$ HOME=$PWD ./bin/assemble_cvd 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 assemble_cvd failed: 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 | device/google/cuttlefish/host/commands/assemble_cvd/assemble_cvd.cc:484 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 | Result<int> cuttlefish::AssembleCvdMain(int, char **) 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 v CF_EXPECT(tty == 0) 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 stdin was a tty, expected to be passed the output of a previous stage. Did you mean to run launch_cvd? Aborted |
The log reminds user to invoke launch_cvd instead of the assemble_cvd command. But this doesn’t mean we really cannot run assemble_cvd command directly. Alternatively, we can save what’s the command and arguments into a shell script (we named it as test.sh):
| Code Block |
|---|
HOME=$PWD ./bin/assemble_cvd --group_id=cvd_1 --report_anonymous_usage_stats=y --vm_manager=qemu_cli --webrtc_device_id=cvd_1-1 --gpu_mode=drm_virgl |
Then we can use the to launch the shell script with below command. This command can help use to close stdin and direct the stdout to the pipe :
| Code Block |
|---|
true | sh test.sh 2>&1 | cat |
Using run_cvd
Same as the assemble_cvd command, it prevents to directly run the run_cvd command. If we want to directly execute it, we can use the command:
| Code Block |
|---|
$ true | HOME=$PWD ./bin/run_cvd 2>&1 | cat |
run_cvd manages backend programs internally for emulating virtual devices (e.g. for WiFi, secure devices, etc). For the rough understanding of the run_cvd’s internal, you could refer to below diagram:
QEMU command
After the configurations for the virtual device’s emulation, it launches VMM (QEMU or Crosvm) to kick off virtual machine instance. Below is the compelete command parameters for the QEMU command:
| Code Block |
|---|
qemu-system-x86_64 -name guest=cvd-1,debug-threads=on -machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off -m size=4096M,maxmem=4102M,slots=2 -overcommit mem-lock=off -smp 2,cores=2,threads=1 -uuid 699acfc4-c8c4-11e7-882b-5065f31dc101 -no-user-config -nodefaults -no-shutdown -rtc base=utc -boot strict=on -chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-1/internal/qemu_monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -display egl-headless -vnc 127.0.0.1:544 -device pcie-pci-bridge,id=hvc-bridge,addr=01.2 -device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280 -chardev file,id=serial0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -serial chardev:serial0 -chardev file,id=hvc0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01 -device virtconsole,bus=virtio-serial0.0,chardev=hvc0 -chardev null,id=hvc1 -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial1,bus=hvc-bridge,addr=02 -device virtconsole,bus=virtio-serial1.0,chardev=hvc1 -chardev file,id=hvc2,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/logcat-pipe,append=on -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial2,bus=hvc-bridge,addr=03 -device virtconsole,bus=virtio-serial2.0,chardev=hvc2 -chardev pipe,id=hvc3,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/keymaster_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial3,bus=hvc-bridge,addr=04 -device virtconsole,bus=virtio-serial3.0,chardev=hvc3 -chardev pipe,id=hvc4,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/gatekeeper_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial4,bus=hvc-bridge,addr=05 -device virtconsole,bus=virtio-serial4.0,chardev=hvc4 -chardev pipe,id=hvc5,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/bt_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial5,bus=hvc-bridge,addr=06 -device virtconsole,bus=virtio-serial5.0,chardev=hvc5 -chardev pipe,id=hvc6,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/gnsshvc_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial6,bus=hvc-bridge,addr=07 -device virtconsole,bus=virtio-serial6.0,chardev=hvc6 -chardev pipe,id=hvc7,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/locationhvc_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial7,bus=hvc-bridge,addr=08 -device virtconsole,bus=virtio-serial7.0,chardev=hvc7 -chardev null,id=hvc8 -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial8,bus=hvc-bridge,addr=09 -device virtconsole,bus=virtio-serial8.0,chardev=hvc8 -chardev pipe,id=hvc9,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/uwb_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial9,bus=hvc-bridge,addr=0a -device virtconsole,bus=virtio-serial9.0,chardev=hvc9 -chardev pipe,id=hvc10,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/oemlock_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial10,bus=hvc-bridge,addr=0b -device virtconsole,bus=virtio-serial10.0,chardev=hvc10 -chardev pipe,id=hvc11,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/keymint_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial11,bus=hvc-bridge,addr=0c -device virtconsole,bus=virtio-serial11.0,chardev=hvc11 -chardev pipe,id=hvc12,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/nfc_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial12,bus=hvc-bridge,addr=0d -device virtconsole,bus=virtio-serial12.0,chardev=hvc12 -chardev null,id=hvc13 -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial13,bus=hvc-bridge,addr=0e -device virtconsole,bus=virtio-serial13.0,chardev=hvc13 -chardev null,id=hvc14 -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial14,bus=hvc-bridge,addr=0f -device virtconsole,bus=virtio-serial14.0,chardev=hvc14 -chardev null,id=hvc15 -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial15,bus=hvc-bridge,addr=10 -device virtconsole,bus=virtio-serial15.0,chardev=hvc15 -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/overlay.img,if=none,id=drive-virtio-disk0,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/persistent_composite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2 -object memory-backend-file,id=objpmem0,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/pstore,size=2097152 -device nvdimm,memdev=objpmem0,id=ramoops -object memory-backend-file,id=objpmem1,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/access-kregistry,size=2097152 -device virtio-pmem-pci,disable-legacy=on,memdev=objpmem1,id=pmem0 -object memory-backend-file,id=objpmem2,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/hwcomposer-pmem,size=2097152 -device virtio-pmem-pci,disable-legacy=on,memdev=objpmem2,id=pmem1 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000 -device virtio-mouse-pci,disable-legacy=on -device virtio-keyboard-pci,disable-legacy=on -device virtio-keyboard-pci,disable-legacy=on -device virtio-balloon-pci-non-transitional,id=balloon0 -netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no -device virtio-net-pci-non-transitional,netdev=hostnet0,id=net0,mac=00:1a:11:e0:cf:00 -device virtio-net-pci-non-transitional,netdev=hostnet1,id=net1,mac=00:1a:11:e1:cf:00 -cpu host -msg timestamp=on -device vhost-vsock-pci-non-transitional,guest-cid=3 -device AC97,audiodev=audio_none -audiodev driver=none,id=audio_none -device qemu-xhci,id=xhci -bios /home/leoy/Dev2/aosp_cuttlefish_test/cf/etc/bootloader_x86_64/bootloader.qemu |
Let’s look into these parameters:-name guest=cvd-1,debug-threads=on : Set the name of the guest as cvd-1 and enable debug-threads so individual threads have dedicated names.
-machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off : a standard PC architecture and enables NVDIMM support, enables KVM accelerator, doesn’t include guest memory in a core dump, disables USB support.
-m size=4096M,maxmem=4102M,slots=2 : Sets guest startup RAM size to 4096MB. Set amount of hotpluggable memory slots as 2 and maximum amount of memory 4102MB.
-overcommit mem-lock=off : Disable memory locking (so can be swapped to swap area).
-uuid 699acfc4-c8c4-11e7-882b-5065f31dc101 : Set virtual system UUID.
-no-user-config : Don’t load any of the user-provided config files on sysconfdir.
-nodefaults : Don’t create default devices.
-no-shutdown : Don’t exit QEMU on guest shutdown.
-smp 2,cores=2,threads=1 : 2 virtual CPUs, and each CPU is part of a 2-core socket with one thread per core.
-rtc base=utc : virtual RTC is initialized with the current UTC time.
-chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-/internal/qemu_monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control : establish a socket-based character device for monitoring and control QEMU instance.
-display egl-headless -vnc 127.0.0.1:544 : Use EGL in a headless mode for graphics rendering. Additionally, a VNC server is enabled on localhost and port 544, allowing remote access to the virtual machine's display. ('-display sdl` is used by Xen)
-device pcie-pci-bridge,id=hvc-bridge,addr=01.2 : Emulate a PCIe to PCI bus with PCI address "01.2". (Needed by Xen)
-device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280 : Enable virgl GPU mode (host rendering). (Needed by Xen)
-chardev file,id=serial0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -serial chardev:serial0 : Create virtual serial device and bind it to the local pipe kernel-log-pipe.
-chardev file,id=hvc0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01 -device virtconsole,bus=virtio-serial0.0,chardev=hvc0 : Create a hypervisor virtual console and attached to local pipe.
Cuttlefish creates HVC 0~15. Except HVC 14 and 15 are not used, other HVCs are bound with the pipes on the host and works as communication channel with the backend program on host. (HVC0 is needed by Xen)
| Code Block |
|---|
- /dev/hvc0 = kernel console - /dev/hvc1 = serial console - /dev/hvc2 = serial logging - /dev/hvc3 = keymaster - /dev/hvc4 = gatekeeper - /dev/hvc5 = bt - /dev/hvc6 = gnss - /dev/hvc7 = location - /dev/hvc8 = confirmationui - /dev/hvc9 = uwb - /dev/hvc10 = oemlock - /dev/hvc11 = keymint - /dev/hvc12 = NFC - /dev/hvc13 = sensors |
-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/overlay.img,if=none,id=drive-virtio-disk0,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 : Enable virtual block device for the Android overlay image, which contains boot images, meta images, super.img and userdata.img, etc. (Needed by Xen)
-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/persistent_composite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1: Enable virtual block device for the Android persistent image, which contains uboot enviriable image, bootconfig (for Android configurations), etc. (Needed by Xen)
-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2 : Enable virtual block device for the Android SD image. (Needed by Xen)
-object memory-backend-file,id=objpmem0,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/pstore,size=2097152 -device nvdimm,memdev=objpmem0,id=ramoops : Enable memory object and store the data into file, which is used for kernel ramoops. (Needed by Xen)
-object memory-backend-file,id=objpmem2,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/hwcomposer-pmem,size=2097152 -device virtio-pmem-pci,disable-legacy=on,memdev=objpmem2,id=pmem1 : Enable memory object for hwcomposer.
-object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000 : Enable random device (Needed by Xen)
-device virtio-mouse-pci,disable-legacy=on : Enable virtual mouse (Needed by Xen)-device virtio-keyboard-pci,disable-legacy=on : Enable virtual keyboard (Needed by Xen)
-device virtio-balloon-pci-non-transitional,id=balloon0 : Enable balloon for dynamically allocation memory size.
-netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no -device virtio-net-pci-non-transitional,netdev=hostnet0,id=net0,mac=00:1a:11:e0:cf:00 -device virtio-net-pci-non-transitional,netdev=hostnet1,id=net1,mac=00:1a:11:e1:cf:00 : Enable virtual network cards (Needed by Xen)
-device AC97,audiodev=audio_none -audiodev driver=none,id=audio_none : Enable sound device.
-bios /home/leoy/Dev2/aosp_cuttlefish_test/cf/etc/bootloader_x86_64/bootloader.qemu : Enable bootloader (U-boot). (Needed by Xen)
Suggested QEMU command for Xen virtual machine
Below is the suggested command for running Android image on Xen virtual machine. It has not been tested yet. Once we have hardware is avaliable for the testing, we might update below arguments.
| Code Block |
|---|
$ /usr/bin/qemu-system-aarch64 -xen-domid 1 -no-shutdown -chardev socket,id=xen-cmd,path=/var/run/xen/qmp-libxl-1,server=on,wait=off -mon chardev=xen-cmd,mode=control -nodefaults -no-user-config -xen-attach -name aaos-vm -vnc none -display sdl,gl=on -device pcie-pci-bridge,id=hvc-bridge,addr=01.2 -device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280 -device virtio-mouse-pci,disable-legacy=on,iommu_platform=on -device virtio-keyboard-pci,disable-legacy=on,iommu_platform=on -smp 4,maxcpus=4 -machine xenpvh -m 6144 -chardev file,id=serial0,path=/path/to/kernel-log-pipe,append=on -serial chardev:serial0 -chardev file,id=hvc0,path=/path/to/kernel-log-pipe,append=on -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01 -device virtconsole,bus=virtio-serial0.0,chardev=hvc0 -drive file=/path/to/overlay.img,if=none,id=drive-virtio-disk0,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/path/to/persistent_composite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/path/to/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2 -object memory-backend-file,id=objpmem0,share=on,mem-path=/path/to/pstore,size=2097152 -device nvdimm,memdev=objpmem0,id=ramoops -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000 -netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no -device virtio-net-pci-non-transitional,netdev=hostnet0,id=net0,mac=00:1a:11:e0:cf:00 -device virtio-net-pci-non-transitional,netdev=hostnet1,id=net1,mac=00:1a:11:e1:cf:00 -bios /path/to/u-boot.bin |