Abstract
This documentation describes how to use the primitive commands for generating Android images and running Android on virtual platform. In the end, we can see how to directly run Android Automotive OS on Xen virtual machine with QEMU command.
What’s Cuttlefish flow?
Cuttlefish allows user to use command line interface (CLI) to launch Android images on Cuttlefish virtual platform, it provides the command launch_cvd for this purpose.
The prerequisites for using the launch_cvd is Cuttlefish package. The package contains binaries and utilites. We can download the Cuttlefish package for Arm64 via http://ci.android.com/, then search the branch aosp-main-throttled and select the device target aosp_cf_arm64_only_phone-trunk_staging-userdebug .
After downloading and uncompress the Cuttlefish package, combining with Android images, we can execute the launch_cvd command to boot up Android on Cuttlefish:
$ HOME=$PWD ./bin/launch_cvd
The launch_cvd command rus two commands assemble_cvd and run_cvd in its child processes. If both commands work as expected, the Android can launch successfully on Cuttlefish machine. The assemble_cvd command generates root file system images with qcow2 overlay format, then the run_cvd command invokes the virtual machine manager (either Crosvm or QEMU) to boot the generated images on virtual machine. The diagram below shows the Cuttlefish flow.
Using assemble_cvd
Let’s we assume we work on TRS and want to run Android Automotive OS on Xen virtual machine, in this case, it’s likely we cannot directly use launch_cvd command for launching Xen virtual machine. Then we need to use assumble_cvd command to help us to generate file system images, and we can execute QEMU command to launch Xen virtual machine. This is why we need to understand how to use assumble_cvd command.
If directly run assumble_cvd command, you will receive an error log and the program exits abnormaly:
leoy@leoy-yangtze:~/Dev2/aosp_cuttlefish_test/cf$ HOME=$PWD ./bin/assemble_cvd 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 assemble_cvd failed: 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 | device/google/cuttlefish/host/commands/assemble_cvd/assemble_cvd.cc:484 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 | Result<int> cuttlefish::AssembleCvdMain(int, char **) 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 v CF_EXPECT(tty == 0) 12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 stdin was a tty, expected to be passed the output of a previous stage. Did you mean to run launch_cvd? Aborted
The log reminds user to invoke launch_cvd instead of the assemble_cvd command. But this doesn’t mean we really cannot run assemble_cvd command directly. Alternatively, we can save what’s the command and arguments into a shell script (we named it as test.sh):
HOME=$PWD ./bin/assemble_cvd --group_id=cvd_1 --report_anonymous_usage_stats=y --vm_manager=qemu_cli --webrtc_device_id=cvd_1-1 --gpu_mode=drm_virgl
Then we can use the to launch the shell script with below command. This command can help use to close stdin and direct the stdout to the pipe :
true | sh test.sh 2>&1 | cat
Using run_cvd
Same as the assemble_cvd command, it prevents to directly run the run_cvd command. If we want to directly execute it, we can use the command:
$ true | HOME=$PWD ./bin/run_cvd 2>&1 | cat
run_cvd manages backend programs internally for emulating virtual devices (e.g. for WiFi, secure devices, etc). For the rough understanding of the run_cvd’s internal, you could refer to below diagram:
QEMU command
After the configurations for the virtual device’s emulation, it launches VMM (QEMU or Crosvm) to kick off virtual machine instance. Below is the compelete command parameters for the QEMU command:
qemu-system-x86_64 -name guest=cvd-1,debug-threads=on -machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off -m size=4096M,maxmem=4102M,slots=2 -overcommit mem-lock=off -smp 2,cores=2,threads=1 -uuid 699acfc4-c8c4-11e7-882b-5065f31dc101 -no-user-config -nodefaults -no-shutdown -rtc base=utc -boot strict=on -chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-1/internal/qemu_monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -display egl-headless -vnc 127.0.0.1:544 -device pcie-pci-bridge,id=hvc-bridge,addr=01.2 -device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280 -chardev file,id=serial0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -serial chardev:serial0 -chardev file,id=hvc0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01 -device virtconsole,bus=virtio-serial0.0,chardev=hvc0 -chardev null,id=hvc1 -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial1,bus=hvc-bridge,addr=02 -device virtconsole,bus=virtio-serial1.0,chardev=hvc1 -chardev file,id=hvc2,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/logcat-pipe,append=on -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial2,bus=hvc-bridge,addr=03 -device virtconsole,bus=virtio-serial2.0,chardev=hvc2 -chardev pipe,id=hvc3,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/keymaster_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial3,bus=hvc-bridge,addr=04 -device virtconsole,bus=virtio-serial3.0,chardev=hvc3 -chardev pipe,id=hvc4,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/gatekeeper_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial4,bus=hvc-bridge,addr=05 -device virtconsole,bus=virtio-serial4.0,chardev=hvc4 -chardev pipe,id=hvc5,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/bt_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial5,bus=hvc-bridge,addr=06 -device virtconsole,bus=virtio-serial5.0,chardev=hvc5 -chardev pipe,id=hvc6,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/gnsshvc_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial6,bus=hvc-bridge,addr=07 -device virtconsole,bus=virtio-serial6.0,chardev=hvc6 -chardev pipe,id=hvc7,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/locationhvc_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial7,bus=hvc-bridge,addr=08 -device virtconsole,bus=virtio-serial7.0,chardev=hvc7 -chardev null,id=hvc8 -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial8,bus=hvc-bridge,addr=09 -device virtconsole,bus=virtio-serial8.0,chardev=hvc8 -chardev pipe,id=hvc9,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/uwb_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial9,bus=hvc-bridge,addr=0a -device virtconsole,bus=virtio-serial9.0,chardev=hvc9 -chardev pipe,id=hvc10,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/oemlock_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial10,bus=hvc-bridge,addr=0b -device virtconsole,bus=virtio-serial10.0,chardev=hvc10 -chardev pipe,id=hvc11,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/keymint_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial11,bus=hvc-bridge,addr=0c -device virtconsole,bus=virtio-serial11.0,chardev=hvc11 -chardev pipe,id=hvc12,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/nfc_fifo_vm -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial12,bus=hvc-bridge,addr=0d -device virtconsole,bus=virtio-serial12.0,chardev=hvc12 -chardev null,id=hvc13 -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial13,bus=hvc-bridge,addr=0e -device virtconsole,bus=virtio-serial13.0,chardev=hvc13 -chardev null,id=hvc14 -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial14,bus=hvc-bridge,addr=0f -device virtconsole,bus=virtio-serial14.0,chardev=hvc14 -chardev null,id=hvc15 -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial15,bus=hvc-bridge,addr=10 -device virtconsole,bus=virtio-serial15.0,chardev=hvc15 -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/overlay.img,if=none,id=drive-virtio-disk0,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/persistent_composite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2 -object memory-backend-file,id=objpmem0,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/pstore,size=2097152 -device nvdimm,memdev=objpmem0,id=ramoops -object memory-backend-file,id=objpmem1,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/access-kregistry,size=2097152 -device virtio-pmem-pci,disable-legacy=on,memdev=objpmem1,id=pmem0 -object memory-backend-file,id=objpmem2,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/hwcomposer-pmem,size=2097152 -device virtio-pmem-pci,disable-legacy=on,memdev=objpmem2,id=pmem1 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000 -device virtio-mouse-pci,disable-legacy=on -device virtio-keyboard-pci,disable-legacy=on -device virtio-keyboard-pci,disable-legacy=on -device virtio-balloon-pci-non-transitional,id=balloon0 -netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no -device virtio-net-pci-non-transitional,netdev=hostnet0,id=net0,mac=00:1a:11:e0:cf:00 -device virtio-net-pci-non-transitional,netdev=hostnet1,id=net1,mac=00:1a:11:e1:cf:00 -cpu host -msg timestamp=on -device vhost-vsock-pci-non-transitional,guest-cid=3 -device AC97,audiodev=audio_none -audiodev driver=none,id=audio_none -device qemu-xhci,id=xhci -bios /home/leoy/Dev2/aosp_cuttlefish_test/cf/etc/bootloader_x86_64/bootloader.qemu
Let’s look into these parameters:-name guest=cvd-1,debug-threads=on : Set the name of the guest as cvd-1 and enable debug-threads so individual threads have dedicated names.
-machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off : a standard PC architecture and enables NVDIMM support, enables KVM accelerator, doesn’t include guest memory in a core dump, disables USB support.
-m size=4096M,maxmem=4102M,slots=2 : Sets guest startup RAM size to 4096MB. Set amount of hotpluggable memory slots as 2 and maximum amount of memory 4102MB.
-overcommit mem-lock=off : Disable memory locking (so can be swapped to swap area).
-uuid 699acfc4-c8c4-11e7-882b-5065f31dc101 : Set virtual system UUID.
-no-user-config : Don’t load any of the user-provided config files on sysconfdir.
-nodefaults : Don’t create default devices.
-no-shutdown : Don’t exit QEMU on guest shutdown.
-smp 2,cores=2,threads=1 : 2 virtual CPUs, and each CPU is part of a 2-core socket with one thread per core.
-rtc base=utc : virtual RTC is initialized with the current UTC time.
-chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-/internal/qemu_monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control : establish a socket-based character device for monitoring and control QEMU instance.
-display egl-headless -vnc 127.0.0.1:544 : Use EGL in a headless mode for graphics rendering. Additionally, a VNC server is enabled on localhost and port 544, allowing remote access to the virtual machine's display.
-device pcie-pci-bridge,id=hvc-bridge,addr=01.2 : Emulate a PCIe to PCI bus with PCI address "01.2".