Skip to end of banner
Go to start of banner

Booting a Android Cuttlefish guest directly with QEMU

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 18 Next »

Abstract

This documentation describes how to use the primitive commands for generating Android images and running Android on virtual platform. In the end, we can see how to directly run Android Automotive OS on Xen virtual machine with QEMU command.

What’s Cuttlefish flow?

Cuttlefish allows user to use command line interface (CLI) to launch Android images on Cuttlefish virtual platform, it provides the command launch_cvd for this purpose.

The prerequisites for using the launch_cvd is Cuttlefish package. The package contains binaries and utilites. We can download the Cuttlefish package for Arm64 via http://ci.android.com/, then search the branch aosp-main-throttled and select the device target aosp_cf_arm64_only_phone-trunk_staging-userdebug .

After downloading and uncompress the Cuttlefish package, combining with Android images, we can execute the launch_cvd command to boot up Android on Cuttlefish:

$ HOME=$PWD ./bin/launch_cvd

The launch_cvd command rus two commands assemble_cvd and run_cvd in its child processes. If both commands work as expected, the Android can launch successfully on Cuttlefish machine. The assemble_cvd command generates root file system images with overlay / qcow2 format, then the run_cvd command invokes the virtual machine manager (either Crosvm or QEMU) to boot the virtual machine with the generated images. The diagram below shows the Cuttlefish flow.

Using assemble_cvd

Let’s we assume we work on TRS and want to run Android Automotive OS on Xen virtual machine, in this case, it’s likely we cannot directly use launch_cvd command for launching Xen virtual machine. Then we need to use assumble_cvd command to help us to generate file system images, and we can execute QEMU command to launch Xen virtual machine. This is why we need to understand how to use assumble_cvd command.

If directly run assumble_cvd command, you will receive an error log and the program exits abnormaly:

leoy@leoy-yangtze:~/Dev2/aosp_cuttlefish_test/cf$ HOME=$PWD ./bin/assemble_cvd 
12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 assemble_cvd failed: 
12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571  | device/google/cuttlefish/host/commands/assemble_cvd/assemble_cvd.cc:484
12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571  | Result<int> cuttlefish::AssembleCvdMain(int, char **)
12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571  v CF_EXPECT(tty == 0)
12-22 15:53:35.219 665720 665720 E assemble_cvd: assemble_cvd.cc:571 stdin was a tty, expected to be passed the output of a previous stage. Did you mean to run launch_cvd?
Aborted

The log reminds user to invoke launch_cvd instead of the assemble_cvd command. But this doesn’t mean we really cannot run assemble_cvd command directly. Alternatively, we can save what’s the command and arguments into a shell script (we named it as test.sh):

HOME=$PWD ./bin/assemble_cvd --group_id=cvd_1 --report_anonymous_usage_stats=y --vm_manager=qemu_cli --webrtc_device_id=cvd_1-1 --gpu_mode=drm_virgl

Then we can use the to launch the shell script with below command. This command can help use to close stdin and direct the stdout to the pipe :

true | sh test.sh 2>&1 | cat

Using run_cvd

Same as the assemble_cvd command, it prevents to directly run the run_cvd command. If we want to directly execute it, we can use the command:

$ true | HOME=$PWD ./bin/run_cvd 2>&1 | cat

run_cvd manages backend programs internally for emulating virtual devices (e.g. for WiFi, secure devices, etc). For the rough understanding of the run_cvd’s internal, you could refer to below diagram:

QEMU command

After the configurations for the virtual device’s emulation, it launches VMM (QEMU or Crosvm) to kick off virtual machine instance. Below is the compelete command parameters for the QEMU command:

qemu-system-x86_64
  -name guest=cvd-1,debug-threads=on
  -machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off
  -m size=4096M,maxmem=4102M,slots=2
  -overcommit mem-lock=off
  -smp 2,cores=2,threads=1
  -uuid 699acfc4-c8c4-11e7-882b-5065f31dc101
  -no-user-config
  -nodefaults
  -no-shutdown
  -rtc base=utc
  -boot strict=on
  -chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-1/internal/qemu_monitor.sock,server=on,wait=off
  -mon chardev=charmonitor,id=monitor,mode=control
  -display egl-headless
  -vnc 127.0.0.1:544
  -device pcie-pci-bridge,id=hvc-bridge,addr=01.2
  -device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280
  -chardev file,id=serial0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on
  -serial chardev:serial0
  -chardev file,id=hvc0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01
  -device virtconsole,bus=virtio-serial0.0,chardev=hvc0
  -chardev null,id=hvc1
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial1,bus=hvc-bridge,addr=02
  -device virtconsole,bus=virtio-serial1.0,chardev=hvc1
  -chardev file,id=hvc2,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/logcat-pipe,append=on
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial2,bus=hvc-bridge,addr=03
  -device virtconsole,bus=virtio-serial2.0,chardev=hvc2
  -chardev pipe,id=hvc3,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/keymaster_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial3,bus=hvc-bridge,addr=04
  -device virtconsole,bus=virtio-serial3.0,chardev=hvc3
  -chardev pipe,id=hvc4,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/gatekeeper_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial4,bus=hvc-bridge,addr=05
  -device virtconsole,bus=virtio-serial4.0,chardev=hvc4
  -chardev pipe,id=hvc5,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/bt_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial5,bus=hvc-bridge,addr=06
  -device virtconsole,bus=virtio-serial5.0,chardev=hvc5
  -chardev pipe,id=hvc6,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/gnsshvc_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial6,bus=hvc-bridge,addr=07
  -device virtconsole,bus=virtio-serial6.0,chardev=hvc6
  -chardev pipe,id=hvc7,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/locationhvc_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial7,bus=hvc-bridge,addr=08
  -device virtconsole,bus=virtio-serial7.0,chardev=hvc7
  -chardev null,id=hvc8
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial8,bus=hvc-bridge,addr=09
  -device virtconsole,bus=virtio-serial8.0,chardev=hvc8
  -chardev pipe,id=hvc9,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/uwb_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial9,bus=hvc-bridge,addr=0a
  -device virtconsole,bus=virtio-serial9.0,chardev=hvc9
  -chardev pipe,id=hvc10,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/oemlock_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial10,bus=hvc-bridge,addr=0b
  -device virtconsole,bus=virtio-serial10.0,chardev=hvc10
  -chardev pipe,id=hvc11,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/keymint_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial11,bus=hvc-bridge,addr=0c
  -device virtconsole,bus=virtio-serial11.0,chardev=hvc11
  -chardev pipe,id=hvc12,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/nfc_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial12,bus=hvc-bridge,addr=0d
  -device virtconsole,bus=virtio-serial12.0,chardev=hvc12
  -chardev null,id=hvc13
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial13,bus=hvc-bridge,addr=0e
  -device virtconsole,bus=virtio-serial13.0,chardev=hvc13
  -chardev null,id=hvc14
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial14,bus=hvc-bridge,addr=0f
  -device virtconsole,bus=virtio-serial14.0,chardev=hvc14
  -chardev null,id=hvc15
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial15,bus=hvc-bridge,addr=10
  -device virtconsole,bus=virtio-serial15.0,chardev=hvc15
  -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/overlay.img,if=none,id=drive-virtio-disk0,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
  -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/persistent_composite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1
  -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2
  -object memory-backend-file,id=objpmem0,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/pstore,size=2097152
  -device nvdimm,memdev=objpmem0,id=ramoops
  -object memory-backend-file,id=objpmem1,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/access-kregistry,size=2097152
  -device virtio-pmem-pci,disable-legacy=on,memdev=objpmem1,id=pmem0
  -object memory-backend-file,id=objpmem2,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/hwcomposer-pmem,size=2097152
  -device virtio-pmem-pci,disable-legacy=on,memdev=objpmem2,id=pmem1
  -object rng-random,id=objrng0,filename=/dev/urandom
  -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000
  -device virtio-mouse-pci,disable-legacy=on
  -device virtio-keyboard-pci,disable-legacy=on
  -device virtio-keyboard-pci,disable-legacy=on
  -device virtio-balloon-pci-non-transitional,id=balloon0
  -netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no
  -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no
  -device virtio-net-pci-non-transitional,netdev=hostnet0,id=net0,mac=00:1a:11:e0:cf:00
  -device virtio-net-pci-non-transitional,netdev=hostnet1,id=net1,mac=00:1a:11:e1:cf:00
  -cpu host
  -msg timestamp=on
  -device vhost-vsock-pci-non-transitional,guest-cid=3
  -device AC97,audiodev=audio_none
  -audiodev driver=none,id=audio_none
  -device qemu-xhci,id=xhci
  -bios /home/leoy/Dev2/aosp_cuttlefish_test/cf/etc/bootloader_x86_64/bootloader.qemu

Let’s look into these parameters:
-name guest=cvd-1,debug-threads=on : Set the name of the guest as cvd-1 and enable debug-threads so individual threads have dedicated names.

-machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off : a standard PC architecture and enables NVDIMM support, enables KVM accelerator, doesn’t include guest memory in a core dump, disables USB support.

-m size=4096M,maxmem=4102M,slots=2 : Sets guest startup RAM size to 4096MB. Set amount of hotpluggable memory slots as 2 and maximum amount of memory 4102MB.

-overcommit mem-lock=off : Disable memory locking (so can be swapped to swap area).

-uuid 699acfc4-c8c4-11e7-882b-5065f31dc101 : Set virtual system UUID.

-no-user-config : Don’t load any of the user-provided config files on sysconfdir.

-nodefaults : Don’t create default devices.

-no-shutdown : Don’t exit QEMU on guest shutdown.

-smp 2,cores=2,threads=1 : 2 virtual CPUs, and each CPU is part of a 2-core socket with one thread per core.

-rtc base=utc : virtual RTC is initialized with the current UTC time.

-chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-/internal/qemu_monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control : establish a socket-based character device for monitoring and control QEMU instance.

-display egl-headless -vnc 127.0.0.1:544 : Use EGL in a headless mode for graphics rendering. Additionally, a VNC server is enabled on localhost and port 544, allowing remote access to the virtual machine's display. ('-display sdl` is used by Xen)

-device pcie-pci-bridge,id=hvc-bridge,addr=01.2 : Emulate a PCIe to PCI bus with PCI address "01.2". (Needed by Xen)

-device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280 : Enable virgl GPU mode (host rendering). (Needed by Xen)

-chardev file,id=serial0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -serial chardev:serial0 : Create virtual serial device and bind it to the local pipe kernel-log-pipe.

-chardev file,id=hvc0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01 -device virtconsole,bus=virtio-serial0.0,chardev=hvc0 : Create a hypervisor virtual console and attached to local pipe.

Cuttlefish creates HVC 0~15. Except HVC 14 and 15 are not used, other HVCs are bound with the pipes on the host and works as communication channel with the backend program on host. (HVC0 is needed by Xen)

- /dev/hvc0 = kernel console
- /dev/hvc1 = serial console
- /dev/hvc2 = serial logging
- /dev/hvc3 = keymaster
- /dev/hvc4 = gatekeeper
- /dev/hvc5 = bt
- /dev/hvc6 = gnss
- /dev/hvc7 = location
- /dev/hvc8 = confirmationui
- /dev/hvc9 = uwb
- /dev/hvc10 = oemlock
- /dev/hvc11 = keymint
- /dev/hvc12 = NFC
- /dev/hvc13 = sensors

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/overlay.img,if=none,id=drive-virtio-disk0,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 : Enable virtual block device for the Android overlay image, which contains boot images, meta images, super.img and userdata.img, etc. (Needed by Xen)

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/persistent_composite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1: Enable virtual block device for the Android persistent image, which contains uboot enviriable image, bootconfig (for Android configurations), etc. (Needed by Xen)

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2 : Enable virtual block device for the Android SD image. (Needed by Xen)

-object memory-backend-file,id=objpmem0,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/pstore,size=2097152 -device nvdimm,memdev=objpmem0,id=ramoops : Enable memory object and store the data into file, which is used for kernel ramoops. (Needed by Xen)

-object memory-backend-file,id=objpmem2,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/hwcomposer-pmem,size=2097152 -device virtio-pmem-pci,disable-legacy=on,memdev=objpmem2,id=pmem1 : Enable memory object for hwcomposer.

-object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000 : Enable random device (Needed by Xen)

-device virtio-mouse-pci,disable-legacy=on : Enable virtual mouse (Needed by Xen)
-device virtio-keyboard-pci,disable-legacy=on : Enable virtual keyboard (Needed by Xen)

-device virtio-balloon-pci-non-transitional,id=balloon0 : Enable balloon for dynamically allocation memory size.

-netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no -device virtio-net-pci-non-transitional,netdev=hostnet0,id=net0,mac=00:1a:11:e0:cf:00 -device virtio-net-pci-non-transitional,netdev=hostnet1,id=net1,mac=00:1a:11:e1:cf:00 : Enable virtual network cards (Needed by Xen)

-device AC97,audiodev=audio_none -audiodev driver=none,id=audio_none : Enable sound device.

-bios /home/leoy/Dev2/aosp_cuttlefish_test/cf/etc/bootloader_x86_64/bootloader.qemu : Enable bootloader (U-boot). (Needed by Xen)

Suggested QEMU command for Xen virtual machine

Below is the suggested command for running Android image on Xen virtual machine. It has not been tested yet. Once we have hardware is avaliable for the testing, we might update below arguments.

$ /usr/bin/qemu-system-aarch64
  -xen-domid 1
  -no-shutdown
  -chardev socket,id=xen-cmd,path=/var/run/xen/qmp-libxl-1,server=on,wait=off
  -mon chardev=xen-cmd,mode=control
  -nodefaults
  -no-user-config
  -xen-attach
  -name aaos-vm
  -vnc none
  -display sdl,gl=on
  -device pcie-pci-bridge,id=hvc-bridge,addr=01.2
  -device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280
  -device virtio-mouse-pci,disable-legacy=on,iommu_platform=on
  -device virtio-keyboard-pci,disable-legacy=on,iommu_platform=on
  -smp 4,maxcpus=4
  -machine xenpvh
  -m 6144
  -chardev file,id=serial0,path=/path/to/kernel-log-pipe,append=on
  -serial chardev:serial0
  -chardev file,id=hvc0,path=/path/to/kernel-log-pipe,append=on
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01
  -device virtconsole,bus=virtio-serial0.0,chardev=hvc0
  -drive file=/path/to/overlay.img,if=none,id=drive-virtio-disk0,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
  -drive file=/path/to/persistent_composite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1
  -drive file=/path/to/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2
  -object memory-backend-file,id=objpmem0,share=on,mem-path=/path/to/pstore,size=2097152
  -device nvdimm,memdev=objpmem0,id=ramoops
  -object rng-random,id=objrng0,filename=/dev/urandom
  -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000
  -netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no
  -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no
  -device virtio-net-pci-non-transitional,netdev=hostnet0,id=net0,mac=00:1a:11:e0:cf:00
  -device virtio-net-pci-non-transitional,netdev=hostnet1,id=net1,mac=00:1a:11:e1:cf:00
  -bios /path/to/u-boot.bin
  • No labels