Page Comparison

...

The launch_cvd command rus two commands assemble_cvd and run_cvd in its child processes. If both commands work as expected, the Android can launch successfully on Cuttlefish machine. The assemble_cvd command generates root file system images with overlay / qcow2 overlay format, then the run_cvd command invokes the virtual machine manager (either Crosvm or QEMU) to boot the virtual machine with the generated images on virtual machine. The diagram below shows the Cuttlefish flow.

...

Using assemble_cvd

Let’s we assume we work on TRS and want to run Android Automotive OS on Xen virtual machine, in this case, it’s likely we cannot directly use launch_cvd command for launching Xen virtual machine. Then Therefore, we need to use assumble_cvd command to help us to generate file system images, and afterwards we can execute QEMU command to launch Xen virtual machine. This is why we need to understand how to use assumble_cvd command.

...

HOME=$PWD ./bin/assemble_cvd --group_id=cvd_1 --report_anonymous_usage_stats=y --vm_manager=qemu_cli --webrtc_device_id=cvd_1-1 --gpu_mode=drm_virgl

Then we can use the to launch the shell script with below command. This command can help use to close closes stdin and direct directs the stdout to the pipe for workaround the failure mentioned above:

true | sh test.sh 2>&1 | cat

Using run_cvd

Same as the Note, as you have noticed that we can pass most launch_cvd’s parameters to assemble_cvd command , it prevents to directly run the run_cvd command. If we want to directly execute it, we can use the commandbut it is not always true that these two commands can support exact same parameters. If you want to check what’s the parameters passed to assemble_cvd , you can firstly run launch_cvd and after assumble_cvd is launched in the sub process, you can check the parameters for the assumble_cvd process:

$ trueps -e | HOME=$PWD ./bin/rungrep assemble_cvd
2>&1 | cat

run_cvd manages backend programs internally for emulating virtual devices (e.g. for WiFi, secure devices, etc). For the rough understanding of the run_cvd’s internal, you could refer to below diagram:

...

QEMU command

After the configurations for the virtual device’s emulation, it launches VMM (QEMU or Crosvm) to kick off virtual machine instance. Below is the compelete command parameters for the QEMU command:

qemu-system-x86_64
  -name guest=cvd-1,debug-threads=on
  -machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off
  -m size=4096M,maxmem=4102M,slots=2
  -overcommit mem-lock=off
  -smp 2,cores=2,threads=1
  -uuid 699acfc4-c8c4-11e7-882b-5065f31dc101
  -no-user-config
  -nodefaults
  -no-shutdown
  -rtc base=utc
  -boot strict=on
  -chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-1/internal/qemu_monitor.sock,server=on,wait=off
  -mon chardev=charmonitor,id=monitor,mode=control
  -display egl-headless
  -vnc 127.0.0.1:544
  -device pcie-pci-bridge,id=hvc-bridge,addr=01.2
  -device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280
  -chardev file,id=serial0 673381 ?        00:00:02 assemble_cvd
$  cat /proc/673381/cmdline | sed -e "s/\x00/ /g"; echo
/home/leoy/Dev2/aosp_cuttlefish_test/cf/bin/assemble_cvd --group_id=cvd_1 --report_anonymous_usage_stats=y --vm_manager=qemu_cli --webrtc_device_id=cvd_1-1 

The Android images

Using above command, the two prebuilt Android images are shared in:
cuttlefish_drm_virgl_venus.tgz : Android automotive images with GPU mode drm_virgl for venus
cuttlefish_gfxstream.tgz : Android automotive images with GPU mode gfxstream for vulkan.

Download the package and uncompress it in the folder /home/leoy/Dev2/aosp_cuttlefish_test/cf to ensure all linkages work well.

Using run_cvd

Same as the assemble_cvd command, we are prevent to directly run the run_cvd command. If we want to execute it, we can use the command:

$ true | HOME=$PWD ./bin/run_cvd 2>&1 | cat

run_cvd manages dozen of backend programs internally for emulating virtual devices (e.g. for WiFi, secure devices, etc). For the brief understanding the run_cvd’s internal, you could refer to below diagram:

...

QEMU command

After the configurations for the virtual device’s emulation, it launches VMM (QEMU or Crosvm) to kick off virtual machine instance. Below is the compelete command parameters for the QEMU command:

qemu-system-x86_64
  -name guest=cvd-1,debug-threads=on
  -machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off
  -m size=4096M,maxmem=4102M,slots=2
  -overcommit mem-lock=off
  -smp 2,cores=2,threads=1
  -uuid 699acfc4-c8c4-11e7-882b-5065f31dc101
  -no-user-config
  -nodefaults
  -no-shutdown
  -rtc base=utc
  -boot strict=on
  -chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-1/internal/qemu_monitor.sock,server=on,wait=off
  -mon chardev=charmonitor,id=monitor,mode=control
  -display egl-headless
  -vnc 127.0.0.1:544
  -device pcie-pci-bridge,id=hvc-bridge,addr=01.2
  -device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280
  -chardev file,id=serial0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on
  -serial chardev:serial0
  -chardev file,id=hvc0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01
  -device virtconsole,bus=virtio-serial0.0,chardev=hvc0
  -chardev null,id=hvc1
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial1,bus=hvc-bridge,addr=02
  -device virtconsole,bus=virtio-serial1.0,chardev=hvc1
  -chardev file,id=hvc2,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/logcat-pipe,append=on
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial2,bus=hvc-bridge,addr=03
  -device virtconsole,bus=virtio-serial2.0,chardev=hvc2
  -chardev pipe,id=hvc3,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/keymaster_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial3,bus=hvc-bridge,addr=04
  -device virtconsole,bus=virtio-serial3.0,chardev=hvc3
  -chardev pipe,id=hvc4,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/gatekeeper_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial4,bus=hvc-bridge,addr=05
  -device virtconsole,bus=virtio-serial4.0,chardev=hvc4
  -chardev pipe,id=hvc5,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/bt_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial5,bus=hvc-bridge,addr=06
  -device virtconsole,bus=virtio-serial5.0,chardev=hvc5
  -chardev pipe,id=hvc6,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/gnsshvc_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial6,bus=hvc-bridge,addr=07
  -device virtconsole,bus=virtio-serial6.0,chardev=hvc6
  -chardev pipe,id=hvc7,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/locationhvc_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial7,bus=hvc-bridge,addr=08
  -device virtconsole,bus=virtio-serial7.0,chardev=hvc7
  -chardev null,id=hvc8
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial8,bus=hvc-bridge,addr=09
  -device virtconsole,bus=virtio-serial8.0,chardev=hvc8
  -chardev pipe,id=hvc9,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/uwb_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial9,bus=hvc-bridge,addr=0a
  -device virtconsole,bus=virtio-serial9.0,chardev=hvc9
  -chardev pipe,id=hvc10,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/oemlock_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial10,bus=hvc-bridge,addr=0b
  -device virtconsole,bus=virtio-serial10.0,chardev=hvc10
  -chardev pipe,id=hvc11,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on
  -serial chardev:serial0keymint_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial11,bus=hvc-bridge,addr=0c
  -device virtconsole,bus=virtio-serial11.0,chardev=hvc11
  -chardev filepipe,id=hvc0hvc12,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=onnfc_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial12,bus=hvc-bridge,addr=0d
  -device virtconsole,bus=virtio-serial12.0,chardev=hvc12
  -chardev null,id=hvc13
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0serial13,bus=hvc-bridge,addr=010e
  -device virtconsole,bus=virtio-serial0serial13.0,chardev=hvc0hvc13
  -chardev null,id=hvc1hvc14
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial1serial14,bus=hvc-bridge,addr=020f
  -device virtconsole,bus=virtio-serial1serial14.0,chardev=hvc1hvc14
  -chardev filenull,id=hvc2,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/logcat-pipe,append=onhvc15
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial2serial15,bus=hvc-bridge,addr=0310
  -device virtconsole,bus=virtio-serial2serial15.0,chardev=hvc2hvc15
  -chardev pipe,id=hvc3,pathdrive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/keymaster_fifo_vmoverlay.img,if=none,id=drive-virtio-disk0,aio=threads
  -device virtio-serialblk-pci-non-transitional,max_portsscsi=1off,iddrive=drive-virtio-serial3,bus=hvc-bridge,addr=04
  -device virtconsole,busdisk0,id=virtio-serial3.0disk0,chardevbootindex=hvc31
  -chardev pipe,id=hvc4,pathdrive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/gatekeeperpersistent_fifo_vmcomposite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads
  -device virtio-serialblk-pci-non-transitional,max_portsscsi=1off,iddrive=drive-virtio-serial4,bus=hvc-bridge,addr=05
  -device virtconsole,busdisk1,id=virtio-serial4.0,chardev=hvc4disk1
  -chardevdrive pipe,id=hvc5,path=file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/bt_fifo_vmsdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads
  -device virtio-serialblk-pci-non-transitional,max_ports=1,id=virtio-serial5,bus=hvc-bridge,addr=06,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2
  -object memory-backend-file,id=objpmem0,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/pstore,size=2097152
  -device virtconsolenvdimm,bus=virtio-serial5.0,chardev=hvc5memdev=objpmem0,id=ramoops
  -chardev pipeobject memory-backend-file,id=objpmem1,share=hvc6on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/gnsshvc_fifo_vmaccess-kregistry,size=2097152
  -device virtio-serialpmem-pci-non-transitional,max_ports=1,id=virtio-serial6,bus=hvc-bridge,addr=07,disable-legacy=on,memdev=objpmem1,id=pmem0
  -device virtconsole,bus=virtio-serial6.0,chardev=hvc6
  -chardev pipe,id=hvc7,object memory-backend-file,id=objpmem2,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/locationhvc_fifo_vmhwcomposer-pmem,size=2097152
  -device virtio-serialpmem-pci-non-transitional,max_ports=1,id=virtio-serial7,bus=hvc-bridge,addr=08,disable-legacy=on,memdev=objpmem2,id=pmem1
  -object rng-random,id=objrng0,filename=/dev/urandom
  -device virtconsole,bus=virtio-serial7.0,chardev=hvc7
  -chardev null,id=hvc8virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000
  -device virtio-serialmouse-pci-non-transitional,max_ports=1,id=virtio-serial8,bus=hvc-bridge,addr=09,disable-legacy=on
  -device virtconsole,bus=virtio-keyboard-serial8.0,chardev=hvc8pci,disable-legacy=on
  -chardev pipe,id=hvc9,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/uwb_fifo_vmdevice virtio-keyboard-pci,disable-legacy=on
  -device virtio-serialballoon-pci-non-transitional,max_ports=1,id=virtio-serial9,bus=hvc-bridge,addr=0aid=balloon0
  -netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no
  -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no
  -device virtconsole,bus=virtio-serial9.0,chardev=hvc9
  -chardev pipenet-pci-non-transitional,netdev=hostnet0,id=hvc10net0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/oemlock_fifo_vmmac=00:1a:11:e0:cf:00
  -device virtio-serialnet-pci-non-transitional,max_portsnetdev=1hostnet1,id=virtio-serial10net1,bus=hvc-bridge,addr=0bmac=00:1a:11:e1:cf:00
  -device virtconsole,bus=virtio-serial10.0,chardev=hvc10
  -chardev pipe,id=hvc11,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/keymint_fifo_vmcpu host
  -msg timestamp=on
  -device virtiovhost-serialvsock-pci-non-transitional,max_ports=1,id=virtio-serial11,bus=hvc-bridge,addr=0cguest-cid=3
  -device AC97,audiodev=audio_none
  -audiodev driver=none,id=audio_none
  -device virtconsole,bus=virtio-serial11.0,chardev=hvc11qemu-xhci,id=xhci
  -chardev pipe,id=hvc12,path=bios /home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/nfc_fifo_vm
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial12,bus=hvc-bridge,addr=0d
  -device virtconsole,bus=virtio-serial12.0,chardev=hvc12
  -chardev null,id=hvc13
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial13,bus=hvc-bridge,addr=0e
  -device virtconsole,bus=virtio-serial13.0,chardev=hvc13
  -chardev null,id=hvc14
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial14,bus=hvc-bridge,addr=0f
  -device virtconsole,bus=virtio-serial14.0,chardev=hvc14
  -chardev null,id=hvc15
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial15,bus=hvc-bridge,addr=10
  -device virtconsole,bus=virtio-serial15.0,chardev=hvc15
  -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/overlay.img,if=none,id=drive-virtio-disk0,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
  -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/persistent_composite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1
  -drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2
  -object memory-backend-file,id=objpmem0,share=on,mem-etc/bootloader_x86_64/bootloader.qemu

Let’s look into these parameters:
-name guest=cvd-1,debug-threads=on : Set the name of the guest as cvd-1 and enable debug-threads so individual threads have dedicated names.

-machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off : a standard PC architecture and enables NVDIMM support, enables KVM accelerator, doesn’t include guest memory in a core dump, disables USB support.

-m size=4096M,maxmem=4102M,slots=2 : Sets guest startup RAM size to 4096MB. Set amount of hotpluggable memory slots as 2 and maximum amount of memory 4102MB.

-overcommit mem-lock=off : Disable memory locking (so can be swapped to swap area).

-uuid 699acfc4-c8c4-11e7-882b-5065f31dc101 : Set virtual system UUID.

-no-user-config : Don’t load any of the user-provided config files on sysconfdir.

-nodefaults : Don’t create default devices.

-no-shutdown : Don’t exit QEMU on guest shutdown.

-smp 2,cores=2,threads=1 : 2 virtual CPUs, and each CPU is part of a 2-core socket with one thread per core.

-rtc base=utc : virtual RTC is initialized with the current UTC time.

-chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-/internal/qemu_monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control : establish a socket-based character device for monitoring and control QEMU instance.

-display egl-headless -vnc 127.0.0.1:544 : Use EGL in a headless mode for graphics rendering. Additionally, a VNC server is enabled on localhost and port 544, allowing remote access to the virtual machine's display. ('-display sdl` is used by Xen)

-device pcie-pci-bridge,id=hvc-bridge,addr=01.2 : Emulate a PCIe to PCI bus with PCI address "01.2". (Needed by Xen)

-device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280 : Enable virgl GPU mode (host rendering). (Needed by Xen)

-chardev file,id=serial0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/

...

internal/kernel-log-pipe,append=on -serial chardev:serial0 : Create virtual serial device and bind it to the local pipe kernel-log-pipe.

-chardev file,id=

...

hvc0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/

...

kernel-log-

...

pipe,

...

append=on -device

...

virtio-

...

serial-pci

...

-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01 -device virtconsole,bus=virtio-serial0.0,chardev=hvc0 : Create a hypervisor virtual console and attached to local pipe.

Cuttlefish creates HVC 0~15. Except HVC 14 and 15 are not used, other HVCs are bound with the pipes on the host and works as communication channel with the backend program on host. (HVC0 is needed by Xen)

- /dev/hvc0 = kernel console
- /dev/hvc1 = serial console
- /dev/hvc2 = serial logging
- /dev/hvc3 = keymaster
- /dev/hvc4 = gatekeeper
- /dev/hvc5 = bt
- /dev/hvc6 = gnss
- /dev/hvc7 = location
- /dev/hvc8 = confirmationui
- /dev/hvc9 = uwb
- /dev/hvc10 = oemlock
- /dev/hvc11 = keymint
- /dev/hvc12 = NFC
- /dev/hvc13 = sensors

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/overlay.img,if=none,id=drive-virtio-disk0,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 : Enable virtual block device for the Android overlay image, which contains boot images, meta images, super.img and userdata.img, etc. (Needed by Xen)

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/

...

Let’s look into these parameters:
-name guest=cvd-1,debug-threads=on : Set the name of the guest as cvd-1 and enable debug-threads so individual threads have dedicated names.

-machine pc,nvdimm=on,accel=kvm,usb=off,dump-guest-core=off : a standard PC architecture and enables NVDIMM support, enables KVM accelerator, doesn’t include guest memory in a core dump, disables USB support.

-m size=4096M,maxmem=4102M,slots=2 : Sets guest startup RAM size to 4096MB. Set amount of hotpluggable memory slots as 2 and maximum amount of memory 4102MB.

-overcommit mem-lock=off : Disable memory locking (so can be swapped to swap area).

-uuid 699acfc4-c8c4-11e7-882b-5065f31dc101 : Set virtual system UUID.

-no-user-config : Don’t load any of the user-provided config files on sysconfdir.

-nodefaults : Don’t create default devices.

-no-shutdown : Don’t exit QEMU on guest shutdown.

-smp 2,cores=2,threads=1 : 2 virtual CPUs, and each CPU is part of a 2-core socket with one thread per core.

-rtc base=utc : virtual RTC is initialized with the current UTC time.

-chardev socket,id=charmonitor,path=/tmp/cf_avd_1000/cvd-/internal/qemu_monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control : establish a socket-based character device for monitoring and control QEMU instance.

-display egl-headless -vnc 127.0.0.1:544 : Use EGL in a headless mode for graphics rendering. Additionally, a VNC server is enabled on localhost and port 544, allowing remote access to the virtual machine's display. ('-display sdl` is used by Xen)

-device pcie-pci-bridge,id=hvc-bridge,addr=01.2 : Emulate a PCIe to PCI bus with PCI address "01.2". cuttlefish/instances/cvd-1/persistent_composite_overlay.img,if=none,id=drive-virtio-disk1,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1: Enable virtual block device for the Android persistent image, which contains uboot enviriable image, bootconfig (for Android configurations), etc. (Needed by Xen)

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2 : Enable virtual block device for the Android SD image. (Needed by Xen)

-object memory-backend-file,id=objpmem0,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/pstore,size=2097152 -device nvdimm,memdev=objpmem0,id=ramoops : Enable memory object and store the data into file, which is used for kernel ramoops. (Needed by Xen)

-object memory-backend-file,id=objpmem2,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/hwcomposer-pmem,size=2097152 -device virtio-pmem-pci,disable-legacy=on,memdev=objpmem2,id=pmem1 : Enable memory object for hwcomposer.

-object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000 : Enable random device (Needed by Xen)

-device virtio-mouse-pci,disable-legacy=on : Enable virtual mouse (Needed by Xen)
-device virtio-keyboard-pci,disable-legacy=on : Enable virtual keyboard (Needed by Xen)

-device virtio-gpuballoon-pci-glnon-pcitransitional,id=gpu0,xres=720,yres=1280 : Enable virgl GPU mode (host rendering). (Needed by Xen)

-chardev file,id=serial0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -serial chardev:serial0 : Create virtual serial device and bind it to the local pipe kernel-log-pipe.

-chardev file,id=hvc0,path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,append=on -device virtio-serialballoon0 : Enable balloon for dynamically allocation memory size.

-netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no -device virtio-net-pci-non-transitional,netdev=hostnet0,id=net0,mac=00:1a:11:e0:cf:00 -device virtio-net-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01 -device virtconsole,bus=virtio-serial0.0,chardev=hvc0 : Create a hypervisor virtual console and attached to local pipe.

Cuttlefish creates HVC 0~15. Except HVC 14 and 15 are not used, other HVCs are bound with the pipes on the host and works as communication channel with the backend program on host. (HVC0 is needed by Xen)

- /dev/hvc0 = kernel console
- /dev/hvc1 = serial console
- /dev/hvc2 = serial logging
- /dev/hvc3 = keymaster
- /dev/hvc4 = gatekeeper
- /dev/hvc5 = bt
- /dev/hvc6 = gnss
- /dev/hvc7 = location
- /dev/hvc8 = confirmationui
- /dev/hvc9 = uwb
- /dev/hvc10 = oemlock
- /dev/hvc11 = keymint
- /dev/hvc12 = NFC
- /dev/hvc13 = sensors

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/overlay.img,if=none,id=drive-virtio-disk0,aio=threads -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 : Enable virtual block device for the Android overlay image, which contains boot images, meta images, super.img and userdata.img, etc. (Needed by Xen)

-drive file=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/persistent_composite_,netdev=hostnet1,id=net1,mac=00:1a:11:e1:cf:00 : Enable virtual network cards (Needed by Xen)

-device AC97,audiodev=audio_none -audiodev driver=none,id=audio_none : Enable sound device.

-bios /home/leoy/Dev2/aosp_cuttlefish_test/cf/etc/bootloader_x86_64/bootloader.qemu : Enable bootloader (U-boot). (Needed by Xen)

Suggested QEMU command for Xen virtual machine

Below is the suggested command for running Android image on Xen virtual machine. It has not been tested yet. Once we have hardware is avaliable for the testing, we might update below arguments.

$ /usr/bin/qemu-system-aarch64
  -xen-domid 1
  -no-shutdown
  -chardev socket,id=xen-cmd,path=/var/run/xen/qmp-libxl-1,server=on,wait=off
  -mon chardev=xen-cmd,mode=control
  -nodefaults
  -no-user-config
  -xen-attach
  -name aaos-vm
  -vnc none
  -display sdl,gl=on
  -device pcie-pci-bridge,id=hvc-bridge,addr=01.2
  -device virtio-gpu-gl-pci,id=gpu0,xres=720,yres=1280
  -device virtio-mouse-pci,disable-legacy=on,iommu_platform=on
  -device virtio-keyboard-pci,disable-legacy=on,iommu_platform=on
  -smp 4,maxcpus=4
  -machine xenpvh
  -m 6144
  -chardev file,id=serial0,path=/path/to/kernel-log-pipe,append=on
  -serial chardev:serial0
  -chardev file,id=hvc0,path=/path/to/kernel-log-pipe,append=on
  -device virtio-serial-pci-non-transitional,max_ports=1,id=virtio-serial0,bus=hvc-bridge,addr=01
  -device virtconsole,bus=virtio-serial0.0,chardev=hvc0
  -drive file=/path/to/overlay.img,if=none,id=drive-virtio-

...

disk0,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-

...

disk0,id=virtio-

...

disk0,bootindex=1
  -drive file=/path/to/persistent_composite_overlay.img,if=none,id=drive-virtio-

...

disk1,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-

...

disk1,id=virtio-

...

-object memory-backend-file,id=objpmem0,share=on,mem-path=/home/leoy/Dev2/aosp_cuttlefish_test/cf/cuttlefish/instances/cvd-1/pstore,size=2097152 -device nvdimm,memdev=objpmem0,id=ramoops : Enable memory object and store the data into file, which is used for kernel ramoops. (Needed by Xen)

...

disk1
  -drive file=/path/to/sdcard_overlay.img,if=none,id=drive-virtio-disk2,aio=threads
  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk2,id=virtio-disk2
  -object memory-backend-file,id=

...

objpmem0,share=on,mem-path=/

...

path/

...

to/pstore,size=2097152
  -device

...

 nvdimm,memdev=

...

objpmem0,id=

...

ramoops
  -object rng-random,id=objrng0,filename=/dev/urandom
  -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max

...

-device virtio-mouse-pci,disable-legacy=on : Enable virtual mouse (Needed by Xen)
-device virtio-keyboard-pci,disable-legacy=on : Enable virtual keyboard (Needed by Xen)

-device virtio-balloon-pci-non-transitional,id=balloon0 : Enable balloon for dynamically allocation memory size.

...

-bytes=1024,period=2000
  -netdev tap,id=hostnet0,ifname=cvd-mtap-01,script=no,downscript=no
  -netdev tap,id=hostnet1,ifname=cvd-etap-01,script=no,downscript=no
  -device virtio-net-pci-non-transitional,netdev=hostnet0,id=net0,mac=00:1a:11:e0:cf:00
  -device virtio-net-pci

...

-device AC97,audiodev=audio_none -audiodev driver=none,id=audio_none : Enable sound device.

...

-non-transitional,netdev=hostnet1,id=net1,mac=00:1a:11:e1:cf:00
  -bios /path/to/u-boot.bin