Debugging QEMU with TRS builds

While TRS makes heavy use of QEMU to run images withing the Yocto build system there are sometimes times when you want to control the image being used so you can debug if the issue is with TRS or the QEMU emulation. To do this there are a few places you can hook into to make things easier:

Building TRS

Refer to the upstream upstream documentation but once you have synced and checked out the trees generally:

make TARGET=tsqemuarm64-secureboot meta-ts   # if you touch the firmware/machine config
make trs # to build the TRS images
make test # to run the tests

Using a different QEMU

The QEMU built (qemu-system-native) will usually be stripped and also not have any debug or local patches applied. The easiest way to hook in is just to patch poky/scripts/runqemu:

    def find_qemu(self):
        qemu_bin = "/home/alex/lsrc/qemu.git/builds/trs.debug/qemu-system-aarch64"
        # qemu_bin = os.path.join(self.bindir_native, self.qemu_system)
        logger.info("Checking QEMU @ : %s" % qemu_bin)

Watching the live log

If you want to follow the details of whats going on you wont see it in the make output until after the fact. You can however tail the job log which points to the latest run:

alex@draig:~/lsrc/tests/trs/build$ tail -f ./tmp_trs-qemuarm64/work/trs_qemuarm64-trs-linux/trs-image/1.0/temp/log.do_testimage

Getting an HMP connection

Having access to the HMP is useful for checking the guest state, stopping execution and invoking the gdbstub for more detailed analysis. You can use QB_OPT_APPEND to add something to the command line when invoking QEMU for testing. This can be found in trs-image.bb:

QB_OPT_APPEND += " -monitor telnet:127.0.0.1:4444,server,nowait"

# Due to rootfs encryption, boot to serial login prompt may
# take really long on a loaded machine
TEST_QEMUBOOT_TIMEOUT = "8000"

The -monitor line sets up a dump terminal you can netcat to and we extend the timeout to give us time to poke around. I’ve generally found running “make trs” is needed after tweaking these files.

The build system deletes work to keep disk usage down. You can exclude certain items by tweaking the RM_WORK_EXCLUDE_ITEMS variable:

# leave native sysroot around so we can use swtpm from there outside of bitbake builds
RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native linux-yocto optee-os"

Using a kernel with symbols

You have to patch the build to ensure you disable debug unfriendly things like ASLR as well as enable things like GDB script support. So for example add trs/meta-trs/recipies-kernel/linux/linux-yocto%.bbappend:

TRSFILESPATHS := "${THISDIR}/files:"

FILESEXTRAPATHS:prepend:qemuarm64 = "${TRSFILESPATHS}"
SRC_URI:append:qemuarm64 = " \
    file://debug.cfg \
    "

with a files directory containing

CONFIG_RANDOMIZE_BASE=n
CONFIG_RANDOMIZE_MODULE_REGION_FULL=n
CONFIG_RANDOMIZE_KSTACK_OFFSET=n
CONFIG_GDB_SCRIPTS=y
CONFIG_FRAME_POINTER=y

You can test the build with:

bitbake mc:trs-qemuarm64:linux-yocto

or enter a devshell to build the gdbstub support:

bitbake -c devshell linux-yocto

Other Notes

There are some notes on booting TRS directly from the command line with a minimal swtpm setup here: https://trs.readthedocs.io/en/latest/firmware/running/distro_installer.html#run-on-qemu-arm64

Joakim did a presentation on debugging OPTEE in QEMU here: https://docs.google.com/presentation/d/1NEBZPrYqnjV55NOkJp_XQ7nGlOwDRuFb7z6cGO0Iboo/edit#slide=id.gad5d1d64ec_0_168

Enable Arm Architecture in QEMU