Some unsorted collection of small but useful notes about AVA’s hardware, firmware and debugging options for Xen and Linux kernel compiled as part of TRS targeting AVA Developer Platform 32 machine.
Xen
Early printk via UART in Xen
Such options are required in .config file of xen if compiled as standalone binary:
CONFIG_VERBOSE_DEBUG=y CONFIG_EARLY_UART_CHOICE_PL011=y CONFIG_EARLY_UART_PL011=y CONFIG_EARLY_PRINTK=y CONFIG_EARLY_UART_BASE_ADDRESS=0x100002600000 CONFIG_EARLY_UART_PL011_BAUD_RATE=0 CONFIG_EARLY_UART_PL011_MMIO32=y CONFIG_EARLY_PRINTK_INC="debug-pl011.inc"
In case of TRS these config options can be added in one of the *.cfg files under meta-ledge-secure/meta-ledge-secure/dynamic-layers/meta-virtualization/recipes-extended/xen/files/
Increasing verbosity of logs for Xen
Xen cmdline’s parameters loglvl=... and guest_loglvl=... can be switched to all.
For TRS this can be achieved by editing the cmdline in file meta-ledge-secure/meta-ledge-secure/dynamic-layers/meta-virtualization/recipes-extended/xen/files/xen.cfg.in :
options=noreboot dom0_mem=${DOM0_MEMORY_SIZE} bootscrub=0 iommu=on loglvl=all guest_loglvl=all
Logs location (xen, qemu)
On the booted machine the useful qemu and xen logs will be located under /var/log/xen/*
PCIe
PCIe AER events flood
It is observed that some PCIe cards might have issues negotiating power management policies and may produce AER events/messages under old UEFI firmware (1.x.y.z-something). The latest firmwares does not have this issue. There is an option in BIOS to handle such messages in firmware instead but it is untested.
However, the PCIe ASPM can be disabled via Linux kernel cmdline:
pcie_aspm=off
The AER messages also could be suppressed using:
pci=noaer
The https://docs.ipi.wiki/com-hpc/ava/firmware_A2Versions.html suggests that AER events can be caused by incorrect payload size which can be re-configured in BIOS.
PCIe speed
By default BIOS/UEFI sets the PCIe speeds to Gen3. There are options in BIOS/UEFI interface to enable Gen4 speed. There is no information if anyone tested this but it should work in theory. In case of potential performance impact, the Gen4 speed could be tested.
PCIe-to-NVMe adapter card
Such card was tested and it worked. I was able to use 3 NVMe disks at some point when 3rd NVMe was connected via pcie-to-nvme card.
SATA ports on motherboard
It looks like they do not work and probably there is no SATA controller on the carrier board.