Raspberry Pi Linux kernel debugging with OpenOCD

Prerequisite: OpenOCD development environment

Hardware requirements:

• Raspberry Pi (Model 3 B or 2 B) development board
• 5V 2.5A power supply for Raspberry Pi
• High speed micro SD card of 16 GB or more
• SD card reader on development host machine
• Optional: Keyboard, Mouse, HDMI cable and LCD screen
• Bus Blaster v3 or any other JTAG probe with Arm 20-pin JTAG connector
• USB mini cable for Bus Blaster v3
• Jumper cables to connect Raspberry Pi GPIO JTAG pins with Arm 20-pin JTAG connector on Bus Blaster v3

SD card set up with Raspberry Pi OS image

We have selected Raspberry Pi OS which is a Debian based Linux distribution for the purpose of this tutorial. Raspberry Pi foundation has developed a GUI based tool called Raspberry Pi Imager for downloading and writing various OS images to SD cards. At the time of this writing, 64-bit Raspberry Pi OS image was not available for download via Imager utility, however we can manually download OS images and Imager utility will be able prepare a SD card using those custom images as well.

This blog post has details on how to use Raspberry Pi Imager utility. Also if your desired OS image is not available via Imager utility click here to download an appropriate image. For writing SD cards manually please follow the link for instructions.

• Click here to download Raspberry Pi OS image used for Raspberry Pi 2 B (Arm v7) and Raspberry Pi 3 B (AArch32)
• Click here to download Raspberry Pi OS image used for Raspberry Pi 3 B (AArch64)

Once the SD card is prepared we need to insert it into relevant Raspberry Pi board and boot it up once where it will resize partition on startup and we will be able to verify that OS installation was successful.

Build and install custom Linux kernel for debugging

Once we have successfully prepared a working SD card for our Raspberry Pi, the next step is to build Linux kernel with debug information and install custom kernel image on to SD card for JTAG debugging via OpenOCD.

Install dependencies

sudo apt install gcc g++ gcc-arm-linux-gnueabihf gcc-aarch64-linux-gnu git gdb-multiarch bison flex libncurses-dev libssl-dev

Download Raspberry Pi Linux kernel sources

git clone --depth=1  https://github.com/raspberrypi/linux.git

We are going to build the default Linux kernel version however you may select other versions by switching to the appropriate GIT branch in the source tree downloaded above.

# Change to linux source directory
cd linux

# Clean up source directory before build
make mrproper

# Create following directories inside Linux source directory
mkdir mnt
mkdir mnt/fat32
mkdir mnt/ext4

Insert SD card containing Raspberry Pi OS image installed above into card reader  on the host machine. Run lsblk before mounting to check sdcard partition names.

sudo mount /dev/mmcblk0p1 mnt/fat32
sudo mount /dev/mmcblk0p2 mnt/ext4

Linux Kernel build for Raspberry Pi 2 B (Arm v7) or Raspberry Pi 3 B (AArch32)

# Import bcm2709_defconfig default kernel configuration
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig

# Initiate menu config to enable debug symbols
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig

From menuconfig tool select > Kernel hacking > Compile-time checks and compiler options

• Enable Compile the kernel with debug info
• Enable Generate dwarf4 debuginfo
• Enable Provide GDB scripts for kernel debugging

# Save config, exit menuconfig tool and initiate kernel build
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-

# Install the modules into Raspberry Pi OS root file system mounted from SD card
sudo make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=mnt/ext4 modules_install

# Copy newly built 32-bit custom Linux kernel and device tree blobs onto the SD card
KERNEL=custom_kernel7l
sudo cp arch/arm/boot/zImage mnt/fat32/$KERNEL.img
sudo cp arch/arm/boot/dts/*.dtb mnt/fat32/
sudo cp arch/arm/boot/dts/overlays/*.dtb* mnt/fat32/overlays/
sudo cp arch/arm/boot/dts/overlays/README mnt/fat32/overlays/

Linux Kernel build for Raspberry Pi 3 B 64-bit (AArch64)

# Import bcmrpi3_defconfig default kernel configuration
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig

# Initiate menu config to enable debug symbols
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig

From menuconfig tool select > Kernel hacking > Compile-time checks and compiler options

• Enable Compile the kernel with debug info
• Enable Generate dwarf4 debuginfo
• Enable Provide GDB scripts for kernel debugging

# Save config, exit menuconfig tool and initiate kernel build
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

# Install the modules into Raspberry Pi OS root file system mounted from SD card
sudo make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=mnt/ext4 modules_install

# Copy newly built 64-bit custom Linux kernel and device tree blobs onto the SD card
KERNEL=custom_kernel8
sudo cp arch/arm64/boot/Image mnt/fat32/$KERNEL.img
sudo cp arch/arm64/boot/dts/broadcom/*.dtb mnt/fat32/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* mnt/fat32/overlays/
sudo cp arch/arm64/boot/dts/overlays/README mnt/fat32/overlays/

Enable JTAG debugging on Raspberry Pi

• JTAG capability is exposed through GPIO pins on both Raspberry Pi 2 and Raspberry Pi 3. GPIO on Raspberry Pi can be configured by writing to the configuration file config.txt. For further details about Raspberry Pi configuration file click here. 
• All Raspberry Pi GPIO pins have at least two alternative functions. Setting enable_jtag_gpio=1 in config.txt selects Alt4 mode for GPIO22 to GPIO27, and sets up SoC’s for JTAG debug. For details on Raspberry Pi GPIO look into BCM2835 SoC reference manual.
• Also click here reference about Raspberry Pi GPIO configuration for JTAG debug. 

Edit mnt/fat32/config.txt and add following lines:

# For 32-bit kernel image
kernel=custom_kernel7l.img

# For 64-bit kernel image
kernel=custom_kernel8.img

# Enable JTAG GPIO configuration
enable_jtag_gpio=1

Kernel command line parameter nokaslr disabled KASLR. If the architecture that you are using enables KASLR by default then virtual addresses where the kernel image is mapped will be randomized and GDB may not be able to resolve kernel symbol addresses from the debug information of vmlinux. Also software breakpoints and writes to read only sections of kernel address space may not work unless we specifically turn off the read only mode. Kernel command line parameter rodata=off allows for this.

# Edit mnt/fat32/cmdline.txt and append following to kernel command line:
nokaslr rodata=off

# Unmount and remove SD card, you all set to boot custom Linux kernel on Raspberry Pi
sudo umount mnt/fat32
sudo umount mnt/ext4

Raspberry Pi Hardware set up for JTAG debugging:

In this article we are going to use the Bus Blaster v3 JTAG debugger which is based on FT2232H USB to UART/FIFO IC. However connection instructions described will work for any JTAG debugger which has a ARM 20 pin JTAG connector. In the previous section we have already configured Raspberry Pi for JTAG debug by selecting GPIO alternate function ALT4 on GPIO22 to GPIO27. We first need to figure out which pins on the Raspberry Pi GPIO header correspond to GPIO22 to GPIO27. Please keep in mind that there is no one to one mapping between GPIO numbers and GPIO header pinouts. 

• Click here for further details on Arm 20-pin JTAG header. 
• Click here for further details about Raspberry Pi GPIO header.

From the pictures above we can figure out the following pin mappings between 40 pin Raspberry Pi GPIO header and Arm 20 pin JTAG connector.

Now that we have figured out pin mappings we can connect Bus Blaster v3 with Raspberry Pi JTAG pins using jumper cables. Please be cautious and double check every jumper connection.

OpenOCD set up for Raspberry Pi JTAG debugging

To start Linux kernel JTAG debugging on Raspberry Pi using OpenOCD we need to spawn OpenOCD session with configuration file for the specific board. First check JTAG connections jumper, insert SD card into Raspberry Pi, power on the board and wait for a minute or so for the kernel to boot. Connect Bus Blaster v3 JTAG probe to host computer via USB mini cable.

Download OpenOCD configuration file for Raspberry Pi 2 or Raspberry Pi 3

Following line at the end of both configuration files enables SMP mode where single GDB will control all cores simultaneously. In case your use-case requires multiple GDB connections to each individual core just comment this line before running OpenOCD.

target smp $_TARGETNAME.0 $_TARGETNAME.1 $_TARGETNAME.2 $_TARGETNAME.3

# Spawn OpenOCD session with Raspberry Pi 3 configuration
./openocd-src/src/openocd -f openocd-src/tcl/interface/ftdi/dp_busblaster.cfg -f ./raspberry-pi3.cfg

# Spawn OpenOCD session with Raspberry Pi 2 configuration
./openocd-src/src/openocd -f openocd-src/tcl/interface/ftdi/dp_busblaster.cfg -f ./raspberry-pi2.cfg

GDB set up for debugging Linux kernel via OpenOCD

Now we can start gdb and connect to OpenOCD GDB stub and debug Linux kernel running on Raspberry Pi.

gdb-multiarch <path-to-raspberrypi-linux-sources>/vmlinux
(gdb) tar remote :3333
(gdb) b do_sys_open

Set a breakpoint on do_sys_open routine, when any process will try to open a file and the do_sys_open() breakpoint will trigger.