How to Reproduce 32 bit Arm Builds Without Hardware
This guide is mainly for people external to Arm and Linaro. There are other methods to do this task, but this is going to be the most straightforward for 80% of people who don’t work primarily on Arm.
This can also be used for 64 bit builds by changing the chosen toolchain and qemu.
Building
With 32 bit support becoming increasingly rare, cross compilation and emulation is the most practical way for most people to do this. It also allows you to use the fast machine you already have rather than a low powered development board.
This guide will only cover Linux. Adapt as needed but remember that qemu-user doesn’t work everywhere, so either get a Linux VM then do this, or look at using qemu-system
to create an Arm VM.
You can get a toolchain from apt
or from Arm directly: https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads
Make sure you choose the correct host and target triples. If you want to target Linux (which you do 99% of the time) choose one with -linux
in the name.
I recommend getting Arm’s because it’ll have the latest feature support. You could also use clang but you’ll need a sysroot, which you get from the same place anyway.
Extract the toolchain and add it to your PATH:
$ export PATH=$(pwd)/arm-gnu-toolchain-13.2.Rel1-x86_64-arm-none-linux-gnueabihf/bin/:$PATH
Set up a symlink inside the toolchain so it can find ld.lld
. I've found that ld
runs out of memory most of the time.
Note that this symink is in the <triple>/bin
folder, not bin/
.
arm-gnu-toolchain-13.2.Rel1-x86_64-arm-none-linux-gnueabihf/arm-none-linux-gnueabihf/bin$ ln -s $(which ld.lld-11) ./ld.lld
(there is probably a gcc flag to fix this issue “properly” but we are not here to learn gcc flags)
Next, configure llvm using this script:
$ cat ../configure-arm.sh
set -ex
cd /work/open_source/build-llvm-arm
HOST_TRIPLE=arm-none-linux-gnueabihf
C_COMPILER=$HOST_TRIPLE-gcc
CXX_COMPILER=$HOST_TRIPLE-g++
cmake -G Ninja \
-DCMAKE_SYSTEM_NAME=Linux \
-DCMAKE_CROSSCOMPILING=1 \
-DCMAKE_C_COMPILER=$C_COMPILER \
-DCMAKE_CXX_COMPILER=$CXX_COMPILER \
-DLLVM_HOST_TRIPLE=$HOST_TRIPLE \
-DLLVM_TARGETS_TO_BUILD=ARM \
-DLLVM_ENABLE_PROJECTS="clang" \
-DLLVM_ENABLE_ASSERTIONS=On \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_FLAGS="-marm -fuse-ld=lld" \
-DCMAKE_C_FLAGS="-marm -fuse-ld=lld" \
/work/open_source/llvm-project/llvm/
ninja
will build native copies of some tools then cross compile everything else.
Testing
Most of them won’t run because they’re not native binaries of course. Some like unit tests you can ninja check-clang-unit
just to build the binary files and then emulate them one by one using qemu-user
. You can install qemu-user-static
from apt
`.
Note the -L
to point qemu to the libraries in the Arm sysroot.
For lit tests, try to extract the RUN
line you need to check and give that to qemu. One way to do this is to run the test in a native build and add the -a
option to lit to make it print them. Failing that, just break the test in some obvious way, and lit will dump what it tried to run.
If you want to ninja check-all
this cross build, look into binfmt_misc to automatically emulate non-native binaries.
Common Failure Causes
Most common is using the wrong type for 64 bit numbers like addresses. If the patch uses size_t
, this will be 32 bit on a 32 bit system. Put a uint64_t
into a size_t
on a 32 bit system and it’ll be truncated.
Another is stack and type alignment. A lot of code that does manual stack management wasn’t written with Arm in mind, so check the ABI (https://github.com/ARM-software/abi-aa/blob/main/aapcs32/aapcs32.rst ) if in doubt.
Sanitizers
Address sanitizer appears to be incompatible with qemu-user. You could build a kernel (https://lldb.llvm.org/resources/test.html#running-tests-in-qemu-system-emulation-environment ) and run it in qemu-system
instead.
Undefined behaviour sanitizer works without issue. Note that there is a decent amount of UB already in the test suite, but not all of it is causing failures. Ideally it wouldn’t exist, but prioritize what you look into.