How to Reproduce 32 bit Arm Builds Without Hardware

This guide is mainly for people external to Arm and Linaro. There are other methods to do this task, but this is going to be the most straightforward for 80% of people who don’t work primarily on Arm.

This can also be used for 64 bit builds by changing the chosen toolchain and qemu.

Building

With 32 bit support becoming increasingly rare, cross compilation and emulation is the most practical way for most people to do this. It also allows you to use the fast machine you already have rather than a low powered development board.

This guide will only cover Linux. Adapt as needed but remember that qemu-user doesn’t work everywhere, so either get a Linux VM then do this, or look at using qemu-system to create an Arm VM.

You can get a toolchain from apt or from Arm directly: https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads

Make sure you choose the correct host and target triples. If you want to target Linux (which you do 99% of the time) choose one with -linux in the name.

I recommend getting Arm’s because it’ll have the latest feature support. You could also use clang but you’ll need a sysroot, which you get from the same place anyway.

Extract the toolchain and add it to your PATH:

$ export PATH=$(pwd)/arm-gnu-toolchain-13.2.Rel1-x86_64-arm-none-linux-gnueabihf/bin/:$PATH

Set up a symlink inside the toolchain so it can find ld.lld. I've found that ld runs out of memory most of the time.

Note that this symink is in the <triple>/bin folder, not bin/.

arm-gnu-toolchain-13.2.Rel1-x86_64-arm-none-linux-gnueabihf/arm-none-linux-gnueabihf/bin$ ln -s $(which ld.lld-11) ./ld.lld

(there is probably a gcc flag to fix this issue “properly” but we are not here to learn gcc flags)

Next, configure llvm using this script:

$ cat ../configure-arm.sh set -ex cd /work/open_source/build-llvm-arm HOST_TRIPLE=arm-none-linux-gnueabihf C_COMPILER=$HOST_TRIPLE-gcc CXX_COMPILER=$HOST_TRIPLE-g++ cmake -G Ninja \ -DCMAKE_SYSTEM_NAME=Linux \ -DCMAKE_CROSSCOMPILING=1 \ -DCMAKE_C_COMPILER=$C_COMPILER \ -DCMAKE_CXX_COMPILER=$CXX_COMPILER \ -DLLVM_HOST_TRIPLE=$HOST_TRIPLE \ -DLLVM_TARGETS_TO_BUILD=ARM \ -DLLVM_ENABLE_PROJECTS="clang" \ -DLLVM_ENABLE_ASSERTIONS=On \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_CXX_FLAGS="-marm -fuse-ld=lld" \ -DCMAKE_C_FLAGS="-marm -fuse-ld=lld" \ /work/open_source/llvm-project/llvm/

ninja will build native copies of some tools then cross compile everything else.

Testing

Most of them won’t run because they’re not native binaries of course. Some like unit tests you can ninja check-clang-unit just to build the binary files and then emulate them one by one using qemu-user. You can install qemu-user-static from apt`.

Note the -L to point qemu to the libraries in the Arm sysroot.

For lit tests, try to extract the RUN line you need to check and give that to qemu. One way to do this is to run the test in a native build and add the -a option to lit to make it print them. Failing that, just break the test in some obvious way, and lit will dump what it tried to run.

If you want to ninja check-all this cross build, look into https://en.wikipedia.org/wiki/Binfmt_misc to automatically emulate non-native binaries.

Common Failure Causes

Most common is using the wrong type for 64 bit numbers like addresses. If the patch uses size_t, this will be 32 bit on a 32 bit system. Put a uint64_t into a size_t on a 32 bit system and it’ll be truncated.

Another is stack and type alignment. A lot of code that does manual stack management wasn’t written with Arm in mind, so check the ABI (https://github.com/ARM-software/abi-aa/blob/main/aapcs32/aapcs32.rst ) if in doubt.

Sanitizers

Address sanitizer appears to be incompatible with qemu-user. You could build a kernel (https://lldb.llvm.org/resources/test.html#running-tests-in-qemu-system-emulation-environment ) and run it in qemu-system instead.

Undefined behaviour sanitizer works without issue. Note that there is a decent amount of UB already in the test suite, but not all of it is causing failures. Ideally it wouldn’t exist, but prioritize what you look into.