WoA Emulation using qemu-user/wine-arm64
We have been experimenting to run Windows on Arm binaries from a Linux-x64 machine using qemu-user and wine-arm64.
How it works
QEMU can run aarch64 binaries on x64 host:
qemu-aarch64 ./program
Wine-arm64 is set of native aarch64 binaries that implements Windows interface, and can run windows-arm64 programs as simply as:
wine ./program.exe
By combining both, we can run windows-arm64 binaries on a linux x64 machine:
qemu-aarch64 /path/to/wine-arm64/wine ./program.exe
QEMU
https://www.qemu.org/ is an open source machine emulator and virtualizer.
QEMU provides one binary per architecture, and comes in two variants under Linux: system and user modes.
System mode (qemu-system-aarch64
) emulates a whole “virtual machine”, which can be accelerated with virtualization by using https://www.linux-kvm.org/page/Main_Page.
User mode provides emulation for a single program (and not a whole machine). In this case, QEMU intercepts system calls, which is the interface between user space and kernel space. It translates those calls from host architecture to guest architecture, talking directly to the Linux kernel. This is why user mode works only on Linux, and not on other OS.
Debian provides qemu-user-static
which contains:
qemu-user binaries statically compiled
binfmt support https://en.wikipedia.org/wiki/Binfmt_misc: In short, when you try to execute an elf-aarch64 binary, it calls automatically qemu-user-aarch64 to run it.
Wine
https://www.winehq.org/ stands for “Wine is not an emulator”. https://en.wikipedia.org/wiki/Wine_(software)
It is an implementation of Windows API interface, and a https://en.wikipedia.org/wiki/Portable_Executable loader to run windows programs. It can run unmodified windows binaries.
Arm64 has been supported since Windows on Arm was released: https://wiki.winehq.org/ARM64
Documentation for this platform can be out of date, or lacking, so be careful.
Setup
Instructions are given for Debian:
Install docker: https://docs.docker.com/engine/install/debian/
Add your user to docker group:
sudo usermod -aG docker $USER
Logout your session and reconnect (so your user groups are changed)
Check docker can execute a container by running:
$ docker run -it --rm debian:bullseye bash -c 'echo Hello World'
Hello World
wine-arm64 Docker image
We created a docker x64 container image with wine-arm64 prebuilt and embedded qemu-user-static (so binfmt support is not required). It can easily be used with any CI system interfacing with containers: GitLab, GitHub, etc.
Image is published as linaro/wine-arm64
$ docker run -it --rm linaro/wine-arm64 wine-arm64 cmd.exe /c 'echo Hello World'
Hello World
Details can be found here: Unified docker image
GitLab and GitHub pipelines examples can be found here: https://gitlab.com/Linaro/windowsonarm/woa-linux-examples
Limitations
For now, it’s not yet possible to run arm64 version of cl.exe using wine-arm64, so it’s better to cross compile from windows-x64/Linux (using cl, clang, or llvm-mingw).
We target to run unit tests and programs, instead of full native compilation process for now.
Warning: Wine can have bugs. Either on the Windows interface, or specific arm64 bugs. In case you encounter a crash, don’t assume your program is necessarily faulty, and if you can, investigate on a Windows on Arm machine.
Experiments
We created a repository to collect binaries than can be use to evaluate wine correctness and qemu/wine performance: https://gitlab.com/Linaro/windowsonarm/woa-linux-test-binaries
OSQuery
Unittests from OSQuery. Some failures (9 tests on 70) found:
filesystem issues:
extension socket not implemented: https://gitlab.com/Linaro/windowsonarm/woa-linux-test-binaries/-/blob/master/fails/osquery/osquery/extensions/tests/osquery_extensions_extensiontests-test.exe
“various” windows issues: https://gitlab.com/Linaro/windowsonarm/woa-linux-test-binaries/-/blob/master/fails/osquery/osquery/tables/system/tests/osquery_tables_system_tests_systemtablestests-test.exe
issue with registry: https://gitlab.com/Linaro/windowsonarm/woa-linux-test-binaries/-/blob/master/fails/osquery/osquery/tables/system/tests/osquery_tables_system_windows_tests-test.exe
issue with
CoCreateInstance
https://gitlab.com/Linaro/windowsonarm/woa-linux-test-binaries/-/blob/master/fails/osquery/osquery/utils/conversions/osquery_utils_conversions_conversionstests-test.exe
OSQuery is using advanced for API on Windows, so it’s not a total surprise to find those issues.
Numpy-benchmarks
Running benchmarks from numpy https://github.com/numpy/numpy/tree/main/benchmarks/benchmarks .
Installing numpy with pip does not work under wine-arm64, as cl.exe is not available to compile native code.
All benchmarks work under wine-arm64
Performance:
volterra: windows-arm64: 0.175s
volterra: linux-arm64-vm + wine: 0.850s
intel i7-10700k: linux-x64 + qemu-user + wine: 1.4s
intel i7-10700k: linux-x64 + qemu-system + windows-arm64: 2.9-3.5s (variations)
Most of the tests are short, and bound by python startup.
Running full windows-arm64 vm is unresponsive (background services can randomly make the cpu busy) and slower than running qemu-user + wine. In more, booting the VM itself takes 4 minutes.