In this page I’d like to show what dependencies and unexpected issues I’ve found during Pillow Python package porting to win-arm64 investigation and how were these handled.

I think these issues and cross-dependencies will be quite common on the road of win-arm64 enablement, so we can all learn from these to make it more visible and easier to solve.

Pillow

Pillow is a Python package for image processing, more precisely it’s a fork of PIL (Python Imaging Library).

Homepage: https://python-pillow.org/
PyPi: https://pypi.org/project/Pillow/

As Pillow is released by platform specific wheel files (https://pypi.org/project/Pillow/#files ) we’d need to build it from source for win-arm64 and check will it pass out-of-the-box or we’d need to submit changes anywhere to make it work.

Pillow’s build procedure relies on setuptools. As setuptools is updated for win-arm64, that’s good.
Pillow relies on external libraries.
Two of them are mandatory, the rest are optional for passing the tests, but preferred to get full functionality.

https://pillow.readthedocs.io/en/latest/installation.html#building-from-source

External library dependencies

https://pillow.readthedocs.io/en/stable/installation.html#external-libraries

libjpeg - required by default
zlib - required by default
lcms2
freetype
openjpg
libwebp
libpng
libimagequant
libtiff
fribidi

So for minimal support, we need to check libjpeg and zlib and for full support we can check all other libraries as well. However as the CI requires all external libraries, accepting win-arm64 as a new platform, still all of them needs to be enabled.

Warning: Any of these 3rd party libraries could depend on other 3rd party libraries as well!

Build from source on Windows

This section shows how Pillow is built from source on Windows when it’s already ported and working on x64 for example. Later on it will shown what’s not working for WoA out-of-the-box and how it’s ported.

Tutorial

https://github.com/python-pillow/Pillow/blob/main/winbuild/build.rst

python winbuild\build_prepare.py
winbuild\build\build_dep_all.cmd
winbuild\build\build_pillow.cmd install

Steps

build_prepare.py

It’s a custom Python script, implemeting several things. So if you’d need to update one of the 3rd party libraries and add or change any build step, the logic will be found here.

Download external libraries
- Example: libjpeg-turbo
  https://github.com/python-pillow/Pillow/blob/main/winbuild/build_prepare.py#L108
  SF_MIRROR = "http://iweb.dl.sourceforge.net" "url": SF_MIRROR + "/project/libjpeg-turbo/2.1.2/libjpeg-turbo-2.1.2.tar.gz",

Create build scripts for external libraries

Example: FreeType is built via Visual Studio project https://github.com/python-pillow/Pillow/blob/main/winbuild/build_prepare.py#L208
Generated example build script (on x64 as a reference): Pillow\winbuild\build\build_dep_freetype.cmd

@echo ======================================================================
@echo ==== Building freetype (freetype-2.10.4)                          ====
@echo ======================================================================
cd /D C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\freetype-2.10.4
set INCLUDE=C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\inc
set INCLIB=C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\lib
set LIB=C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\lib
path %PATH%;C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\bin
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x86_amd64
@echo on
rmdir /S /Q "objs"
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe" builds\windows\vc2010\freetype.sln /t:"Clean" /p:Configuration="Release Static" /p:Platform=x64 /m
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe" builds\windows\vc2010\freetype.sln /t:"Build" /p:Configuration="Release Static" /p:Platform=x64 /m
xcopy /Y /E "include" "C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\inc"
copy /Y /B "objs\x64\Release Static\freetype.lib" "C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\lib"

Setup environment by vcvars

https://github.com/python-pillow/Pillow/blob/main/winbuild/build_prepare.py#L362

architectures = {
    "x86": {"vcvars_arch": "x86", "msbuild_arch": "Win32"},
    "x64": {"vcvars_arch": "x86_amd64", "msbuild_arch": "x64"},
}

build_dep_all.cmd

Build all external libraries, by calling all the created scripts.
Dependencies are handled by the order of the calls.

@echo on

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_libjpeg.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_zlib.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_libtiff.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_libwebp.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_libpng.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_freetype.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_lcms2.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_openjpeg.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_libimagequant.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_harfbuzz.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

cmd.exe /c "C:\kg\python_test\github_packages\Pillow\winbuild\build\build_dep_fribidi.cmd"

if errorlevel 1 echo Build failed! && exit /B 1

@echo All Pillow dependencies built successfully!

This step can be replaced by individual build_dep_<library>.cmd calls, if any of the optional libraries are not supported, as this script exists in case of error.

Building libraries

The external libraries are built by the following methods, based on the host project’s approach:

Library	Project
lcms2	Visual Studio (2017)
freetype	Visual Studio (2010)
zlib	nmake
libwebp	nmake
libjpeg	cmake + nmake
libpng	cmake + nmake
libimagequant	cmake + nmake
libtiff	cmake + nmake
openjpg	cmake + nmake
fribidi	cmake + nmake

nmake

Basically it’s the Microsoft version of Make.
So any project which includes an nmake file, can be built with MSVC compiler via command line.

build_pillow.cmd

It’s a wrapper for calling setup.py, including external libraries to the environment.

Example on x64 machine:

@echo ---- Building Pillow (build_ext %*) ----
cd /D C:\kg\stuff\WoA\test\python_packages\github\Pillow
set INCLUDE=C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\inc
set INCLIB=C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\lib
set LIB=C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\lib
path %PATH%;C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\bin
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x86_amd64
@echo on
set DISTUTILS_USE_SDK=1
set MSSdk=1
set py_vcruntime_redist=true
"C:\kg\stuff\WoA\test\venv_win\Scripts\python.exe" setup.py build_ext --vendor-raqm --vendor-fribidi %*

Testing

Regression tests

pytest Tests

Issue found on Windows

The following test case is failed on Windows locally:

FAILED Tests/test_image_access.py::TestEmbeddable::test_embeddable - distutils.errors.LinkError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\...

It’s explicitly disabled on Linux and for Windows on CI:
https://github.com/python-pillow/Pillow/blob/main/Tests/test_image_access.py#L399
https://github.com/python-pillow/Pillow/runs/4152656413?check_suite_focus=true#step:25:2227
It was reported, but basically it’s ignored:

https://github.com/python-pillow/Pillow/pull/5811#issuecomment-965165151

Lesson learnt: Unfortunately, if a package is released for Windows, it doesn’t guarantee all tests are running and passing and/or maintained.

selftest

(venv_dir) PS C:\kg\python_test\github_packages\Pillow> python selftest.py
--------------------------------------------------------------------
Pillow 8.4.0.dev0
Python 3.10.0rc1 (tags/v3.10.0rc1:cc115e5, Aug  3 2021, 15:04:07) [MSC v.1929 64 bit (ARM64)]
--------------------------------------------------------------------
Python modules loaded from C:\kg\python_test\venv_dir\lib\site-packages\pillow-8.4.0.dev0-py3.10-win-arm64.egg\PIL
Binary modules loaded from C:\Users\gabker01\AppData\Local\Python-Eggs\Python-Eggs\Cache\pillow-8.4.0.dev0-py3.10-win-arm64.egg-tmp\PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 8.4.0.dev0
*** TKINTER support not installed
--- FREETYPE2 support ok, loaded 2.10.4
--- LITTLECMS2 support ok, loaded 2.12
--- WEBP support ok, loaded 1.2.1
--- WEBP Transparency support ok
--- WEBPMUX support ok
--- WEBP Animation support ok
--- JPEG support ok, compiled for libjpeg-turbo 2.1.1
--- OPENJPEG (JPEG2000) support ok, loaded 2.4.0
--- ZLIB (PNG/ZIP) support ok, loaded 1.2.11
--- LIBTIFF support ok, loaded 4.3.0
--- RAQM (Bidirectional Text) support ok, loaded 0.7.1, fribidi 1.0.11, harfbuzz 3.0.0
--- LIBIMAGEQUANT (Quantization method) support ok, loaded 2.16.0
*** XCB (X protocol) support not installed
--------------------------------------------------------------------
Running selftest:
--- 59 tests passed.

Port for win-arm64

The basic support required only the vcvars environment setup update for ARM64.

recognise ARM64 as a Windows platform by platform.machine
call vcvars by x86_arm64 msbuild_architecture
- even though this architecture sounds strange, it makes sense as the MSVC tools are x86 hosted for ARM64

https://github.com/python-pillow/Pillow/pull/5811

Tested with all external libraries disabled, replacing the build_pillow script by direct setup.py call:

python setup.py build_ext --disable-zlib --disable-jpeg --disable-tiff --disable-freetype --disable-lcms --disable-webp --disable-webpmux --disable-jpeg2000 --disable-imagequant --disable-xcb install

(venv_dir) C:\kg\python_test\github_packages\Pillow>python selftest.py
--------------------------------------------------------------------
Pillow 8.4.0.dev0
Python 3.10.0rc1 (tags/v3.10.0rc1:cc115e5, Aug  3 2021, 15:04:07) [MSC v.1929 64 bit (ARM64)]
--------------------------------------------------------------------
Python modules loaded from c:\kg\python_test\venv_dir\lib\site-packages\pillow-8.4.0.dev0-py3.10-win-arm64.egg\PIL
Binary modules loaded from C:\Users\gabker01\AppData\Local\Python-Eggs\Python-Eggs\Cache\pillow-8.4.0.dev0-py3.10-win-arm64.egg-tmp\PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 8.4.0.dev0
*** TKINTER support not installed
*** FREETYPE2 support not installed
*** LITTLECMS2 support not installed
*** WEBP support not installed
*** WEBP Transparency support not installed
*** WEBPMUX support not installed
*** WEBP Animation support not installed
*** JPEG support not installed
*** OPENJPEG (JPEG2000) support not installed
*** ZLIB (PNG/ZIP) support not installed
*** LIBTIFF support not installed
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed
*** XCB (X protocol) support not installed
--------------------------------------------------------------------
Running selftest:
--- 59 tests passed.

External libraries

According to Pillow’s tutorial, only libjpeg-turbo and zlib are required external libraries by default, but in order to run regression tests, all libraries are required as the build system fails if any of these are failing.

Library	Status for win-arm64 out-of-the-box
lcms2
freetype
zlib
libwebp
libjpeg
libpng	(Pillow build is successful, but Neon optimization disabled)
libimagequant
libtiff
openjpg
fribidi

FreeType

FreeType is a widely used library to render fonts: https://freetype.org/

Host project: https://gitlab.freedesktop.org/freetype

Pillow downloads from (example)

https://github.com/python-pillow/Pillow/blob/main/winbuild/build_prepare.py#L187

https://download.savannah.gnu.org/releases/freetype/freetype-2.11.1.tar.gz

Windows build

FreeType supports various platforms with various build systems, but for Windows it relies on Visual Studio 2010 project: https://gitlab.freedesktop.org/freetype/freetype/-/tree/master/builds/windows/vc2010

Test

The tests are only built with the Meson build system, so if we want to test at least locally on win-arm64, we should support Meson as well.

https://gitlab.freedesktop.org/freetype/freetype/-/tree/master/tests#build-the-test-programs

Issue found

One of the test cases wasn’t ported to Windows (not just for win-arm64).

Fix: https://gitlab.freedesktop.org/freetype/freetype/-/commit/e990c33f218dc7ca619444e17b0bf5085b4b727c

Port for win-arm64

Visual Studio build

VS2017 project should be updated to include ARM64 (as this was the first version to support ARM64).

https://gitlab.freedesktop.org/freetype/freetype/-/commit/80bd4cba7eb25ecae313a6ac376ad6b363655243

libpng

libpng is the official PNG reference library.

http://www.libpng.org/pub/png/libpng.html

Source: https://github.com/glennrp/libpng

Complicated

Marked as complicated, because there are multiple ways to include libpng and build on Windows and for Arm. Some of the combinations work for ARM64 out-of-the-box, some don’t.

Short version: For Pillow it’s working, for FreeType it’s not.

Host project’s build system’s on Windows:

Visual Studio 2010: https://github.com/glennrp/libpng/blob/libpng16/projects/vstudio/vstudio.sln
nmake script: https://github.com/glennrp/libpng/blob/libpng16/scripts/makefile.vcwin32

FreeType’s Meson build:

Custom build script, building with MSVC https://github.com/mesonbuild/wrapdb/blob/master/subprojects/packagefiles/libpng/meson.build

Host project	build system	importing project	Explanation
libpng	VS 2010 solution		ARM64 as a new supported platform should be added
libpng	nmake	Pillow	Doesn’t include any Arm related code as it assumes Win == x86
Meson wrapdb	Meson	FreeType	Includes Arm Neon code, because `__ARM_NEON__` is defined, but the preprocessing is failing, so it tries to build Armv7 assembly code.

Neon optimization

The Neon configuration makes it also complicated.

Neon optimization was implemented in libpng by:

assembly: up to armv7
- PNG_ARM_NEON_IMPLEMENTATION=2
  https://github.com/glennrp/libpng/blob/libpng16/pngpriv.h#L150
C intrinsics: starting with armv8
- PNG_ARM_NEON_IMPLEMENTATION=1
  https://github.com/glennrp/libpng/blob/libpng16/pngpriv.h#L149

Which version is required to be built is decided by preprocessor macros in the assembly, intrinsic C source and by pngpriv.h configuration header (https://github.com/glennrp/libpng/blob/libpng16/pngpriv.h#L110 )

The decision depends on the compiler and the target platform.

Port for win-arm64

libpng

Pull request including all commits: https://github.com/glennrp/libpng/pull/397/commits

Visual Studio
- Update to Visual Studio 2019
- Fix C2220 warning errors for VS2019
- Add ARM64 as a new platform
- nmake
  - Create nmake file for win-arm64 to include neon optimizations
- ARM Neon config fix to recognise win-arm64 as a valid ARM plaftorm
  - https://github.com/glennrp/libpng/issues/398

libpng Meson wrapdb

Assembly source file requires preprocessing. Gcc granted this by default, while MSVC doesnt, so for MSVC it should be done explicitly by C preprocessor
- https://github.com/mesonbuild/wrapdb/pull/205/commits/da6a13fb61f64baca0567763c51efaed9b14c41f
- The assembly optimization file is preprocessed, which results empty assembly file then C intrinsic Neon optimization will be compiled, if the work-around is used below.
- As libpng was inactive at the time I’ve created these patches (and written this summary), the ARM Neon config issue was fixed as a temporary work-around here
  - https://github.com/mesonbuild/wrapdb/pull/216/commits/a32c61a15724dfcf1af559160c7b1c1412f773d8
  - Even if libpng won’t explicitly support MSVC ARM64 define, we can still support ARM related source code for MSVC.
- Integrate updated Meson config of libpng: https://gitlab.freedesktop.org/freetype/freetype/-/merge_requests/105/diffs?commit_id=e6e6cbf1648d4a776da0857921872f2fbc853205

imagequant

Project homepage: https://pngquant.org/lib/

Source code: https://github.com/ImageOptim/libimagequant

This is another specilaized image library: “Palette quantization library that powers pngquant and other PNG optimizers.”

As it’s shown in the summary table up, it’s built by CMake + nmake.

The only problem for win-arm64 is that Intel specific feature, Streaming SIMD Extensions (SSE) is default enabled.

Port for win-arm64

Fix in cmake file: Do not enable SSE feature for ARM64 by default https://github.com/ImageOptim/libimagequant/pull/66/commits/94adecb956eac74801bc035a0c403aaf44866d16

little cms2 (lcms2)

Project homepage: https://www.littlecms.com/

Source code: https://github.com/mm2/Little-CMS

It’s a small-footprint color management engine.

For Windows, it implements Visual Studio projects: https://github.com/mm2/Little-CMS/tree/master/Projects

Port for win-arm64

Add ARM64 to Visual Studio 2019: https://github.com/mm2/Little-CMS/pull/288

Integrate updated external libraries into Pillow

As soon as the host project accepted the win-arm64 port and released a new verson, it can be integrated into our consumer project.

FreeType

https://github.com/python-pillow/Pillow/commit/5549e629c108ef50a44353eba0ea567dc06248c3

libimagequant

https://github.com/python-pillow/Pillow/pull/5916

lcms2

https://github.com/python-pillow/Pillow/pull/6017

libpng

The host project is still in review at the time this page is last updated, however it’s working for win-arm64 for Pillow, because it builds via nmake, which ignores the Arm related optimization source code. (details: https://linaro.atlassian.net/wiki/spaces/WOAR/pages/28654632982/A+tale+of+a+dependency+chain+Pillow+and+the+crew#Complicated ).

Build tools

Windows On Arm (WOA) > A tale of a dependency chain: Pillow and the crew > image-20220204-152111.png

Meson

https://mesonbuild.com/index.html

Meson is another 3rd party project, which is needed only for building FreeType’s test now.
However, as it’s a generic build system, if we port it, it can be useful for other projects as well.

Port for win-arm64

Invoke arm specific vcvars.bat for arm: https://github.com/mesonbuild/meson/pull/9409/commits/466ad78239e0f30bed16c40b863a79dc1a80b4b0

Meson wrapdb

This is a collection of projects that use Meson as their build system, or have a meson port maintained by the Meson team.

https://github.com/mesonbuild/wrapdb

It implements 2 functionalities:

Resolve the dependency by downloading the external project
- example: https://github.com/mesonbuild/wrapdb/blob/master/subprojects/libpng.wrap
Build the project by Meson
- example: https://github.com/mesonbuild/wrapdb/blob/master/subprojects/packagefiles/libpng/meson.build

Debug dll issue

Meson creates debug build by default.
During Meson config, it creates a simple C program and tries to build and run it.
- If running it is failing by the following errors, you reproduced the debug dll issue.

Windows On Arm (WOA) > A tale of a dependency chain: Pillow and the crew > 9fcb3576-74b8-462b-8b30-209033396d33#media-blob-url=true&id=75180722-d464-487d-a5c8-127bf5742729&contextId=40798&collection=

Windows On Arm (WOA) > A tale of a dependency chain: Pillow and the crew > 03dbebfa-f270-4b4a-9d43-8347dcd67fca#media-blob-url=true&id=d6dc04b1-bc0f-40b9-ab6f-831ca4f3a1cd&contextId=40798&collection=

Work-arounds

1.Add other paths where the debug dlls are available

cmd
	PATH=c:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\arm64\ucrt\;c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Redist\MSVC\14.29.30036\debug_nonredist\arm64\Microsoft.VC142.DebugCRT\%PATH%
PS
	$env:PATH="c:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\arm64\ucrt\;c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Redist\MSVC\14.29.30036\debug_nonredist\arm64\Microsoft.VC142.DebugCRT\;$env:PATH"

2.Use release build

with Meson:

meson setup out --buildtype release

Visual Studio CMake

cmake --build . --config Release

Details: Debug run-time DLL issue

ninja

Ninja is a widely used build system, probably doesn’t needs to be introduced. It is designed to have its input files generated by a higher-level build system, and it is designed to run builds as fast as possible.

Project homepage: https://ninja-build.org/

Source code: https://github.com/ninja-build/ninja

Now it’s required for Meson higher-level build system.

Port for win-arm64

Ninja

CMake

CMake is also a widely used open-source, cross-platform family of tools designed to build, test and package software.

Project homepage: https://cmake.org/

Source code: https://gitlab.kitware.com/cmake/cmake

Bonus

CMake is a bonus investigation in this context, because:

x86 version is released and can be used
- https://cmake.org/download/ ->cmake-3.22.1-windows-i386.msi
none of the dependant projects require it
- for Meson it’s optional
- Ninja Python distribution requires it, but Ninja the stand-alone distribution doesn’t

However, now it’s not critical, still would be nice to understand what is missing for CMake for native win-arm64, if any.

Port for win-arm64

https://linaro.atlassian.net/wiki/spaces/WOAR/pages/28684353912/CMake

OpenSSL

OpenSSL is a software library for applications that secure communications over computer networks against eavesdropping or need to identify the party at the other end. It is widely used by Internet servers, including the majority of HTTPS websites.

For Windows binary releases are available for x86 and x64 also: https://sourceforge.net/projects/openssl/

Project homepage: https://www.openssl.org/

Port for win-arm64

Even though emulated x86 binary would work on win-arm64, for native applications (like CMake) a win-arm64 OpenSSL build is needed.

Fortunately the project is already ported for win-arm64, just a local build should be created:

https://github.com/openssl/openssl/issues/11289

Dependencies

Install x86 Perl: https://strawberryperl.com/

x86 emulated version doesn’t break the native chain here

Steps

"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsx86_arm64.bat"
c:\kg\python_test\3rdparty\openssl>c:\Strawberry\perl\bin\perl.exe Configure VC-WIN64-ARM
nmake
nmake test
nmake install # do it as admin, otherwise installation copy to c:\Program Files\OpenSSL fails

The default installation directory is “c:\Program Files\OpenSSL”. CMake is checking that path by default.