A tale of a dependency chain: Pillow and the crew

Owned by Gabor Kertesz (Deactivated)
Last updated: Apr 22, 2022
10 min read

In this page I’d like to show what dependencies and unexpected issues I’ve found during Pillow Python package porting to win-arm64 investigation and how were these handled.

I think these issues and cross-dependencies will be quite common on the road of win-arm64 enablement, so we can all learn from these to make it more visible and easier to solve.

 

 

Pillow

Pillow is a Python package for image processing, more precisely it’s a fork of PIL (Python Imaging Library).

As Pillow is released by platform specific wheel files (https://pypi.org/project/Pillow/#files ) we’d need to build it from source for win-arm64 and check will it pass out-of-the-box or we’d need to submit changes anywhere to make it work.

Pillow’s build procedure relies on setuptools. As setuptools is updated for win-arm64, that’s good.
Pillow relies on external libraries.
Two of them are mandatory, the rest are optional for passing the tests, but preferred to get full functionality.

https://pillow.readthedocs.io/en/latest/installation.html#building-from-source

External library dependencies

https://pillow.readthedocs.io/en/stable/installation.html#external-libraries

  • libjpeg - required by default

  • zlib - required by default

  • lcms2

  • freetype

  • openjpg

  • libwebp

  • libpng

  • libimagequant

  • libtiff

  • fribidi

So for minimal support, we need to check libjpeg and zlib and for full support we can check all other libraries as well. However as the CI requires all external libraries, accepting win-arm64 as a new platform, still all of them needs to be enabled.

Warning: Any of these 3rd party libraries could depend on other 3rd party libraries as well!

Build from source on Windows

This section shows how Pillow is built from source on Windows when it’s already ported and working on x64 for example. Later on it will shown what’s not working for WoA out-of-the-box and how it’s ported.

Tutorial

https://github.com/python-pillow/Pillow/blob/main/winbuild/build.rst

python winbuild\build_prepare.py winbuild\build\build_dep_all.cmd winbuild\build\build_pillow.cmd install

Steps

build_prepare.py

It’s a custom Python script, implemeting several things. So if you’d need to update one of the 3rd party libraries and add or change any build step, the logic will be found here.

  1. Download external libraries

  2. Create build scripts for external libraries

    • Example: FreeType is built via Visual Studio project https://github.com/python-pillow/Pillow/blob/main/winbuild/build_prepare.py#L208
      Generated example build script (on x64 as a reference): Pillow\winbuild\build\build_dep_freetype.cmd

      @echo ====================================================================== @echo ==== Building freetype (freetype-2.10.4) ==== @echo ====================================================================== cd /D C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\freetype-2.10.4 set INCLUDE=C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\inc set INCLIB=C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\lib set LIB=C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\lib path %PATH%;C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\bin call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x86_amd64 @echo on rmdir /S /Q "objs" "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe" builds\windows\vc2010\freetype.sln /t:"Clean" /p:Configuration="Release Static" /p:Platform=x64 /m "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe" builds\windows\vc2010\freetype.sln /t:"Build" /p:Configuration="Release Static" /p:Platform=x64 /m xcopy /Y /E "include" "C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\inc" copy /Y /B "objs\x64\Release Static\freetype.lib" "C:\kg\stuff\WoA\test\python_packages\github\Pillow\winbuild\build\lib"

  3. Setup environment by vcvars

build_dep_all.cmd

Build all external libraries, by calling all the created scripts.
Dependencies are handled by the order of the calls.

This step can be replaced by individual build_dep_<library>.cmd calls, if any of the optional libraries are not supported, as this script exists in case of error.

Building libraries

The external libraries are built by the following methods, based on the host project’s approach:

Library

Project

Library

Project

lcms2

Visual Studio (2017)

freetype

Visual Studio (2010)

zlib

nmake

libwebp

nmake

libjpeg

cmake + nmake

libpng

cmake + nmake

libimagequant

cmake + nmake

libtiff

cmake + nmake

openjpg

cmake + nmake

fribidi

cmake + nmake

nmake

Basically it’s the Microsoft version of Make.
So any project which includes an nmake file, can be built with MSVC compiler via command line.

build_pillow.cmd

It’s a wrapper for calling setup.py, including external libraries to the environment.

Example on x64 machine:

Testing

Regression tests

Issue found on Windows

The following test case is failed on Windows locally:

It’s explicitly disabled on Linux and for Windows on CI:
https://github.com/python-pillow/Pillow/blob/main/Tests/test_image_access.py#L399
https://github.com/python-pillow/Pillow/runs/4152656413?check_suite_focus=true#step:25:2227
It was reported, but basically it’s ignored:

https://github.com/python-pillow/Pillow/pull/5811#issuecomment-965165151

Lesson learnt: Unfortunately, if a package is released for Windows, it doesn’t guarantee all tests are running and passing and/or maintained.

selftest

Port for win-arm64

The basic support required only the vcvars environment setup update for ARM64.

  • recognise ARM64 as a Windows platform by platform.machine

  • call vcvars by x86_arm64 msbuild_architecture

    • even though this architecture sounds strange, it makes sense as the MSVC tools are x86 hosted for ARM64

Enable arm64 for MSVC on Windows by gaborkertesz-linaro · Pull Request #5811 · python-pillow/Pillow

Tested with all external libraries disabled, replacing the build_pillow script by direct setup.py call:

External libraries

According to Pillow’s tutorial, only libjpeg-turbo and zlib are required external libraries by default, but in order to run regression tests, all libraries are required as the build system fails if any of these are failing.

Library

Status for win-arm64 out-of-the-box

Library

Status for win-arm64 out-of-the-box

lcms2

freetype

zlib

libwebp

libjpeg

libpng

(Pillow build is successful, but Neon optimization disabled)

libimagequant

libtiff

openjpg

fribidi

FreeType

 

FreeType is a widely used library to render fonts: https://freetype.org/

Host project: https://gitlab.freedesktop.org/freetype

Pillow downloads from (example)

Pillow/winbuild/build_prepare.py at main · python-pillow/Pillow

Windows build

FreeType supports various platforms with various build systems, but for Windows it relies on Visual Studio 2010 project: https://gitlab.freedesktop.org/freetype/freetype/-/tree/master/builds/windows/vc2010

Test

The tests are only built with the Meson build system, so if we want to test at least locally on win-arm64, we should support Meson as well.

https://gitlab.freedesktop.org/freetype/freetype/-/tree/master/tests#build-the-test-programs

Issue found

One of the test cases wasn’t ported to Windows (not just for win-arm64).

Fix: https://gitlab.freedesktop.org/freetype/freetype/-/commit/e990c33f218dc7ca619444e17b0bf5085b4b727c

Port for win-arm64

Visual Studio build

VS2017 project should be updated to include ARM64 (as this was the first version to support ARM64).

https://gitlab.freedesktop.org/freetype/freetype/-/commit/80bd4cba7eb25ecae313a6ac376ad6b363655243

libpng

libpng is the official PNG reference library.

http://www.libpng.org/pub/png/libpng.html

Source: https://github.com/glennrp/libpng

Complicated

Marked as complicated, because there are multiple ways to include libpng and build on Windows and for Arm. Some of the combinations work for ARM64 out-of-the-box, some don’t.

Short version: For Pillow it’s working, for FreeType it’s not.

Host project’s build system’s on Windows:

FreeType’s Meson build:

Host project

build system

works on win-arm64 out-of-the-box

importing project

Explanation

Host project

build system

works on win-arm64 out-of-the-box

importing project

Explanation

libpng

VS 2010 solution

ARM64 as a new supported platform should be added

libpng

nmake

Pillow

Doesn’t include any Arm related code as it assumes Win == x86

Meson wrapdb

Meson

FreeType

Includes Arm Neon code, because __ARM_NEON__ is defined, but the preprocessing is failing, so it tries to build Armv7 assembly code.

Neon optimization

The Neon configuration makes it also complicated.

Neon optimization was implemented in libpng by:

Which version is required to be built is decided by preprocessor macros in the assembly, intrinsic C source and by pngpriv.h configuration header (https://github.com/glennrp/libpng/blob/libpng16/pngpriv.h#L110 )

The decision depends on the compiler and the target platform.

Port for win-arm64

libpng

Pull request including all commits: https://github.com/glennrp/libpng/pull/397/commits

  • Visual Studio

    • Update to Visual Studio 2019

    • Fix C2220 warning errors for VS2019

    • Add ARM64 as a new platform

    • nmake

      • Create nmake file for win-arm64 to include neon optimizations

    • ARM Neon config fix to recognise win-arm64 as a valid ARM plaftorm

libpng Meson wrapdb

 

imagequant

Project homepage: https://pngquant.org/lib/

Source code: https://github.com/ImageOptim/libimagequant

This is another specilaized image library: “Palette quantization library that powers pngquant and other PNG optimizers.”

As it’s shown in the summary table up, it’s built by CMake + nmake.

The only problem for win-arm64 is that Intel specific feature, Streaming SIMD Extensions (SSE) is default enabled.

Port for win-arm64

Fix in cmake file: Do not enable SSE feature for ARM64 by default https://github.com/ImageOptim/libimagequant/pull/66/commits/94adecb956eac74801bc035a0c403aaf44866d16

little cms2 (lcms2)

Project homepage: https://www.littlecms.com/

Source code: https://github.com/mm2/Little-CMS

It’s a small-footprint color management engine.

For Windows, it implements Visual Studio projects: https://github.com/mm2/Little-CMS/tree/master/Projects

Port for win-arm64

Add ARM64 to Visual Studio 2019: https://github.com/mm2/Little-CMS/pull/288

 

Integrate updated external libraries into Pillow

As soon as the host project accepted the win-arm64 port and released a new verson, it can be integrated into our consumer project.

FreeType

https://github.com/python-pillow/Pillow/commit/5549e629c108ef50a44353eba0ea567dc06248c3

libimagequant

https://github.com/python-pillow/Pillow/pull/5916

lcms2

https://github.com/python-pillow/Pillow/pull/6017

libpng

The host project is still in review at the time this page is last updated, however it’s working for win-arm64 for Pillow, because it builds via nmake, which ignores the Arm related optimization source code. (details: https://linaro.atlassian.net/wiki/spaces/WOAR/pages/28654632982/A+tale+of+a+dependency+chain+Pillow+and+the+crew#Complicated ).

Build tools

 

Meson

https://mesonbuild.com/index.html

Meson is another 3rd party project, which is needed only for building FreeType’s test now.
However, as it’s a generic build system, if we port it, it can be useful for other projects as well.

Port for win-arm64

Invoke arm specific vcvars.bat for arm: https://github.com/mesonbuild/meson/pull/9409/commits/466ad78239e0f30bed16c40b863a79dc1a80b4b0

Meson wrapdb

This is a collection of projects that use Meson as their build system, or have a meson port maintained by the Meson team.

https://github.com/mesonbuild/wrapdb

It implements 2 functionalities:

  1. Resolve the dependency by downloading the external project

  2. Build the project by Meson

Debug dll issue

  • Meson creates debug build by default.

  • During Meson config, it creates a simple C program and tries to build and run it.

    • If running it is failing by the following errors, you reproduced the debug dll issue.

Work-arounds

1.Add other paths where the debug dlls are available

 

2.Use release build

with Meson:

Visual Studio CMake

Details: https://linaro.atlassian.net/wiki/spaces/WOAR/pages/28677636097

ninja

Ninja is a widely used build system, probably doesn’t needs to be introduced. It is designed to have its input files generated by a higher-level build system, and it is designed to run builds as fast as possible.

Project homepage: https://ninja-build.org/

Source code: https://github.com/ninja-build/ninja

Now it’s required for Meson higher-level build system.

Port for win-arm64

https://linaro.atlassian.net/wiki/spaces/WOAR/pages/28697624600

CMake

CMake is also a widely used open-source, cross-platform family of tools designed to build, test and package software.

Project homepage: https://cmake.org/

Source code: https://gitlab.kitware.com/cmake/cmake

Bonus

CMake is a bonus investigation in this context, because:

However, now it’s not critical, still would be nice to understand what is missing for CMake for native win-arm64, if any.

Port for win-arm64

https://linaro.atlassian.net/wiki/spaces/WOAR/pages/28684353912/CMake

OpenSSL

OpenSSL is a software library for applications that secure communications over computer networks against eavesdropping or need to identify the party at the other end. It is widely used by Internet servers, including the majority of HTTPS websites.

For Windows binary releases are available for x86 and x64 also: https://sourceforge.net/projects/openssl/

Project homepage: https://www.openssl.org/

Port for win-arm64

Even though emulated x86 binary would work on win-arm64, for native applications (like CMake) a win-arm64 OpenSSL build is needed.

Fortunately the project is already ported for win-arm64, just a local build should be created:

https://github.com/openssl/openssl/issues/11289

Dependencies

Install x86 Perl: https://strawberryperl.com/

  • x86 emulated version doesn’t break the native chain here

Steps

The default installation directory is “c:\Program Files\OpenSSL”. CMake is checking that path by default.