1. Overview

As package build process is controlled by distutils (superseded by setuptools), it’s important to know the current status of distutils and setuptools, why there are mutiple instances of an installation, how it’s possible that different packages uses different versions and understand what is the win-arm64 current support of them and what could be expected in short and long term.

2. Important issues

https://github.com/pypa/setuptools/issues/2372

https://www.python.org/dev/peps/pep-0632/

3. Different distutils instances

3.1. Built-in lib in Python interpreter

• Historical distutils of Python, included in CPython

  • example path in Windows

     c:\Users\<user>\AppData\Local\Programs\Python\<Python version>\Lib\distutils\

• This is going to be deprecated as PEP632 defines above.

• Source: https://github.com/python/cpython/tree/main/Lib/distutils

• Update

  • officially only with new Python installation can be updated

  • unofficially the directory can be overwritten by newer version

• venv

  • Not copied into venv

• Usage: default

  • This is invoked by Python if the SETUPTOOLS_USE_DISTUTILS environment variable is either:

    • SETUPTOOLS_USE_DISTUTILS=

    • SETUPTOOLS_USE_DISTUTILS=stdlib

3.2. Within stand-alone setuptools package

• new, forked version

• Source: https://github.com/pypa/setuptools/tree/main/setuptools/_distutils

• Update

  • as a stand-alone Python package, setuptools can be updated

  • pip install -U setuptools

  • python -m pip install -U setuptools

• example path in Windows

  c:\Users\<user>\AppData\Local\Programs\Python\<Python version>\Lib\site-packages\setuptools\_distutils\

• venv

  • It is copied into venv

  • Can be updated and used regardless of global setuptools package, if venv is activated

  • example path

    <venv dir>\Lib\site-packages\setuptools\_distutils\

• Usage: can be forced to use by SETUPTOOLS_USE_DISTUTILS environment variable

  • SETUPTOOLS_USE_DISTUTILS=local

4. WoA (win-arm64) support

Todo: fake native compilation where win-arm64 == x86/arm64 !

4.1. Built-in lib in Python interpreter

• Described https://linaro.atlassian.net/wiki/spaces/WOA/pages/28593914166/setuptools-distutils#3.1.-Built-in-lib-in-Python-interpreter

• As this is going to be deprecated, I won’t expect any further support.

• Short summary

  • On WoA machine, the cross-compile MSVC is called by default, but on x64 it can’t be enforced.

• win-arm64 has been added as a host platform (in 2019)

  • This means on win-arm64 platform without any env var setting, it will invoke x86/arm64 MSVC compiler by default.

    • Whis is good on WoA

  • https://github.com/python/cpython/blob/v3.10.0rc1/Lib/distutils/util.py#L39

    def get_host_platform():
        if os.name == 'nt':
            if 'amd64' in sys.version.lower():
                return 'win-amd64'
            if '(arm)' in sys.version.lower():
                return 'win-arm32'
            if '(arm64)' in sys.version.lower():
                return 'win-arm64'
            return sys.platform

  • https://github.com/python/cpython/blob/v3.10.0rc1/Lib/distutils/_msvccompiler.py#L163

        PLAT_TO_VCVARS = {
            'win32' : 'x86',
            'win-amd64' : 'x86_amd64',
            'win-arm32' : 'x86_arm',
            'win-arm64' : 'x86_arm64'
        }        

• win-arm64 is not added as a target platform

  • it means win-arm64 as build target can’t be enforced by VSCMD_ARG_TGT_ARCH=arm64 environment variable on x64

  • as this is going to be deprecated and not recommended to use for any projects it’s not a blocker, but any project that is tied to old distutils, can’t be cross-compiled from x64 to win-arm64, for example numpy!

    • see https://linaro.atlassian.net/wiki/spaces/WOA/pages/28593914166/setuptools-distutils#2.-Important-issues

  • https://github.com/python/cpython/blob/v3.10.0rc1/Lib/distutils/util.py#L99

    def get_platform():
        if os.name == 'nt':
            TARGET_TO_PLAT = {
                'x86' : 'win32',
                'x64' : 'win-amd64',
                'arm' : 'win-arm32',
            }
            return TARGET_TO_PLAT.get(os.environ.get('VSCMD_ARG_TGT_ARCH')) or get_host_platform()
        else:
            return get_host_platform()

Test

Test command:
    python setup.py build_ext

CPython(3.10_rc1)/distutils (old)
SETUPTOOLS_USE_DISTUTILS=
    x64
        VSCMD_ARG_TGT_ARCH=
            self.plat_name: win32
            get_platform() win32
            get_host_platform() win32
            self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x86\cl.exe        
        VSCMD_ARG_TGT_ARCH=arm64
            self.plat_name: win32
            get_platform() win32
            get_host_platform() win32
            self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x86\cl.exe
            
            Can't build for win-arm64!

    win-arm64
        VSCMD_ARG_TGT_ARCH=
            self.plat_name: win-arm64
            get_platform() win-arm64
            get_host_platform() win-arm64
            self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\ARM64\cl.exe

        VSCMD_ARG_TGT_ARCH=arm64
            self.plat_name: win-arm64
            get_platform() win-arm64
            get_host_platform() win-arm64
            self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\ARM64\cl.exe        
        
        Tested projects:
            Cython: successful
            numpy:  successful

4.2. Within stand-alone setuptools package

• Described https://linaro.atlassian.net/wiki/spaces/WOA/pages/28593914166/setuptools-distutils#3.2.-Within-stand-alone-setuptools-package

• As this deprecates the internal lib of CPython, any further support should be added and expected here.

• Short summary

  • On WoA machine, the cross-compile MSVC is called by default, while on x64 it can be enforced by VSCMD_ARG_TGT_ARCH=arm64 env var.

• win-arm64 is supported as host platform

  • https://github.com/pypa/setuptools/blob/main/setuptools/_distutils/util.py#L40

    def get_host_platform():
        if os.name == 'nt':
            if 'amd64' in sys.version.lower():
                return 'win-amd64'
            if '(arm)' in sys.version.lower():
                return 'win-arm32'
            if '(arm64)' in sys.version.lower():
                return 'win-arm64'
            return sys.platform

  • https://github.com/pypa/setuptools/blob/main/setuptools/_distutils/_msvccompiler.py#L92

        PLAT_TO_VCVARS = {
            'win32' : 'x86',
            'win-amd64' : 'x86_amd64',
            'win-arm32' : 'x86_arm',
            'win-arm64' : 'x86_arm64'
        }

• win-arm64 is supported as a target platform as well

  • https://github.com/pypa/setuptools/blob/0c302d232dd1a1f4ec531050532a84274edac2f4/setuptools/_distutils/util.py#L100

    def get_platform():
        if os.name == 'nt':
            TARGET_TO_PLAT = {
                'x86' : 'win32',
                'x64' : 'win-amd64',
                'arm' : 'win-arm32',
                'arm64': 'win-arm64',
            }
            return TARGET_TO_PLAT.get(os.environ.get('VSCMD_ARG_TGT_ARCH')) or get_host_platform()
        else:
            return get_host_platform()

Test

Test command:
    python setup.py build_ext

setuptools-57.4.0/distutils (new)
SETUPTOOLS_USE_DISTUTILS=local
    x64
        VSCMD_ARG_TGT_ARCH=
            self.plat_name: win32
            get_platform() win32
            get_host_platform() win32
            self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x86\cl.exe
        VSCMD_ARG_TGT_ARCH=arm64
            self.plat_name: win-arm64
            get_platform() win-arm64
            get_host_platform() win32
            self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\ARM64\cl.exe

            Tested projects:
                Cython: successful
                numpy:  failed (as numpy relies on old distutils)
    win-arm64
        VSCMD_ARG_TGT_ARCH=
            self.plat_name: win-arm64
            get_platform() win-arm64
            get_host_platform() win-arm64
            self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\ARM64\cl.exe
        VSCMD_ARG_TGT_ARCH=arm64
            self.plat_name: win-arm64
            get_platform() win-arm64
            get_host_platform() win-arm64
            self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\ARM64\cl.exe

        Tested projects:
            Cython: successful
            numpy:  failed (as numpy relies on old distutils)