setuptools-distutils

1. Overview

As package build process is controlled by distutils (superseded by setuptools), it’s important to know the current status of distutils and setuptools, why there are mutiple instances of an installation, how it’s possible that different packages uses different versions and understand what is the win-arm64 current support of them and what could be expected in short and long term.

2. Important issues

https://github.com/pypa/setuptools/issues/2372

https://www.python.org/dev/peps/pep-0632/

3. Different distutils instances

3.1. Built-in lib in Python interpreter

  • Historical distutils of Python, included in CPython

    • example path in Windows

      1 c:\Users\<user>\AppData\Local\Programs\Python\<Python version>\Lib\distutils\
  • This is going to be deprecated as PEP632 defines above.

  • Source: https://github.com/python/cpython/tree/main/Lib/distutils

  • Update

    • officially only with new Python installation can be updated

    • unofficially the directory can be overwritten by newer version

  • venv

    • Not copied into venv

  • Usage: default

    • This is invoked by Python if the SETUPTOOLS_USE_DISTUTILS environment variable is either:

      • 1 SETUPTOOLS_USE_DISTUTILS=
      • 1 SETUPTOOLS_USE_DISTUTILS=stdlib

3.2. Within stand-alone setuptools package

  • new, forked version

  • Source: https://github.com/pypa/setuptools/tree/main/setuptools/_distutils

  • Update

    • as a stand-alone Python package, setuptools can be updated

    • 1 pip install -U setuptools
    • 1 python -m pip install -U setuptools
  • example path in Windows

    1 c:\Users\<user>\AppData\Local\Programs\Python\<Python version>\Lib\site-packages\setuptools\_distutils\
  • venv

    • It is copied into venv

    • Can be updated and used regardless of global setuptools package, if venv is activated

    • example path

      1 <venv dir>\Lib\site-packages\setuptools\_distutils\
  • Usage: can be forced to use by SETUPTOOLS_USE_DISTUTILS environment variable

    • 1 SETUPTOOLS_USE_DISTUTILS=local

4. WoA (win-arm64) support

Limitation of the cross-compilation

  • From Python point of view, we can only invoke native compilation on x64 and win-arm64 as well.

  • But it's cross-compilation by the C compiler (MSVC), so it’s not a perfectly clean cross-compilation as we convince Python to invoke the native build flow by a cross-compiler.

    • By VSCMD_ARG_TGT_ARCH env var, we're telling to Python it runs on the given platform

    • As there is no native compiler for win-arm64 this works (for now) - as seen in sources below - win-arm64 platform always results x86/arm64 compiler.

      • VSCMD_ARG_TGT_ARCH=arm64 -> x86_arm64  (PLAT_TO_VCVARS) -> VC\Tools\MSVC\<version>\bin\HostX86\ARM64\cl.exe

  • build_ext has plat_name as cross-compiling target platform parameter, which looks hacky at first sight and I suppose that's why it’s not preferred to use

  • I see one long-term problem here:

    • If there will be ever a native win-arm64/win-arm64 MSVC compiler, this method should be updated heavily as it can't fallback to win-arm64 = x86/win-arm64 anymore.

      • The current cross-compilation would fail as it would try to use the real native one.

    • It’s not a blocker, but it should be known.

4.1. Built-in lib in Python interpreter

Test

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 Test command: python setup.py build_ext CPython(3.10_rc1)/distutils (old) SETUPTOOLS_USE_DISTUTILS= x64 VSCMD_ARG_TGT_ARCH= self.plat_name: win32 get_platform() win32 get_host_platform() win32 self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x86\cl.exe VSCMD_ARG_TGT_ARCH=arm64 self.plat_name: win32 get_platform() win32 get_host_platform() win32 self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x86\cl.exe Can't build for win-arm64! win-arm64 VSCMD_ARG_TGT_ARCH= self.plat_name: win-arm64 get_platform() win-arm64 get_host_platform() win-arm64 self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\ARM64\cl.exe VSCMD_ARG_TGT_ARCH=arm64 self.plat_name: win-arm64 get_platform() win-arm64 get_host_platform() win-arm64 self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\ARM64\cl.exe Tested projects: Cython: successful numpy: successful

4.2. Within stand-alone setuptools package

Test

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 Test command: python setup.py build_ext setuptools-57.4.0/distutils (new) SETUPTOOLS_USE_DISTUTILS=local x64 VSCMD_ARG_TGT_ARCH= self.plat_name: win32 get_platform() win32 get_host_platform() win32 self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x86\cl.exe VSCMD_ARG_TGT_ARCH=arm64 self.plat_name: win-arm64 get_platform() win-arm64 get_host_platform() win32 self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\ARM64\cl.exe Tested projects: Cython: successful numpy: failed (as numpy relies on old distutils) win-arm64 VSCMD_ARG_TGT_ARCH= self.plat_name: win-arm64 get_platform() win-arm64 get_host_platform() win-arm64 self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\ARM64\cl.exe VSCMD_ARG_TGT_ARCH=arm64 self.plat_name: win-arm64 get_platform() win-arm64 get_host_platform() win-arm64 self.cc C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\ARM64\cl.exe Tested projects: Cython: successful numpy: failed (as numpy relies on old distutils)