setuptools-distutils
- Gabor Kertesz (Deactivated)
1. Overview
As package build process is controlled by distutils (superseded by setuptools), it’s important to know the current status of distutils and setuptools, why there are mutiple instances of an installation, how it’s possible that different packages uses different versions and understand what is the win-arm64 current support of them and what could be expected in short and long term.
2. Important issues
how to handle numpy.distutils and setuptools interaction · Issue #2372 · pypa/setuptools
PEP 632 – Deprecate distutils module | peps.python.org
3. Different distutils instances
3.1. Built-in lib in Python interpreter
Historical distutils of Python, included in CPython
example path in Windows
c:\Users\<user>\AppData\Local\Programs\Python\<Python version>\Lib\distutils\
This is going to be deprecated as PEP632 defines above.
Source: https://github.com/python/cpython/tree/main/Lib/distutils
Update
officially only with new Python installation can be updated
unofficially the directory can be overwritten by newer version
venv
Not copied into venv
Usage: default
This is invoked by Python if the SETUPTOOLS_USE_DISTUTILS environment variable is either:
SETUPTOOLS_USE_DISTUTILS=SETUPTOOLS_USE_DISTUTILS=stdlib
3.2. Within stand-alone setuptools package
new, forked version
Source: setuptools/setuptools/_distutils at main · pypa/setuptools
Update
as a stand-alone Python package, setuptools can be updated
example path in Windows
venv
It is copied into venv
Can be updated and used regardless of global setuptools package, if venv is activated
example path
Usage: can be forced to use by SETUPTOOLS_USE_DISTUTILS environment variable
4. WoA (win-arm64) support
Limitation of the cross-compilation
From Python point of view, we can only invoke native compilation on x64 and win-arm64 as well.
https://github.com/pypa/setuptools/blob/main/setuptools/_distutils/command/build_ext.py#L312
(self.plat_name == get_platform()) is set to true by our usecases
But it's cross-compilation by the C compiler (MSVC), so it’s not a perfectly clean cross-compilation as we convince Python to invoke the native build flow by a cross-compiler.
By VSCMD_ARG_TGT_ARCH env var, we're telling to Python it runs on the given platform
As there is no native compiler for win-arm64 this works (for now) - as seen in sources below - win-arm64 platform always results x86/arm64 compiler.
VSCMD_ARG_TGT_ARCH=arm64 -> x86_arm64 (PLAT_TO_VCVARS) -> VC\Tools\MSVC\<version>\bin\HostX86\ARM64\cl.exe
build_ext has plat_name as cross-compiling target platform parameter, which looks hacky at first sight and I suppose that's why it’s not preferred to use
I see one long-term problem here:
If there will be ever a native win-arm64/win-arm64 MSVC compiler, this method should be updated heavily as it can't fallback to win-arm64 = x86/win-arm64 anymore.
The current cross-compilation would fail as it would try to use the real native one.
It’s not a blocker, but it should be known.
4.1. Built-in lib in Python interpreter
Described setuptools-distutils | 3.1. Built in lib in Python interpreter
As this is going to be deprecated, I won’t expect any further support.
Short summary
On WoA machine, the cross-compile MSVC is called by default, but on x64 it can’t be enforced.
win-arm64 has been added as a host platform (in 2019)
This means on win-arm64 platform without any env var setting, it will invoke x86/arm64 MSVC compiler by default.
Which is good on WoA
https://github.com/python/cpython/blob/v3.10.0rc1/Lib/distutils/util.py#L39
https://github.com/python/cpython/blob/v3.10.0rc1/Lib/distutils/_msvccompiler.py#L163
win-arm64 is not added as a target platform
it means win-arm64 as build target can’t be enforced by VSCMD_ARG_TGT_ARCH=arm64 environment variable on x64
as this is going to be deprecated and not recommended to use for any projects it’s not a blocker, but any project that is tied to old distutils, can’t be cross-compiled from x64 to win-arm64, for example numpy!
https://github.com/python/cpython/blob/v3.10.0rc1/Lib/distutils/util.py#L99
Test
4.2. Within stand-alone setuptools package
Described setuptools-distutils | 3.2. Within stand alone setuptools package
As this deprecates the internal lib of CPython, any further support should be added and expected here.
Short summary
On WoA machine, the cross-compile MSVC is called by default, while on x64 it can be enforced by VSCMD_ARG_TGT_ARCH=arm64 env var.
win-arm64 is supported as host platform
win-arm64 is supported as a target platform as well