XSIMD Investigation
Arrow SIMD Library
- xsimd: https://github.com/xtensor-stack/xsimd
header only
900+ stars, 28 contributors
Why Adopting It
Arrow need to decouple the SIMD level available at compile-time from the SIMD level available at runtime.
That is, we typically build optional AVX512 accelerations at compile-time, but only enable them at runtime if the CPU supports AVX512 (and if the environment variable ARROW_USER_SIMD_LEVEL wasn't forced to a lower value). XSIMD supports that level of control.
Arrow just need to compile with the right compiler flags to select the desired SIMD level, like we already do currently when compiling multiple versions of a function.
What We Need To Do
xsimd isn't very complete.
ximd is designed for mathematics calculating and it lacks the functions like bit/byte shuffling, byte stream split encoding, ARM SVE supporting, etc.
It also seems to lack the functions required for byte stream split encoding and decoding. Those functions are exported by libsimdpp under the names "zip_lo" and "zip_hi".