sentencepiece
- Niyas Sait (Deactivated)
Owned by Niyas Sait (Deactivated)
GitHub - google/sentencepiece: Unsupervised text tokenizer for Neural Network-based text generation.
Steps to build sentencepiece library and python package is given below.
git clone https://github.com/google/sentencepiece.git
cd sentencepiece
mkdir build
cd build
set CURRENT_PATH=%~dp0
set LIBRARY_PATH=%CURRENT_PATH%build\root
cmake .. -A ARM64 -DSPM_BUILD_TEST=ON -DSPM_ENABLE_SHARED=OFF -DCMAKE_INSTALL_PREFIX=%LIBRARY_PATH%
cmake --build . --config Release --target install
cd ../python
python setup.py bdist_wheelÂ
Â