Skip to main content
Back

SentencePiece v0.2.2 Drops SWIG to Fix LLaMA and Gemma Tokenizer Builds

Google's veteran subword tokenizer swaps its aging SWIG wrapper for pybind11, adds zero-copy NumPy paths, and hardens offset mapping.

Repo, Full Access
  • SentencePiece v0.2.2 migrates the Python wrapper from SWIG to pybind11 for cleaner builds.
  • Adds zero-copy NumPy decoding, inline .pyi type stubs, and experimental free-threading support.
  • New offset mapping exposes both Unicode character and raw byte offsets with bounds checking.
  • Wheels are 1-2 MB smaller thanks to externalized normalization rules via SPM_DISABLE_EMBEDDED_DATA.
  • Ships wheels for Python 3.9-3.14 including Windows ARM64 and free-threaded builds.
  • Install with pip install -U sentencepiece from PyPI.

SentencePiece, the tokenizer sitting quietly under LLaMA, T5, Gemma and countless multilingual models, just shipped one of its largest overhauls in years. The v0.2.2 release from Taku Kudo touches almost every piece of the Python surface, and it does so without changing the model format or breaking existing training scripts.

The release lands on PyPI with wheels covering Python 3.9 through 3.14, including the free-threaded 3.14t builds, ARM64 for Windows, and the usual manylinux/macOS matrix. pip install -U sentencepiece is all you need.

The wrapper finally leaves SWIG behind

The headline change is structural. The release migrates the Python wrapper from SWIG to pybind11, enabling better performance, safer memory management, and cleaner code integration. SWIG has been the source of most of SentencePiece's Python build headaches for years, especially on newer CPython versions and Apple Silicon, so this rewrite matters even if you never look at the bindings directly.

The practical fallout for users:

  • The Python wrapper now supports zero-copy decoding for NumPy arrays, significantly reducing memory overhead during large-scale data processing.
  • The pybind11 migration provides inline type stubs (.pyi files) for better IDE auto-completion and static type safety.