Architecture

SentencePiece

SentencePiece is a tokenization library that learns subword units directly from raw text, without assuming word boundaries. This makes it language-agnostic—it works equally well for languages without spaces. SentencePiece uses byte-pair encoding or unigram language models to learn the vocabulary. It's widely used in modern models including LLaMA. SentencePiece simplifies preprocessing since you feed raw text directly. It's particularly effective for multilingual models.