Every language model starts by cutting text into small pieces called tokens. But there is more than one way to make those cuts, and the choice shapes how well the model reads, how much memory it uses, and whether it ever gets stuck on a new word.
In simple words, there are four ways to split text: by whole words, by single characters, by subwords, and by raw bytes. Each one trades something for something else. In this blog, we will learn how each method works, and which one models like BERT, GPT, and Llama use.
Let's start with the big picture.
Why Split Text at All?
A neural network only does math on numbers. It cannot read the letters in "chess" directly. So the very first step is always to chop the text into tokens and give each token a number, called a token id. The model then works only with those ids.
The question is not whether to split the text, but where to make the cuts. If we cut in the wrong place, we either waste memory or lose the meaning. The four methods below are four different answers to that one question.
Word-Level Tokenization
The most natural idea is to treat every word as one token. We just split on spaces:
# word-level: one token per word
"playing chess".split() # -> ["playing", "chess"]This feels clean, and for short sentences it is. But it has two serious problems.
First, the vocabulary explodes. English alone has hundreds of thousands of words. Once we add names, typos, and other languages, the list of known words becomes huge. Every extra word costs memory.
Second, and worse, is the out-of-vocabulary problem. If the model never saw the word "tokenizer" during training, it has no id for it. The word becomes a single "unknown" token with no meaning at all. A tiny typo can turn a real word into a blank.
Character-Level Tokenization
The opposite idea is to make every single character its own token:
# character-level: one token per character
list("chess") # -> ["c", "h", "e", "s", "s"]Now the problems flip. The vocabulary is tiny: just the letters, digits, and a few symbols. And nothing is ever unknown, because any word is just a string of characters the model already knows.
But we pay a heavy price in length. A short sentence becomes a very long list of tokens. The model has to do far more work to see whole words, and it can only fit a fraction of the text in its limited context window. Reading gets slow and shallow.
Subword Tokenization
So, here comes the method almost every modern model uses: subword tokenization. The idea is to keep common words whole and split only rare words into smaller, known pieces.
The word "playing" might split into "play" plus "ing". A rare word like "tokenization" might become "token" plus "ization". The exact cuts depend on the vocabulary each model learns. Common words are one token, so sentences stay short. Rare words break into familiar parts, so nothing is ever truly unknown.
This is the middle ground we want. We get a reasonable vocabulary size, and we never get stuck on a new word. The next articles cover the three ways to build these subwords: BPE, WordPiece, and SentencePiece.
Byte-Level Tokenization
There is one more level, below characters: bytes. Every piece of text on a computer is stored as raw bytes, and byte-level tokenization works directly on those.
Why go this low? Because every text is stored as bytes. There are only 256 possible byte values, and text in every language, including emoji and symbols, is written with them. A byte-level model can never hit a character it does not know, not even an emoji or a rare script. It works as a safety net for everything else.
Modern systems often combine ideas: they run subword tokenization on top of bytes, so they get short sequences and the byte-level guarantee that nothing ever fails.
The Core Trade-Off: Vocabulary Size vs Sequence Length
Every method above is really balancing two numbers that pull in opposite directions.
A bigger vocabulary keeps more words whole, so each sentence becomes fewer tokens and the model reads faster. But a big vocabulary costs more memory and needs more data to learn well.
A smaller vocabulary uses less memory, but it chops words into more pieces. So sentences become longer and slower, and the model can fit less text at once.
Word-level sits at one extreme (huge vocabulary, short sequences), character-level at the other (tiny vocabulary, long sequences). Subword tokenization is popular because it lands in the middle.
Handling New Words: The OOV Test
The clearest way to compare the four methods is to hand each one a brand-new word it never saw in training, say "unbelievable", and watch what happens.
Word-level fails outright: the whole word becomes one blank "unknown" token. Character-level and byte-level never fail, but they stretch the word into a long string of tiny pieces. Subword handles it gracefully, splitting into known parts like "un", "believ", and "able", so the meaning mostly survives.
This single test is why subword tokenization won. It keeps sentences short like word-level, yet it never breaks on a new word like character-level.
Where Multilingual Text and Emoji Break the Rules
For plain English, character-level and byte-level look almost the same. The difference shows up the moment we leave English.
Many languages, like Japanese and Chinese, do not put spaces between words, so word-level splitting has nothing to split on. Accented letters and emoji are single characters to a human but several bytes to a computer. This is where byte-level helps most. It treats every script and symbol as just bytes, so one tokenizer can handle text in any language.
A Simple Decision Guide
Let me tabulate the four methods for your better understanding.
| Method | Vocabulary | Sequence length | Unknown words | Best for |
|---|---|---|---|---|
| Word-level | Huge | Short | Fails | Simple, small demos |
| Character-level | Tiny | Very long | Never fails | Small, character-heavy tasks |
| Subword | Medium | Medium | Splits gracefully | Almost every modern LLM |
| Byte-level | Tiny (256) | Long | Never fails | Multilingual, emoji, code |
In practice, the answer is almost always "subword, built on top of bytes." That combination gives short sentences, a manageable vocabulary, and a guarantee that nothing ever breaks.
Which Method Do Real Models Use?
Now, which method do the models we use every day run on? They all use a form of subword tokenization, just built in different ways.
BERT uses WordPiece. GPT models use byte-level BPE. T5 and Llama 2 use SentencePiece, while Llama 3 switched to a BPE tokenizer built on tiktoken, OpenAI's tokenizer library. None of them use plain word-level or plain character-level splitting, because the subword middle ground beats both. The next three articles open up each of these three subword methods in detail.
Recap
This is how text gets split. There are four ways, and each one balances two numbers that fight each other: vocabulary size and sequence length. Word-level keeps sentences short but drowns in a huge vocabulary and breaks on new words. Character-level never breaks but stretches text into long, slow sequences. Byte-level does the same at the level of raw bytes, which makes it a good fit for many languages and emoji. And subword tokenization lands in the middle, keeping common words whole while splitting rare ones into known pieces.
That middle ground is why every real model uses a subword method. In the next article, we open up the most popular one of all and watch it build its vocabulary from scratch: Byte-Pair Encoding, explained step by step.
