In the last article we saw that subword tokenization is the best middle ground. Now we open up the most popular subword method of all: Byte-Pair Encoding, or BPE. It powers GPT, RoBERTa, and Llama 3, and the idea behind it is simple.
In simple words, BPE starts with single characters and slowly glues together the pairs that show up most often, building longer and longer pieces. Common chunks like "ing" and "tion" get their own tokens because they appear so much. By the end, we will be able to run BPE by hand.
Let's see the whole process at a glance.
Where Does BPE Start?
BPE begins at the smallest possible pieces: single characters. Before it learns anything, the word "low" is just the characters l, o, w. We also add a special end-of-word marker, written </w>, so the model can tell where one word stops and the next begins.
# every word starts as a list of characters plus an end marker
tokens = list("low") + ["</w>"] # ['l', 'o', 'w', '</w>']At this starting point, the vocabulary is only the individual characters. That is tiny and safe, but it means every word is a long string of letters. BPE's job is to fix that by learning bigger pieces.
Step One: Count the Pairs
Now BPE looks at all the words and counts how often each pair of neighbouring pieces appears together. The end marker </w> counts as a piece here, just like any letter. Our words are "low", "newest" and "widest", each counted once. Three pairs tie at the top with a count of 2: "e s", "s t" and "t ". When pairs tie, BPE takes the first pair it found, so "e s" wins this round.
This counting is the heart of BPE. The pairs that appear most often are the ones worth gluing, because a shortcut for a common pair saves the most effort overall.
Step Two: Merge the Winner
BPE takes the most frequent pair and glues it into a single new token everywhere it appears. Here, "e" and "s" is the winner, so every "e s" in the corpus becomes one piece, "es". That new piece is added to the vocabulary, and the rule "merge e and s into es" is saved as a merge.
# conceptual: merge the most frequent pair into one token
# ('e','s') is most frequent -> add new token 'es'
# every 'e','s' in the corpus becomes 'es'This is one round. BPE just repeats it: count the pairs again, find the new winner, merge it, and save the rule.
Repeat Until Done
BPE keeps merging, and the pieces keep growing. On our three words, it makes five merges in total. After "es", it merges "es" and "t" into "est". Next, it glues "est" and </w> into "est", which means "est" at the end of a word. Then it glues "l" and "o" into "lo", and last, "lo" and "w" into "low". Each round builds on the last, so a common ending like "est" and a whole word like "low" both become single tokens.
We stop when the vocabulary reaches the size we chose, often tens of thousands of tokens. Our tiny example stops after 5 merges. The result is a mix: single characters for rare cases, common chunks like "ing", and whole common words.
What BPE Actually Learns
When training finishes, BPE has produced two things: a vocabulary of tokens, and an ordered list of merges. In our example, the vocabulary has 15 tokens: the 10 starting symbols plus the 5 new pieces. The merge list is the recipe. It says, in order, which pairs to glue.
That order matters. To tokenize any new text later, BPE simply replays these merges from the top, gluing pairs in the exact order it learned them. Nothing is random; it is the same recipe every time.
Encoding a Brand-New Word
Here is where BPE proves its worth. When we give it a word it never saw in training, it does not panic. It starts the word as characters, then applies its learned merges in order, gluing whatever pairs it can.
So a new word like "lowest" starts as seven pieces: l, o, w, e, s, t and </w>. After the five merges replay, it becomes just two: "low" and "est". Both are pieces BPE already knows. The word is new, but every piece is familiar. This only works when BPE knows every character in the word. That is the weak spot we look at next.
Byte-Level BPE: How GPT Does It
Plain BPE has one weak spot. If it never saw a character during training, like a rare emoji or a symbol from another script, it has no token for it. In our example, the base alphabet is only the 10 symbols seen in training, so an emoji or a kanji gets no token at all. GPT fixes this with a small change called byte-level BPE.
Instead of starting from characters, it starts from raw bytes. There are only 256 possible bytes. In UTF-8, every character in any language is stored as a short run of them. In the figure, the emoji becomes 4 bytes and the kanji becomes 3. So the base alphabet is complete before training even begins. BPE then merges frequent byte pairs just like normal. As a result, the tokenizer never meets a character it cannot represent.
Why the Merge Order Matters
It is worth pausing on one detail, because it trips people up. BPE is not just a dictionary of pieces. It is an ordered recipe. The same word always tokenizes the same way, because the merges always replay in the same order.
This makes BPE fast and predictable. There is no searching and no guessing at tokenize time. It simply walks the merge list from top to bottom and glues what it can, every single time.
Where BPE Is Used
BPE is used very widely. The plain version and the byte-level version together tokenize a large share of the models people use daily. GPT-2, GPT-3, and GPT-4 use byte-level BPE. So do RoBERTa and many open models. Llama 3 uses a BPE tokenizer built on tiktoken, while Llama 2 used SentencePiece. Vocabulary sizes differ from model to model. GPT-2 has 50,257 tokens, while GPT-4's tokenizer (cl100k) has about 100,000.
The next article looks at a close cousin that BERT uses instead, WordPiece, which merges by a slightly different rule.
Recap
This is how BPE works. It starts every word as characters, counts which neighbouring pairs appear most, and glues the winner into a new token. Then it repeats, so the pieces grow from single letters into common endings like "est" and whole words like "low". What it learns is an ordered list of merges, a recipe it replays to tokenize any new text, even a word it never saw.
GPT takes this one step lower with byte-level BPE, starting from the 256 raw bytes so it can never meet a character it cannot handle. In the next article, we look at how BERT does it differently with WordPiece, and the small change in the merge rule that sets the two apart.
