Machine Learning

Tokenization and Byte Pair Encoding

A language model never sees your letters, it sees tokens, chunks of text that fall between letters and words. Train a real BPE merge table on a small corpus, then watch your own text break into the pieces a model actually reads.

tokenizationbpesubwordlanguage-modelsmachine-learning

LiveInteractive · drag, toggle, run it

Machine Learning · How a model reads text

Tokenization and Byte Pair Encoding

Ask a model how many r's are in strawberry and it stumbles, because it never sees the letters. It sees tokens, chunks of text that usually land somewhere between a single letter and a whole word. Below, a merge table is trained on a short corpus the way real tokenizers learn theirs, then your text is broken into the exact pieces a model would read. Watch where the splits fall and where the letters disappear inside a chunk.

The input

Type or paste text to tokenize

Presets:

common words the corpus saw often collapse to one token each.

What the model reads

Each colored chunk is one token, with its vocabulary id below it. The id is what the model actually embeds: it never receives the characters, only this sequence of integers. Spaces show as · and a question mark means the piece is not in the trained vocabulary.

the40·?q25u32i13c3k16·?b2r26o22w34n21·?fox61

Characters

19

what you typed, letter by letter

Tokens

15

chunks the model receives

Words

4

what a human counts

Here 15 tokens carry 4 words and 19 characters. Tokens sit between the two: rarer or longer words split into several, while words the corpus saw often stay whole.

lossless: joining the tokens back rebuilds your text exactly

Why letter counting is hard

Take strawberry. With the current merges it lands as these tokens:

s28t30ra49w34berry53

The three r's are buried inside those chunks, never lined up as separate symbols. A model that only sees token ids has no direct view of the letters within a token, which is why counting characters or spelling backwards trips it up. The information is there in the bytes, just not in a form the model reads position by position.

The vocabulary size

Tokenizing starts from single characters and learns merges. More merges means a larger vocabulary and coarser tokens: common words fuse into one piece and the same text needs fewer tokens. Drag to add or remove merges and watch the chunks above grow.

Merges applied (vocabulary growth)40

Vocabulary holds 79 tokens: 39 base symbols plus 40 learned merges.

Base symbols

39

characters to start

Merges applied

40

of 54 learned

Vocabulary

79

distinct token ids

Watch the merges being learned

Training scans the corpus, finds the most frequent adjacent pair, and merges it into a new token, then repeats. Early merges glue common letter pairs like t + h; soon after, th + e builds the word the. The underscore marks a word end. Step through and watch whole words emerge.

Training step6

After 6 merges, the table holds the rules below.

#1t+hth17x

#2th+e_the_12x

#3e+rer9x

#4a+nan6x

#5an+d_and_6x

#6f+ofo5x

#7e+r_er_5x

#8a+s_as_5x

#9b+erber5x

#10ber+rberr5x

#11r+ara4x

#12d+s_ds_4x

#13r+e_re_4x

#14i+t_it_4x

#15berr+y_berry_4x

#16m+omo3x

#17t+ttt3x

#18w+owo3x

#19wo+rwor3x

#20wor+ds_words_3x

#21t+o_to_3x

#22i+s_is_3x

#23fo+x_fox_2x

#24d+odo2x

#25do+g_dog_2x

#26l+ele2x

#27mo+dmod2x

#28mod+emode2x

#29mode+l_model_2x

#30t+ete2x

#31t+oto2x

#32to+ktok2x

#33tok+etoke2x

#34toke+ntoken2x

#35token+s_tokens_2x

#36tt+ertter2x

#37tter+s_tters_2x

#38c+oco2x

#39co+mcom2x

#40b+ebe2x

#41i+nin2x

#42a+re_are_2x

#43s+psp2x

#44s+msm2x

#45sm+asma2x

#46sma+lsmal2x

#47i+eie2x

#48o+f_of_2x

#49i+n_in_2x

#50th+atha2x

#51tha+t_that_2x

#52fo+r_for_2x

#53th+erther2x

#54w+awa2x

Characters to subwords to tokens

characterThe smallest unit text starts as. A tokenizer begins here, with every distinct character as its own token, so any text can always be encoded even if no merge applies.

merge ruleA learned pair, the most frequent two adjacent tokens fused into one. BPE keeps the ordered list of these rules; encoding replays them in the same order on new text.

subwordA token between a letter and a word. Common words become a single subword token; a rare word falls back to several, so the vocabulary stays bounded yet nothing is unencodable.

token idThe integer a model receives. Each vocabulary entry has a fixed id, and the model embeds that id, never the characters inside it. Letter-level questions are hard for exactly this reason.

The merge table here is trained live on the corpus by counting adjacent pairs, not loaded from a file. Encoding replays those merges on your text, and joining the tokens back reproduces the input exactly, the same property a production tokenizer guarantees.

BPE trained by greedy pair frequency, encoding by ordered merge replay.