The Crucible #2: How Language Models Read Text

Introduction

Post #1 ended on a question. How does language become numbers?

I knew the theoretical answer before I sat down to build anything. Text gets tokenized, tokens get IDs, IDs get looked up in an embedding table. I could have explained that in a sentence and moved on.

Building it myself was a different experience entirely. Watching a sentence I typed get chopped into pieces, turned into integers, and then turned into vectors that a neural network can actually operate on, that was the first moment in this project where something genuinely amazed me. Not because the concept is complicated. It isn't. But there's a real difference between knowing that text becomes numbers and watching it happen to your own input, one step at a time, in code you wrote yourself.

This post covers Phases 1 and 2 of The Crucible: tokenization and embeddings. By the end, we'll have taken a raw sentence all the way to the final input representation that a transformer actually sees.

The code for this project lives at github.com/Arman001/the-crucible, including the notebook this post is based on.

Prerequisites

If you haven't read Post #1, a quick read will help since this post picks up exactly where that one left off. Beyond that, I'm assuming you're comfortable with Python and have touched PyTorch before. I won't be explaining what a tensor is.

What You'll Build

Two things.

First, a tokenizer built entirely by hand using regular expressions, no libraries. We'll watch it fail on a word it hasn't seen before, patch it with a workaround, and then replace it with the same Byte Pair Encoding approach GPT actually uses.

Second, the pipeline that takes those tokens all the way to the final input embeddings a transformer receives: a data loader that slices text into training examples, a token embedding lookup, and positional embeddings added on top.

Why Tokenization Exists

I went in assuming this part would be almost mechanical. Neural networks do math on numbers, not words, so obviously something has to convert text into numbers first. Fine. I did not expect that "something" to involve real design decisions, or for the simplest possible approach to break as fast as it did.

The obvious first idea is to split on whitespace and punctuation and treat each word as a unit. I started there, using a regex split on the-verdict.txt, a short story I used as a test corpus:

"Hello, world. Is this-- a test?"
→ ['Hello', ',', 'world', '.', 'Is', 'this', '--', 'a', 'test', '?']

From there, building a vocabulary is just collecting every unique token and assigning each one an integer. On my test corpus that came out to 1,130 unique tokens. Encoding is looking up the ID for each token. Decoding is the reverse, joining the words back with a bit of regex cleanup so punctuation lands next to the word it belongs to instead of floating with its own space.

I wrapped this into a class, SimpleTokenizerV1, with encode() and decode() methods. Round-tripped a test sentence through it and got the exact same sentence back. Satisfying, and also exactly what you'd expect. The real lesson was waiting one cell later.

Watching It Break

I ran a new sentence through the same tokenizer, one that mentioned "LLM." And it threw a KeyError.

That was the moment it actually clicked why this problem exists. My vocabulary only knows the 1,130 tokens that appeared in the training text. Any word outside that set doesn't have an ID. The tokenizer has no fallback, so it just crashes.

The first instinct is to patch it. Add an <|unk|> token to the vocabulary, and route any word that isn't recognized to that token instead of failing. I did exactly that in SimpleTokenizerV2, along with an <|endoftext|> token to mark boundaries between separate documents, which matters once you're training on more than one source of text.

It works. But it's a hack, and you can feel it being a hack. Every unknown word collapses into the same generic <|unk|> token, which means the model loses all information about what that word actually was:

"Hi, do you like working with LLMs?"
→ "<|unk|>, do you like working with <|unk|>? ..."

"Hi" and "LLMs" both vanish into the same token. That's a real loss of information, and it gets worse the more specialized or unusual your text is.

The Actual Fix: Byte Pair Encoding

Before I actually built any of this, I think I had a vague assumption that GPT just had a really, really big vocabulary, big enough to contain basically every word you'd throw at it. Watching my own tokenizer crash on a single unfamiliar word made it obvious why that assumption doesn't hold. No fixed vocabulary, however large, covers every word that will ever show up. Something has to handle the words that were never seen during training, and "assign it a generic unknown token" throws away exactly the information you'd want to keep.

GPT's actual answer is Byte Pair Encoding. It starts from individual characters and repeatedly merges the most frequently occurring pairs until it has built up a vocabulary of subword pieces, common fragments, whole common words, everything in between. The payoff is that BPE almost never needs an unknown token at all. Instead of giving up on a word it hasn't seen, it breaks that word down into pieces it does know. I tested this directly using tiktoken, OpenAI's own BPE implementation, on a sentence containing the made-up word "someunknownPlace":

tokenizer.decode([617])        →  "some"
tokenizer.decode([34680])      →  "unknown"
tokenizer.decode([27271])      →  "Place"

Nobody trained this tokenizer on the word "someunknownPlace." It doesn't need to have seen it. It just recognized "some," "unknown," and "Place" as pieces it already knows and stitched them back together. That's a genuinely elegant way to sidestep the unknown-word problem, and it's a big part of why <UNK> tokens have mostly disappeared from modern tokenizers.

From here on, tiktoken's GPT-2 encoding is what the rest of the project uses. Encoding the full test corpus with it produced 5,145 tokens, and decoding the first 50 gave back exact, readable text.

Turning Tokens Into Training Examples

Tokenizing text gets you a long flat list of integers. A model can't train on that whole list at once, so it needs to be sliced into fixed-size chunks, each with an input and a target.

The target is simply the input shifted one position to the right. Given four tokens of context, the model is trained to predict the fifth:

x: [290, 4920, 2241, 287]
y: [4920, 2241, 287, 257]

or, in decoded form, something closer to how it actually reads:

"and"                          → "established"
"and established"              → "himself"
"and established himself"      → "in"
"and established himself in"   → "a"

That's the entire training signal for a language model. Predict the next token, given everything before it.

To generate many of these training pairs from one long sequence of tokens, you slide a fixed-size window across the data, moving it forward by some number of tokens each time. The window size is the context window. How far it moves each step is the stride. A stride equal to the context window means no overlap between examples. A smaller stride means more overlapping examples, which gives you more training data from the same text at the cost of more redundancy.

I'll be honest, this is the part where I slowed down the most, and it surprised me. I've used PyTorch DataLoaders plenty of times before this project, so I expected this to be a five-minute step. It wasn't, not because the code is hard, but because keeping the relationship between context window, stride, and the resulting input/target pairs straight in my head took real focus. The code itself, a Dataset subclass that slices token_ids into overlapping chunks and a DataLoader wrapping it, is short. Understanding exactly what shape of data comes out the other end, and why, took longer than I expected.

With batch size 8, context length 4, and stride 4, the first batch out of the loader looked like this:

Inputs:
tensor([[   40,   367,  2885,  1464],
        [ 1807,  3619,   402,   271],
        [10899,  2138,   257,  7026],
        ...])

Eight independent sequences of four tokens each, ready to be fed into the model as a batch, each one paired with its own shifted target.

From Token IDs to Vectors

Token IDs are just integers. They don't encode any relationship between tokens; ID 8912 isn't "more" than ID 1325 in any meaningful sense, they're just row indices. The next step is turning each ID into a dense vector using an embedding table, a matrix of shape (vocab_size, embedding_dim) where each row is the learned vector for one token.

This is a lookup, not a matrix multiplication. Token ID 2 always retrieves row 2 of the embedding matrix:

embedding_layer(torch.tensor([2]))
# tensor([[ 1.2753, -0.2010, -0.1606]])

embedding_layer.weight[2]
# tensor([ 1.2753, -0.2010, -0.1606])

Same values either way, because that's literally what the embedding layer does under the hood, index into its own weight matrix.

Adding Position

Here's the catch. Token embeddings alone carry no information about order. "Dog bites man" and "man bites dog" would produce the exact same set of token vectors, just assigned to different token IDs, with nothing telling the model which one came first. Self-attention, which we'll get into properly in the next post, has no built-in sense of sequence order either. Left alone, it treats the input as an unordered set.

The fix is positional embeddings: a second embedding table, this one indexed by position rather than by token identity, added directly on top of the token embeddings.

token_embeddings = token_embedding_layer(inputs)      # [8, 4, 256]
pos_embeddings = pos_embedding_layer(torch.arange(4))  # [4, 256]
input_embeddings = token_embeddings + pos_embeddings   # [8, 4, 256]

Position 0 always gets the same positional vector, regardless of which token happens to sit there. Add that to the token's own embedding, and you get a vector that encodes both what the token is and where it sits in the sequence. That combined tensor, shape [8, 4, 256] in this case, is what actually enters the transformer.

Results

Running the full pipeline end to end on real text:

• Vocabulary built from the test corpus: 1,130 tokens (word-level), later replaced by GPT-2's BPE vocabulary via tiktoken
• Full corpus encoded with BPE: 5,145 tokens
• Token embedding shape for a batch of 8 sequences, context length 4, embedding dim 256: [8, 4, 256]
• Positional embedding shape: [4, 256], broadcast and added across the batch
• Final input embedding shape: [8, 4, 256], ready for the transformer

Nothing in this pipeline has learned anything yet. The token and positional embeddings at this stage are randomly initialized; training is what will eventually shape them into something meaningful. What we've built is the on-ramp: a deterministic, well-defined path from raw text to the exact numerical shape a transformer expects as input.

Key Takeaways

• Tokenization exists because neural networks operate on numbers, not characters or words
• A word-level tokenizer has no way to handle a word it's never seen. It either crashes or falls back to a generic unknown token, both of which lose information
• BPE sidesteps this by decomposing unseen words into subword pieces it already knows, which is why modern tokenizers rarely need an <UNK> token at all
• The training signal for a language model is just the token sequence shifted by one position: predict what comes next
• Context window and stride control how many overlapping training examples you generate from a fixed amount of text
• Token embeddings encode identity. Positional embeddings encode order. The model needs both, since self-attention alone doesn't know what order anything came in

What's Next?

Everything up to this point has been preprocessing. Necessary, but the model hasn't actually looked at any relationships between tokens yet.

Next post, we get into self-attention: how a token figures out which other tokens in the sequence actually matter to it, and why Query, Key, and Value exist as separate concepts instead of just one. This is the mechanism the entire transformer architecture is built around, and it's the flagship post of this series.

Resources

• The Crucible on GitHub, full code for this project
• Build a Large Language Model (From Scratch) by Sebastian Raschka
• tiktoken, OpenAI's BPE tokenizer implementation
• Attention Is All You Need (2017)