The Crucible #1: Why I'm Building an LLM From Scratch

Building an LLM from first principles to understand what happens between the input and the output.

AI Has Changed How We Build Software

Anyone can access incredibly capable language models through a simple API call. In just a few years we've gone from experimenting with chatbots to building copilots, AI agents, retrieval-augmented generation (RAG) systems, coding assistants, and intelligent automation.

Building with AI has never been easier.

But the easier it became to use these models, the more one question stayed in the back of my mind.

What actually happens between the input and the output?

That question is the reason this series exists.

Why The Crucible?

I've been working in AI for nearly two years.

During that time I've built AI-powered applications using large language models, retrieval-augmented generation, AI agents, automations, and traditional machine learning. I've integrated models like GPT, Claude, and Gemini into real products and solved practical problems with them.

Yet despite all of that, I realized something surprising.

I knew how to use these models.

I didn't truly understand how they worked.

I could prompt them, integrate them into applications, design AI workflows, and evaluate their outputs. But if someone asked me to explain, step by step, how a prompt becomes a response, I couldn't confidently answer.

Somewhere between the input and the output there was still a black box.

That realization became the starting point for The Crucible.

The Crucible is my journey to understand large language models from first principles by building one myself. Not because I want to compete with GPT or Claude, and not because I'm trying to train the next state-of-the-art model. I'm building something small and understanding every part of it, because building something yourself forces a level of understanding that reading documentation or calling APIs never can.

Instead of treating modern language models as magic, I want to understand every major component that makes them work. One layer at a time. One concept at a time. One line of code at a time.

What Is a Large Language Model?

Before building one, it's worth understanding what we're actually building.

A Large Language Model (LLM) is a deep neural network trained to predict the next token in a sequence.

That objective sounds surprisingly simple. Given some text like:

"The sky is..."

the model predicts what token should come next.

During training it repeats this prediction billions of times across enormous amounts of text. There's no separate step where it's taught grammar or facts or how to reason. All of that emerges as a side effect of getting extremely good at one narrow task: predicting the next token. That gap between the simplicity of the objective and the complexity of what comes out of it is a big part of what this series is trying to unpack.

Most modern language models, including GPT, Claude, Gemini, Llama, Qwen, DeepSeek, and Mistral, are built on the Transformer architecture, introduced in the paper Attention Is All You Need in 2017. Throughout this series we'll build many of the same core components ourselves instead of treating them as a black box.

Figure 1: The journey from using AI models to understanding how they work.

Why Build One From Scratch?

At this point you might reasonably ask:

"Why build an LLM from scratch when amazing models already exist?"

It's a fair question. There are already excellent books, videos, research papers, and open-source implementations explaining how transformers work. So why spend weeks rebuilding one?

Because there's a huge difference between recognizing an idea and truly understanding it.

Reading about self-attention is valuable. Implementing self-attention yourself is different. Watching your model fail because of one incorrect matrix multiplication teaches something no tutorial can.

By building every major component ourselves, we'll understand not only what each part does, but why it exists.

My hope is that this journey will also make me a better AI engineer. Understanding the internals of language models helps you make better architectural decisions, recognize their limitations, debug unexpected behavior, and use them more effectively in production systems.

Figure 2: The roadmap we'll follow throughout The Crucible.

How We'll Build It

Rather than trying to understand everything at once, we'll build the model incrementally. Each phase focuses on one major concept.

Phase	Focus
0	Foundations & Repository Setup
1	Tokenization
2	Embeddings
3	Self-Attention
4	Multi-Head Attention
5	Transformer Blocks
6	Assembling GPT
7	Training
8	Inference & Sampling
9	KV Cache & Quantization
10	LoRA Fine-Tuning

By the end of the core phases we'll have a working miniature GPT model, generating text entirely from our own implementation. More importantly, we'll understand why every major component exists and how they fit together.

Learn. Build. Share.

Everything in The Crucible follows three simple principles.

Learn. Study one concept deeply enough to explain it in plain English. Not memorize it. Understand it.

Build. Implement it from scratch instead of copying existing implementations. Write the code. Break it. Fix it. Repeat.

Share. Document everything along the way — the successes, the mistakes, the concepts that finally click. If this journey helps someone else understand language models more deeply, then it will have achieved more than simply producing another GitHub repository.

What to Expect

This isn't a tutorial series promising perfect explanations. It's a public learning journey.

I'm treating this as a public engineering notebook. I'll document the ideas that click, the implementation decisions I make, and the mistakes I encounter along the way.

And that's exactly the point. The goal isn't perfection. The goal is understanding.

What's Next?

In the next post we'll begin with the very first building block of every language model: how language becomes numbers.

We'll explore tokenization, why computers can't understand raw text, and build our own tokenizer before writing a single line of transformer code.

The journey starts there.

The repository, code, notes, and every article in this series will evolve together. If you're interested in understanding language models from first principles, I'd be glad to have you follow along.

Resources

Build a Large Language Model (From Scratch) — Sebastian Raschka
Attention Is All You Need (2017)
Andrej Karpathy — Let's Build GPT: From Scratch

If you're interested in understanding large language models from first principles, follow along. Every post in this series will build on the previous one as we gradually transform a blank repository into a working GPT-style language model.