AI Engineering from Scratch — Part 1
With this article, we’re starting a series on AI engineering — building reliable products and workflows on top of large language models (LLMs: the AI models behind ChatGPT, Claude, and similar assistants). We’ll go from fundamentals to practice: first how LLMs work under the hood, then how AI agents and tools work, and finally how to fit an agent into real development — from prompts and context all the way to deployment and monitoring.
We start at the very bottom of the stack. Send a model the same sentence in English and in Hindi, and the bill can differ several times over. Not because Hindi is “harder”, but because the model doesn’t see words at all. It sees tokens. They are the foundation for everything else — price, how much text the model can handle at once, speed.
A vocabulary sits between your text and the model
Inside, a language model is a huge mathematical function: numbers go in, numbers come out. It can’t read letters. So in front of it sits a tokenizer — a separate program with its own vocabulary. It cuts text into fragments and replaces each one with its number in the vocabulary. The model receives a list of numbers.
Think of a phrasebook where each common phrase or word part has a page number. Instead of handing over the text, you hand over the page numbers: “page 15339, page 2065”. The model only ever works with those numbers.
Here’s what that looks like in the cl100k_base tokenizer (used by GPT-4):
| String |
Tokens |
IDs |
| hello |
hello |
15339 |
| Hello |
Hello |
9906 |
| Hello (leading space) |
Hello |
22691 |
| HELLO |
HEL · LO |
51812, 1623 |
| He11o |
He · 11 · o |
1548, 806, 78 |
| tokenization |
token · ization |
5963, 2065 |
| unbelievable |
un · belie · vable |
359, 32898, 24694 |
One table shows several things at once:
- To the model, hello, Hello, and Hello are three different cases, and even a leading space changes the token.
- Common words are one token. Rare forms (all caps, typos) fall apart into pieces.
- A token is neither a word nor a syllable. It’s just a fragment that showed up often.
Where the vocabulary comes from: BPE in a nutshell
Nobody writes the vocabulary by hand — it’s “grown” automatically from a huge collection of texts by an algorithm called BPE (Byte Pair Encoding). The idea: find the two pieces that most often stand next to each other, glue them into one, and repeat.
Take a toy corpus: low, lower, lowest, slow.
- Start: everything is split into single symbols — l o w, l o w e r, l o w e s t, s l o w.
- The most frequent adjacent pairs are l o and o w (4 times each). Merge the first: lo w, lo w e r, …
- Now the most frequent pair is lo w. Merge: low, low e r, low e s t, s low.
- Repeat until the vocabulary reaches the target size.
In the end, low is one token and lowest is low plus something else. The algorithm knows nothing about grammar: if a token looks like a meaningful word part (like un- or -ization), it’s simply because that combination is common.
Vocabulary size is chosen by the model developers. A bigger vocabulary → longer fragments → fewer tokens for the same text. But the model gets bigger and needs more memory. That’s why vocabularies vary between models: tens or hundreds of thousands of tokens.
The non-English tax
BPE merges what is frequent. And English dominates the texts most models learn from (their “training data”) by a wide margin. The result is a systematic skew: English words more often become whole tokens, while text in other languages gets cut finer.
Add a second factor — bytes, the smallest units computers store text in. In the standard encoding (UTF-8), a Latin letter takes 1 byte, a Cyrillic or Greek letter takes 2, and a Hindi or Japanese character takes 3. The tokenizer starts from bytes, so more bytes means more pieces to glue together.
We ran the same sentence (about models reading text as tokens), translated into seven languages, through two OpenAI tokenizers — an older one with ~100K vocabulary entries and a newer one with ~200K. The numbers in brackets show how many times more tokens a language needs than English:
| Language |
Bytes |
cl100k_base (GPT-4) |
o200k_base (GPT-4o) |
| English |
103 |
20 (1.0×) |
20 (1.0×) |
| Spanish |
117 |
28 (1.4×) |
23 (1.15×) |
| German |
126 |
33 (1.65×) |
30 (1.5×) |
| Ukrainian |
222 |
67 (3.35×) |
42 (2.1×) |
| Japanese |
156 |
53 (2.65×) |
38 (1.9×) |
| Greek |
218 |
100 (5.0×) |
41 (2.05×) |
| Hindi |
285 |
118 (5.9×) |
36 (1.8×) |
What follows from this:
- Same meaning, different price. Even Latin-script languages close to English cost more, and languages with other scripts cost several times more.
- The vocabulary matters a lot. Going from ~100K to ~200K tokens took Hindi from 5.9× down to 1.8×. The bigger vocabulary made room for more non-English fragments.
- Numbers depend on the tokenizer. These are OpenAI tokenizers; Claude has its own, and the ratios will differ. The numbers show the scale of the effect, not the exact price for your model.
How to compare fairly: use the same meaning across languages, not two random texts of equal length. And don’t mix up two different questions: “language A vs language B on one model” and “one text across different models”.
The model doesn’t translate to English
A popular myth: “the model thinks in English and then translates”. It doesn’t. The tokenizer just cuts bytes — there’s no translation. After that, the model’s knowledge is stored as numeric patterns, not as words in any language. “Tree”, “árbol”, and “дерево” end up as very similar patterns, and none of them is the “original”.
But there’s more English data, so the model has “seen” more examples of English contexts. That’s why answers in other languages are sometimes noticeably weaker. It’s an effect of data volume, not hidden translation.
Your question is only part of the bill
The tokens in a request are not just the user’s text:
- System prompt — hidden instructions the app gives the model (“You are a helpful assistant, answer briefly…”). Sent with every request.
- Conversation history — the model doesn’t remember the chat by itself, so the whole conversation so far is re-sent with each new message.
- Tool descriptions. Modern assistants can use tools: search the web, read files, call other services (often connected via MCP, a standard way to plug tools into AI apps). Each tool comes with a description of what it does and how to call it, and all of that goes into the request too. Otherwise the model wouldn’t know what’s available.
- Structured data. In JSON (a common format for data exchanged between programs), quotes, braces, and field names are tokens too — and there are many of them.
- For Claude, roughly width × height / 750 tokens per image. A 1000×750 screenshot is about 1,000 tokens.
- PDFs and documents — text plus, usually, page images.
Measure, don’t guess
For developers. Not writing code? Skip to “Why it matters” — the takeaway is simply that token counts can be checked for free before you spend money.
Anthropic, the company behind Claude, offers a free way for programs to count tokens — POST /v1/messages/count_tokens in its API (the interface programs use to talk to the model). It takes the same format as a regular request: system, messages, tools, images, PDFs. In Python:
import anthropic
client = anthropic.Anthropic() # ANTHROPIC_API_KEY from the environment
def count(text: str, model: str = “claude-sonnet-5-5”) -> int:
return client.messages.count_tokens(
model=model,
messages=[{“role”: “user”, “content”: text}],
).input_tokens
en = count(open(“prompt_en.txt”).read())
other = count(open(“prompt_translated.txt”).read())
print(f”EN: {en}, translation: {other}, ratio: {other / en:.2f}”)
Caveats:
- Counting is free, but you need a Claude Console API key and active API access (a Pro/Max subscription doesn’t include it).
- input_tokens includes the request’s own formatting overhead, so ratios on a single short word are misleading. Use a paragraph or more.
- The tokenizer is tied to the model. For example, starting with Claude Opus 4.7 a new tokenizer is used, and the same text can produce noticeably more tokens. Switched models — recount.
Why it matters
Tokens determine three things:
- You pay for input and output tokens separately, and output is usually more expensive.
- Context window — the model’s working memory: the maximum number of tokens it can take in a single request. The “heavier” the language and the more tools you attach, the less useful data fits.
- More input — longer processing. Output is generated one token per step.
Checklist
Write system prompts and tool descriptions in English if it doesn’t hurt results, and ask for the answer in the language you need.
Don’t translate blindly: compare size, answer quality, and how easy the prompt is to maintain.
Run prompts, tools, and typical documents through count_tokens before running at scale.
Disable MCP servers and tools you don’t need — their descriptions eat context on every request.
Clean up noise: extra whitespace, all caps, typos, bloated JSON — and verify the effect by measuring.
Recount tokens when you switch models.
In short
- The model sees not text but the numbers of fragments from the tokenizer’s vocabulary.
- The vocabulary is built by BPE from the most frequent symbol pairs.
- Non-English text almost always costs more — from ~1.15× to several times, depending on the language and tokenizer.
- Questions aren’t the only thing that spends tokens: system prompt, history, tools, images.
- count_tokens is free — measure.
Next in the series
We’ve covered what a token is. The next step is what it costs. We’ll look at input and output tokens: how they differ, why the model’s answer costs noticeably more than your question, how to estimate a request’s cost up front, and how that gap shapes architecture decisions — from prompt length to model choice.