P6-2.2 Model Input: Source Strings, Tokens, and Token IDs¶

Section ID: P6-2.2 Version: v2026.07.26

In P6-2.1, we held onto the token as a computational unit for explaining length, cost, truncation, and context preservation. Now we need to read how that computational unit appears in tokenizer output and model input.

The center of this section is not explaining the word token broadly. It is to distinguish the token pieces, token count, and token IDs visible in tokenizer output, and to hold onto the fact that those IDs become a sequence used as model input.

Values to Distinguish First in Tokenizer Output¶

A token is the basic computational unit a model counts when handling inputs and outputs. But in an actual output screen or log, a token does not appear as only one value. The source string, token pieces, token count, and token IDs often appear together, and if these values are mixed together, it becomes unclear how model input is made.

Tokens are not decided as meaning units that are convenient for people to read. They are determined by the vocabulary and splitting rules of a particular model's tokenizer. So even for the same source text, token pieces and token counts can change when the model or tokenizer changes.

The minimum distinction needed here is as follows.

This table is the standard for this section. First read separately how the source was split, how many pieces there are, and which number each piece became, and then look at how those numbers are placed in sequence as model input.

Misunderstandings to Drop First¶

Depending on the case, a token can look like the following.

• One word
• Part of a word
• A short piece including a space
• A piece mixed with numbers and symbols
• A piece attached to punctuation

In other words, a token is not always the same as the intuitive word a person feels. Something that looks like one word can be split into several tokens, and a part people do not easily see as a word, such as a piece with a leading space, can be included in a token boundary.

The difference becomes clearer in the following table.

This difference often appears especially in the following scenes.

The important point is not that the split shown above appears exactly the same in every model. The actual split differs by tokenizer. But whatever result appears, the reading standard is the same. Look at the source first, then the boundaries of token pieces, and finally the token count and IDs.

Another misunderstanding is reading a token like the symbol from symbolic AI seen in Part 1. A symbolic-AI symbol was closer to the name of a meaning or rule chosen by people. By contrast, an LLM token is a computational piece made by a tokenizer splitting the source text, and the model turns that piece into an ID and vector for probabilistic computation.

Therefore, we should not overinterpret tokens as logical symbols. On the other hand, we should not pass over token IDs as meaningless internal numbers either. What is needed now is not reading token pieces as definitions, but reading which vocabulary number each piece becomes and where it is placed in the model input.

How Tokens Are Used Inside the Model¶

Tokens are not computed as-is. A model usually handles input in the following flow.

flowchart TD
  A["source string"]
  B["token pieces<br/>piece boundaries"]
  C["token count<br/>number of pieces"]
  D["token ID sequence<br/>vocabulary numbers"]
  E["embedding vectors<br/>calculation input"]
  F["model computation"]

  A --> B
  B --> C
  B --> D
  D --> E
  E --> F

Two points matter in this flow.

• The model does not compute the source string directly. It first converts it into token pieces and a token-ID sequence.
• Token IDs are then looked up as vector representations and enter the actual computation.

In other words, a token is closer to the entrance where model computation starts than to a finished meaning unit by itself.

This flow matters from an engineering perspective because the values people can actually inspect also change along this flow.

So knowing what a token is does not mean memorizing one definition. It means being able to distinguish what you are reading later when you see tokenizer output or logs.

When reading token IDs, one more caution is needed. An ID is closer to an identifier saying which entry in the vocabulary this piece is. If 4012 is larger than 812, that does not mean the token is more important or more complex. What matters in model computation is not the size of the ID itself, but which embedding vector that ID retrieves and how it enters later context computation.

Output-Value Distinction Cases and Examples¶

Case 1. When Visible Words Differ from Tokens¶

Suppose the expression refundpolicy keeps appearing in a customer-support document. A person reads this expression as one word with a clear meaning. So it is easy to think the model also receives this one word as one computational unit.

But in model input, whether a person feels it is one word is not the standard. In tokenizer output, we must check whether this expression remains as one piece, splits into several pieces, and which ID is attached to each piece. Especially in sentences where endings, particles, or closed-up expressions are mixed, the word boundary people see and the tokenizer boundary can easily diverge.

The result to check in this case is that a token is not always the same as the word a person feels. Even when looking at the same expression, source string, token pieces, token count, and token IDs must be read separately.

Case 2. When One Sentence Becomes A Token Sequence¶

A person reads The meeting is tomorrow at 10:00 AM as one schedule sentence. By human standards, one sentence, one meaning, and one time value appear first.

But the model does not place the whole sentence into computation at once. It first converts the source string into a sequence of token pieces, counts those pieces, attaches IDs to each piece, and then prepares for computation.

At first, separating only the following four levels is enough to reveal the difference between a sentence and computational input.

The result to check in this case is that the model first receives not the meaning of the whole sentence, but a sequence of pieces that can be numbered. What matters more than the exact token count is distinguishing which level of value you are looking at. With this distinction, when model input is read as a sequence of token IDs, the values do not feel like internal implementation details that suddenly appeared.

Case 3. When Tokens Are Not Computed Directly¶

Suppose a log prints token pieces and token IDs together. At first, token pieces alone can seem sufficient. Because the pieces are visible, it is easy to feel that the model computes those character pieces themselves.

But token pieces are displays that people can inspect easily, and when the input moves into model computation, each piece needs a corresponding token ID. That ID must again become a vector representation to enter actual computation. In other words, the token piece is an easy-to-observe marker, while the token ID is closer to a key used to look up an embedding in the model's vocabulary.

The same flow can be shortened as follows.

The result to check in this case is that token pieces and token IDs should not be treated as the same thing. Token pieces show how the source was split, and token IDs show which numbers the pieces are handled as inside the model.

The same scene can be compressed once more into a comparison table.

Practice Asking at the Right Level¶

Even if we cannot run an actual tokenizer yet, we can practice changing what people first say into a model-standard checking question as follows.

The important point in this table is not guessing the exact values in advance. What is needed first is distinguishing whether what you are seeing now is a word definition, a computational piece, a piece count, or a numbered input.

These are exactly the levels that often get mixed together.

• It is easy to feel that model computation has already started just by seeing the source string.
• It is easy to accept token pieces as word definitions.
• Token IDs can look so internal that it is easy to pass over them as meaningless.
• Because tokens look like markers, it is easy to read them at the same level as symbolic-AI symbols.

To read model input as a sequence of token IDs, these levels must first be distinguishable.

When the same distinction is applied to an actual log or explanation screen, read it as follows.

This table is a device for pausing again inside the current integrated section to check the distinction among piece, count, number, and position. If readers pass through this table, they can read the levels between tokenizer output and model input without seeing a token as an isolated name.

Example: Reading Tokenizer Output Values¶

Before comparing tokenization types, it is better to first check how to read the token itself. The example below simplifies three values often seen in real tokenizer output. It is not a table for memorizing the exact result of a particular model, but an example for distinguishing what to read from output.

• Token pieces
• Token count
• Token IDs

For example, suppose a tool outputs the following format. The token pieces and ID numbers below are simple examples for conceptual explanation. In an actual model tokenizer, piece boundaries and IDs can differ.

The result readers should immediately read here is as follows.

When reading this table, judge as follows.

• Token pieces show the boundaries where the source was split.
• Token count shows how many computational pieces the source became.
• Token IDs are not meanings read by people, but vocabulary lookup results.
• Even for the same source, token pieces, token count, and IDs can change when the tokenizer changes.

Read one English word in the same way.

The point to check here is that the human sense of one English word and the model sense of two computational pieces can differ.

Conversely, a short symbol or punctuation mark can also become a separate token. For example, a period . or colon : looks like an additional mark to a person, but it can appear as an independent piece in tokenizer output. So when looking at token count, first ask how many pieces the tokenizer actually made, not how many important words there are.

The goal of this example is not to memorize the exact numbers from a particular tokenizer. What is needed first is to clearly distinguish that in tokenizer output, what I am seeing now is not a word definition but a computational piece, count, and number.

Tokenizer output can be closed with the following three questions.

The judgment to close in this example is not memorizing what a token is like a dictionary entry. It is enough to distinguish what piece, count, and number mean in tokenizer output.

How A Token-ID Sequence Becomes Model Input¶

Earlier, we read tokens separately as token pieces, token count, and token IDs. Now we need to see how those values are used in model input.

The key is not the definition of one token but the sequence of tokens. The model does not receive the source sentence as a whole. It receives input where token IDs are lined up, and it continues the next computation on top of that order.

Sequence Matters More Than One Token¶

Tokens are not used only one at a time. In model input, multiple tokens are placed in order, and that order becomes material for the structure of the sentence.

Without token order, expressions with similar pieces, such as refund possible and refund not possible, are difficult to distinguish reliably. The model must read not only what each token is, but also which tokens were before and after it.

Here, sequence does not simply mean lining up tokens. Each token has its own position in the input, and that position is interpreted together with the tokens before and after it. Even the same token ID can play a different role in the next computation depending on what token came before and what token follows.

Written very small, input can be read as follows.

What matters in this table is not memorizing ID numbers. It is that even with the same piece list, we must see where each piece is placed and which pieces are attached before and after it to read model input.

Token IDs Are Number Sequences That Enter Computation¶

The token IDs seen above are not simple labels. The model receives the sequence of token IDs rather than the string itself, then turns those IDs into vector representations and computes with them.

Written very simply, the flow is as follows.

The important point here is not explaining tokenization rules in detail yet. It is first holding onto the structure that the model does not directly read the whole sentence, but receives a token-ID sequence as input.

More carefully, when reading model input, we look at three things together.

Therefore, model input is neither just an ID list nor just a word-meaning list. It is a structure that sees together which ID is placed in which order and position.

ID-Sequence Cases and Examples¶

Case 1. When Input Changes Because Same Pieces Change Order¶

Suppose we automatically classify refund possible and refund not possible in a customer-support document. A person immediately distinguishes the two expressions by meaning, but a reader seeing tokenizer output for the first time may feel they are similar inputs because both contain refund and possible.

The limit of that standard is that it sees tokens like a list of included pieces. In model input, what matters is not only which pieces are included, but where each piece is inserted. If a negating piece such as not appears before possible, the input sequence changes even though it contains the same refund and possible pieces.

The result to check in this case is that tokens should be read not as a set collected together, but as a number sequence continuing from front to back. The numbers are examples for explanation.

So the safer judgment in this case is not stopping at which tokens are included, but also checking in what order those tokens are placed.

Case 2. When Same Token Repeats but Position Differs¶

The input The meeting is tomorrow. Please send the meeting materials. contains meeting twice. A person may first see the same word repeated, and tokenizer output may also show the same token piece or the same ID twice.

But the fact that the same ID appeared twice is not enough to say the two positions play the same role. The first meeting appears in a position opening a schedule notice, while the second meeting appears before the request materials. Even the same token can be read differently in model input when its position and surrounding tokens differ.

The result to check in this case is that model input is not a list of token types but a token sequence with positions.

So the safer judgment in this case is to look first not at is it the same token, but at where the same token is placed in the input and with which surrounding pieces.

Case 3. When Output Tokens Continue After Input Tokens¶

Suppose the user asks, Tell me in one sentence when refunds are not possible, and the model begins creating an answer. People usually see the answer sentence first as a completed artifact. That makes it easy to feel that the output is created all at once in sentence units.

The limit of that standard is that it sees input and output as disconnected chunks. When an LLM creates an answer, new output tokens continue after the input tokens already provided, and the output token just attached also becomes part of the previous context when the next output is chosen. What we need to hold here is not the details of the selection method, but the fact that input and output both continue on the same sequence.

The result to check in this case is that tokens are not markers used only when reading input. They are also the basic units by which output continues.

So the safer judgment in this case is not separating input as input, output as result, but reading that output tokens continue after input tokens and the same sequence becomes longer.

Reading Tokens as a Sequence¶

When you see the following scenes, first distinguish whether you are seeing tokens one by one or tokens as a sequence.

The application level needed here is not complicated internal model equations, but reading tokens as ordered input units rather than as individual names.

If you see the same judgment in an actual log or explanation screen, check in this order.

Standard to Keep from Token-ID Sequences¶

The token is not simply a concept word. It is the basic unit through which model input and output continue. The question to close now is not an operation judgment, but the fact that tokens are used inside the model as ordered input.

Once this flow is understood, the need for tokenization, the procedure that turns a source string into this token sequence, also follows naturally.

Exercises and Examples¶

The goal of this exercise is not guessing tokenization rules. It is practice in looking at given token pieces and IDs and reading model input not as an individual list but as a sequence with positions. For each item, first choose your own answer, then compare it with the explanation immediately below.

Exercise 1. Finding Order Differences in the Same Piece List¶

The two inputs below contain all the same pieces: refund, not, and possible.

First answer for yourself.

• Can the two inputs be treated as the same input?
• If they differ, which position difference appears first?
• Can this difference be explained only by a token-type list?

Explanation: The two inputs are not the same even though they contain the same pieces. In A, 920 is placed before 7731, and in B, 920 is placed at the end. By the standard of this section, which ID is placed in what order comes before what is included.

Exercise 2. Reading the Position of the Same Token¶

Below is a simplified input where the same token ID 5100 appears twice.

First answer for yourself.

• Since the two meeting tokens have the same ID, can we say they play the same role?
• To read the two positions differently, what value must we also see?
• What disappears if this input is reduced to a token-type list such as {meeting}?

Explanation: Even if the same ID repeats, we cannot conclude that it plays the same role. The two tokens are in different positions, and the pieces that follow them also differ. If we reduce the input to a token-type list, position and surrounding context disappear, so the key information needed to read model input is lost.

Exercise 3. Tracking Where Output Tokens Attach¶

The table below simplifies the process where output tokens continue one by one after a question.

First answer for yourself.

• Is the second output token product a new sentence chunk separated from the question?
• What is included in the earlier sequence the model refers to when the second output attaches?
• What is missed if the output process is read only as one completed sentence appears?

Explanation: product is a new token continuing after the earlier question and first output token. At the moment the second output attaches, not only the question tokens but also the already generated damaged are part of the previous context. So output should be read not as a completed sentence appearing at once, but as the existing sequence growing token by token.

After the three exercises, you should be able to summarize the point in one sentence.

Model input is not a collection of token types, but a number sequence where token IDs continue with order and position.

Checklist¶

• Can you explain that model input is not a whole sentence but a token-ID sequence?
• Can you distinguish token pieces, token count, and token IDs as different values in tokenizer output?
• Can you say that surrounding order and position matter more than the name of one token?
• Can you explain that answers are made as output tokens continue after input tokens?

Sources and References¶

• OpenAI Help Center, What are tokens and how to count them?, accessed 2026-07-19. Used to confirm that API request text is split into tokens and that responses are generated as token sequences before being converted into text.
• OpenAI, tiktoken README, accessed 2026-07-19. Used to confirm the explanation that language models see text as a sequence of token numbers rather than in the way people read it.
• Daniel Jurafsky, James H. Martin, Speech and Language Processing, 3rd ed. draft, online manuscript released January 6, 2026, accessed 2026-07-19. Used as background support for token sequences and language-model input in the Words and Tokens and Large Language Models chapters.