P6-2.3 Tokenization Changing Length, Cost, and Chunks¶

Section ID: P6-2.3 Version: v2026.07.26

In P6-2.2, we separated token pieces, token count, token IDs, and the token-ID sequence in tokenizer output. Now we move one step beyond how to read it and look at how tokenization results change actual judgments.

The center of this section is not rereading tokenizer output values again. It is to hold onto the fact that the moment a source string changes into certain boundaries and an ID sequence, input length, cost, chunk boundaries, and output-preservation judgments change together.

What Procedure Is Tokenization?¶

Once we know that tokens are computational units, we need to see how a string actually becomes those computational units. Tokenization is the procedure that changes source text into a token sequence read by the model.

The characters people see are not computed as-is. We first need to see the flow where a tokenizer sets boundaries, chooses pieces in a vocabulary, and changes those pieces into IDs.

Why Tokenization Is Needed Separately¶

People see sentences as characters, words, and sentences. But models do not use these units directly for computation. First, the input must be changed into computable pieces, and those pieces must be numbered and passed into computation.

So tokenization is not an add-on feature. It is the first gate from source string -> token sequence -> token ID.

The important point here is that tokenization is not simple whitespace splitting. A tokenizer looks at the source, makes possible piece boundaries, checks whether those pieces are in the vocabulary, and finally creates the ID sequence the model will receive. So when reading tokenization results, it is safer to ask the following three questions in order.

What Tokenization Actually Does¶

In tokenization, the following usually happen together.

• It decides boundaries for splitting the source string.
• It connects the split pieces to a token vocabulary.
• It changes each piece into a token ID and passes it to model computation.

Something that looks like one word to a person can be split more finely, and parts people do not strongly notice as meaning units, such as spaces or punctuation, can affect token boundaries.

Written more step by step, the same procedure looks like this.

In other words, tokenization does not end at cutting a string. It includes which piece will be treated as a vocabulary item and which ID will be passed for that piece.

The same procedure becomes clearer in a small example. The values below are not actual tokenizer output; they are explanatory examples for showing the reading order.

What this table asks us to check is not the numbers themselves. During the change from source -> token pieces -> token IDs, the word boundaries a person felt are reset as computational boundaries made by the tokenizer.

Drawing the Tokenization Procedure in One Line¶

flowchart TD
  A["Source text"]
  B["Split into tokens"]
  C["Model reads the token sequence"]
  D["Predict next token"]

  A --> B
  B --> C
  C --> D

In this flow, tokenization is the stage that changes a string into a token sequence and token IDs. Only after that are IDs looked up as vector representations and connected to computation.

Tokenization-Procedure Cases and Examples¶

Case 1. When We Trust a Word-Looking Unit as Is¶

Suppose the expression refundpolicy repeats in a customer-support document. A person quickly reads this as one meaningful word. So it is easy to think the tokenizer will also pass this string as one piece.

The limit of that standard is that it treats the word boundary a person feels as the same as the tokenizer boundary. In tokenization, refundpolicy may remain as one vocabulary item, but it may also split into refund, policy, or smaller pieces.

The result to check in this case is not Does it look like a word?, but At what boundary is the source split, which vocabulary item is that piece connected to, and what ID sequence does it become? The numbers are explanatory examples.

So the safer judgment is not guessing token count or IDs immediately from the source, but checking the boundary, vocabulary pieces, and ID sequence made by the tokenizer in order.

Case 2. When Spaces Seem to Mean Same Units¶

When people see a sentence such as The meeting starts tomorrow at 10 AM, they first think of words separated by spaces. So it is easy to feel that the number of whitespace-separated words will be similar to the token count.

The limit of that standard is that a tokenizer does not split only by spaces. Numbers, endings, particles, and time expressions can make different piece boundaries depending on the vocabulary and splitting rules. 10시에 may remain one bundle, or it may split into 10, 시, and 에.

The result to check in this case is that tokenization is not whitespace splitting, but a procedure that resets boundaries for numbers, particles, endings, and other elements outside spaces too.

So the safer judgment is not converting whitespace word count into token count, but looking at the piece boundaries actually made by the tokenizer and the ID sequence corresponding to those boundaries.

Case 3. When the Sentence Seems Computed Directly¶

When a user enters Summarize the refund policy, it is easy to imagine that the model reads the whole sentence and immediately grasps the meaning. But from the tokenization view, the string is first split into pieces, each piece connects to a vocabulary item, and then it changes into a token-ID sequence.

The limit of that standard is that it erases the conversion step between sentence input and computational input. If tokenization is skipped, we cannot distinguish whether what the model actually receives is the source string, token pieces, or token IDs.

The result to check in this case is that tokenization is not an auxiliary task, but the starting stage that passes a string into actual computational input. This starting stage includes not only the model received one sentence but also it chose pieces and changed them into an ID sequence.

How Far the Procedure Explanation Needs to Go¶

When first reading tokenization, it is enough to separate the unit people see and the unit the model passes into computation as follows.

If we stop here, tokenization can become only a procedure to memorize. The more important question in this section is the next one: once the pieces and IDs change that way, which of cost budget, chunk boundary, and output-preservation judgment must be checked again?

When looking at actual tokenizer output or logs, read not only the procedure itself but also the later judgment changes.

What Must Be Distinguished from What?¶

When understanding tokenization, it is safer to separate the following three levels.

Without this distinction, confusions such as is a token just a word? or does the model read the sentence as-is? repeat.

The important point here is not memorizing tokenizer-family names. It is reading the flow where the source string is not computed directly, but changes into pieces and IDs in the tokenization stage before moving into model input. This flow must be in place before we can understand why cost, chunks, and output length move together later.

What Tokenization Results Change¶

Once we know what tokenization is, we need to see what its result actually changes. The core is simple.

Tokenization is not simple splitting. It is a process that changes length, cost, chunks, and output interpretation together.

The value to see before tokenization rules themselves is the observed value that appears after tokenization. Even for the same source, token count can change, the position where context splits can change, and the room left for output can change. So after seeing how it is split, we should immediately ask what judgment must change because of that result?

Why the Same Sentence Can Differ¶

Even with the same meaning, token boundaries and token counts can change when notation changes.

• Use of abbreviations
• Number notation
• Whether special characters are included
• Mixing English and Korean

This is where the reason appears for why sentences that look similar to people can be read as different lengths and costs by the model.

If we unpack very small where the difference appears even with the same meaning, it looks like this.

The numbers are not actual tokenizer results, but explanatory values for showing the judgment flow. The key is not comparing sentence meaning, but first seeing which notation elements can increase computational pieces. If this step is missing, it is easy to move too quickly into the intuition that the sentences are similar, so the cost should be similar too.

When Token Boundaries Lead to Operation Judgments¶

flowchart TD
  A["Source text"]
  B["Tokenization"]
  C["Token count"]
  D["Cost and context usage"]
  E["Search / generation behavior"]

  A --> B
  B --> C
  C --> D
  B --> E

There is one result to check in this diagram. When token boundaries change, later service judgments change together.

What Actually Changes 1. Cost¶

Even a sentence that looks short can make the actual token count grow quickly when numbers, symbols, emails, or URLs are mixed in. So the intuition that it is a short inquiry, so the cost should also be small often fails.

What Actually Changes 2. Context Length and Chunks¶

When splitting documents, people first look at paragraph count or rough visual length. But by the tokenization standard, principle and exception, or question and condition, can split into different pieces. Then retrieval may bring back only the principle and miss an important exception.

What Actually Changes 3. Output Length¶

The longer the response format becomes, the more output tokens are used. When tables, lists, code blocks, and additional explanations increase, the risk that the final key sentence is cut off also grows.

Scenes Where Operation Judgment Shifts Immediately¶

Tokenization can look like a preprocessing detail, but it directly shifts operation judgments in practice.

• A sentence that looks short can actually be more expensive.
• A visually clean paragraph split can be bad for retrieval.
• A friendly output format can cut off the final key sentence.

In other words, tokenization differences do not end at how a string is split. They lead to operation choices about what to preserve and what to give up.

Operation-Judgment Cases and Examples¶

The diagram below first groups the three scenes we repeatedly see. The reading flow is why it looks fine on the surface -> what changes from the tokenization view -> where operation judgment changes.

flowchart TD
  A["Same tokenization question"]
  B["Cost estimate<br/>Does the token count grow faster?"]
  C["RAG chunking<br/>Does the context stay together?"]
  D["Response length limit<br/>Does the critical ending remain?"]

  A --> B
  A --> C
  A --> D

The diagram alone can still leave what failure actually appears somewhat abstract, so the three cases below read the same scenes more slowly.

The same scenes can be grouped very briefly again as follows.

Case 1. When A Short Inquiry Changes Cost Judgment¶

A sentence such as The meeting is tomorrow at 10:00 AM. The Zoom link was sent to mail@example.com looks short on the screen. But when numbers, a colon, English text, and an email address are mixed in, the actual token count can grow faster than expected.

The problem scene in this case is the moment we judge that it is a short schedule notice, so the cost should also be small. The standard people first use is the number of visible lines and characters. The limit of that standard is that it misses elements a tokenizer may split more finely, such as numbers, special characters, English text, and emails. So the judgment changed by tokenization moves from is the character count short? to how many actual computational pieces are there?

If we turn it into small observed values, the judgment difference becomes clearer. The numbers are explanatory examples.

The sentence to close in this case is this: without seeing the tokenization result, the first judgment that it looks short can incorrectly pull cost judgment along with it.

Case 2. When Retrieval Results Change Chunk Judgment¶

If annual leave must be requested three days in advance and emergency sick leave may be reported afterward split into different chunks in a policy document, retrieval may bring back only the principle and miss the important exception.

The problem scene in this case is the moment we judge that retrieval succeeded because the search result brought back the principle document. The standard people first use is whether the related document appears in the search result. The limit of that standard is that it does not check whether the principle and exception needed for the answer remain in the same token bundle. So the judgment changed by tokenization moves from is the search result relevant? to does the context that must remain together stay in the same chunk?

The same scene can be read as observed values as follows.

The sentence to close in this case is this: without seeing chunk boundaries after tokenization, the judgment that retrieval worked can hide missing exception conditions.

Case 3. When A Friendly Format Changes Output Preservation Judgment¶

If an answer that could end as one paragraph explaining a shipping delay keeps adding tables and caution lists, the final refund-condition sentence may be cut off by the output limit. So we must separate friendly format from the core that must remain to the end.

The problem scene in this case is the moment we judge that adding tables and lists makes the answer friendlier. The standard people first use is whether the answer looks clean and sufficiently detailed. The limit of that standard is that table headers, separators, and repeated phrases all use output tokens. So the judgment changed by tokenization moves from is the format friendlier? to does the sentence that must remain fit within the output limit?

On the output side too, tokenization results appear as observed values.

The sentence to close in this case is this: without seeing where output tokens are used after tokenization, a friendly format can push out the final key condition.

Now tokenization results can be read as observed values that change the three judgments of cost, chunks, and output preservation.

Judgments That Change After Tokenization¶

To hold onto what tokenization actually changes, it is useful to place surface change and operation-judgment change side by side.

The key application in this section is reading tokenization difference not as a preprocessing detail, but as something connected immediately to cost, chunk, and output choices.

Written again so readers can apply the same judgment by hand, it looks like this.

Without this middle application table, tokenization explanation easily stops at how strings are split. But the goal of this Section is not memorizing cutting rules. It is changing input budget, chunk boundary, and output-preservation judgments after seeing the cutting result.

Exercises and Examples¶

The exercise below is not a problem of guessing token count exactly. First, use an actual tokenizer SDK to check how many tokens the input becomes, and then move that value into cost, chunk, and output-preservation judgments. For each question, first answer for yourself, then compare with the explanation immediately below.

Example. Checking Input Budget and Output Room with tiktoken¶

This example directly counts input tokens with OpenAI's tiktoken library under the same encoding. It is not an example for memorizing the latest context length of a particular model, but for seeing how operation judgment changes when input tokens and expected output tokens are added together. Here we use the o200k_base encoding.

The value the actual tokenizer calculates here is input_tokens. expected_output_tokens, token_budget, and chunk_size are operation assumptions set by the service designer. This distinction prevents the misunderstanding that the SDK automatically makes every judgment. The tokenizer tells us how many pieces the input became, and the person connects that value to output room and chunk room judgments.

The values to manipulate directly are text, expected_output_tokens, token_budget, and chunk_size inside samples. The values to look at first in the result are input_tokens, remaining_tokens, and chunk_margin.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

# Count real input tokens with tiktoken and connect them to cost, chunk, and output-room judgments.
import tiktoken

encoding = tiktoken.get_encoding("o200k_base")

samples = [
    {
        "case": "plain_notice",
        "text": "회의는 내일 열립니다.",
        "expected_output_tokens": 40,
        "token_budget": 120,
        "chunk_size": 80,
    },
    {
        "case": "mixed_schedule",
        "text": "회의는 내일 10:00 AM에 열립니다. Zoom 링크는 mail@example.com으로 보냈어요.",
        "expected_output_tokens": 55,
        "token_budget": 120,
        "chunk_size": 80,
    },
    {
        "case": "policy_with_exception",
        "text": "연차는 3일 전 신청합니다. 단, 긴급 병가는 사후 보고가 가능하며 증빙을 첨부해야 합니다.",
        "expected_output_tokens": 70,
        "token_budget": 120,
        "chunk_size": 30,
    },
    {
        "case": "verbose_output_request",
        "text": "배송 지연 사유를 표로 정리하고, 주의사항 목록과 환불 제한 조건을 마지막에 덧붙여 주세요.",
        "expected_output_tokens": 95,
        "token_budget": 120,
        "chunk_size": 80,
    },
]

for sample in samples:
    input_tokens = len(encoding.encode(sample["text"]))
    total_tokens = input_tokens + sample["expected_output_tokens"]
    remaining_tokens = sample["token_budget"] - total_tokens
    chunk_margin = sample["chunk_size"] - input_tokens
    print(
        sample["case"],
        "input_tokens=", input_tokens,
        "expected_output_tokens=", sample["expected_output_tokens"],
        "total_tokens=", total_tokens,
        "remaining_tokens=", remaining_tokens,
        "chunk_margin=", chunk_margin,
    )

The sample result can be read as follows. The output below was checked locally with tiktoken==0.13.0 inside .venv.

1
2
3
4

plain_notice input_tokens= 7 expected_output_tokens= 40 total_tokens= 47 remaining_tokens= 73 chunk_margin= 73
mixed_schedule input_tokens= 24 expected_output_tokens= 55 total_tokens= 79 remaining_tokens= 41 chunk_margin= 56
policy_with_exception input_tokens= 31 expected_output_tokens= 70 total_tokens= 101 remaining_tokens= 19 chunk_margin= -1
verbose_output_request input_tokens= 30 expected_output_tokens= 95 total_tokens= 125 remaining_tokens= -5 chunk_margin= 50

The key to read from this result is not one number but the movement of judgment. Input tokens are actual tokenizer results, while output tokens, budget, and chunk size are conditions readers can change.

If we see the number movement as a figure, it becomes visible that expected output format can eat into the total budget faster than long input.

The purpose of this example is not memorizing the tokenizer's internal rules. It is to check actual token count and then change human standards such as does it look short, are the paragraphs natural, and is the output friendly into input budget, chunk room, and output-preservation standards.

Exercise 1. Choosing the Judgment Value for a Short Notice¶

Observed values:

First answer for yourself.

• What value should be rechecked first in this scene?
• Is the screen standard of short notice enough?
• If input and expected output are added, how many tokens should the judgment use?
• Does this judgment connect first to cost, chunk, or output?

Explanation: The values to recheck first are input token count and the resulting cost. The input is 12 + 31 = 43 tokens, and with 50 expected output tokens, the total judgment value is 93 tokens. Even if it is one paragraph on screen, URL, coupon codes, and date notation can quickly increase token pieces. So this scene should first check how many tokens a short-looking input actually became, before chunks or output.

Exercise 2. Finding Why Exceptions Disappear from Search Results¶

Observed values:

First answer for yourself.

• What value should be rechecked first in this scene?
• How many tokens are needed at minimum to keep principle and exception in one chunk?
• Why can the answer be wrong even if the retriever found the principle sentence?
• Does this judgment connect first to cost, chunk, or output?

Explanation: The values to recheck first are chunk size and overlap. Principle 42 tokens and exception 18 tokens require at least 60 tokens together. Current chunk size is 50 and overlap is also 0, so they split easily. Even if the retriever found the principle sentence, the answer can miss an important condition if the exception sentence is not in the same token bundle. So this scene should first check whether context that must stay together remains in the same chunk, before cost.

Exercise 3. Finding Why the Final Conclusion Is Cut Off¶

Observed values:

First answer for yourself.

• What value should be rechecked first in this scene?
• Does the current output composition fit within the limit?
• Why can a friendly format become a failure cause?
• Does this judgment connect first to cost, chunk, or output?

Explanation: The values to recheck first are max output tokens and output format. The current composition is 35 + 32 + 18 = 85 tokens, so it exceeds the 80-token limit. Tables and lists look friendly, but separators and repeated phrases use output tokens first. So this scene should reduce the output format by the standard will the core restriction condition remain to the end, not will the answer look prettier.

After the three exercises, you should be able to summarize the point in one sentence.

Tokenization results are not preprocessing details, but observed values that make us recheck input cost, chunk context, and output preservation.

Checklist¶

• Can you say that sentences with the same meaning can have different token counts when notation changes?
• Can you explain that tokenization shakes cost, chunk design, and output length limits together?
• Do you understand that tokenization changes do not stop at string splitting but continue into operation judgment?

Sources and References¶

• OpenAI Help Center, What are tokens and how to count them?, accessed 2026-07-19. Used to confirm that input and output token counts connect to usage, cost, and request-length judgment.
• OpenAI Help Center, Controlling the length of OpenAI model responses, accessed 2026-07-19. Used to confirm that response length is controlled with output-token limits such as max_output_tokens or max_completion_tokens.
• OpenAI, tiktoken README, accessed 2026-07-19. Used as background support that tokenizer encoding results become observed values for service judgment even for the same text.
• Rico Sennrich, Barry Haddow, Alexandra Birch, Neural Machine Translation of Rare Words with Subword Units, ACL 2016, accessed 2026-07-19. Used as background support that subword units are used for rare-word and out-of-vocabulary handling.
• Daniel Jurafsky, James H. Martin, Speech and Language Processing, 3rd ed. draft, online manuscript released January 6, 2026, accessed 2026-07-19. Used as general NLP background for tokenization and word-boundary explanation in the Words and Tokens chapter.