P4-5.2 Generalization¶

Section ID: P4-5.2 Version: v2026.07.24

P4-5.1 separated overfitting from underfitting. Now we need to move one step higher. Why do we care so much about that distinction? In the end, the goal of machine learning is not raising the score on training data, but working usefully even on data the model has not seen yet. The word that organizes that question is generalization.

Generalization may sound difficult, but the starting point is simple. It asks whether a model stops at matching examples it has already seen, or whether it also responds properly to new examples with similar structure.

Scope Of This Section¶

This Section explains the meaning of generalization. It does not yet cover the formula of generalization error or theoretical bounds. The focus here is to connect why new data matter, why validation and test are needed, and why training score alone is not enough.

The detailed calculation of metrics is handled in P4-6, and the practical use of cross-validation returns in P4-8.1 and P4-9.2. This Section fixes generalization first not as a mathematical theory, but as a goal sentence that runs through machine learning as a whole.

• What is generalization?
• Why is the goal of machine learning described as generalization rather than training score?
• What is the relationship between overfitting and generalization?
• What exactly does unseen data mean?
• How is generalization felt in practice?

Goals Of This Section¶

• You can explain generalization in one sentence.
• You can explain that training score and generalization are not the same thing.
• You can explain why overfitting can weaken generalization.
• You can understand that validation data and test data are ultimately tools for checking generalization.
• You can connect this Section to why later chapters on metrics and model choice are needed.

Learning Background¶

What Does Generalization Ask?¶

Google's machine-learning glossary effectively explains generalization as the question can the model make good predictions on examples that are not in the training set? The meaning to hold onto is the following.

Generalization is the property that lets a model make usable judgments even on data it has not seen yet.

The important part here is even though the data are not the same as before.

We also need to ask why such a word is used separately. Because machine learning learns rules from data, it is easy to leave only the sentence the score is high on the surface. But high only on the data it has seen and maintained even on the data it has not seen are entirely different questions. We use the word generalization to separate those two.

So generalization asks less whether the model adapted to this dataset and more whether it learned the structure of the problem to some degree.

This can be restated more simply like this.

• Did it memorize the shape of the dataset?
• Did it learn the structure of the problem?

Generalization is closer to the second side.

The Historical Background That Made Generalization Important¶

Generalization did not appear suddenly only in recent generative AI discussions. It is a question that has long been treated in the history of statistical learning theory. A survey paper by Ulrike von Luxburg and Bernhard Schoelkopf explains that statistical learning theory began in Russia in the 1960s and became widely known with the rise of support vector machines in the 1990s. The same paper describes the central question of the field as how can valid conclusions be drawn from empirical data?

This history can be read through the following flow.

1. Statistics and learning research long tried to separate rules that fit only the observed data from rules that also work on new data.
2. That question was handled more systematically inside statistical learning theory.
3. Later, in machine-learning practice, procedures such as validation, test, and cross-validation became standard habits for checking generalization.

So generalization is not a trend word. It is a long-standing criterion that separates was this really learning?

Main Learning Content¶

Why Generalization Becomes The Goal¶

Machine learning is almost always used for data that will arrive next.

• A spam classifier must classify the emails that arrive tomorrow.
• A customer-churn model must predict next month's customers.
• A price-prediction model must estimate houses that have not been traded yet.

So the real usage scene is always on the side of data not seen yet.

This table shows immediately why training score alone is not enough. A real service is not about reviewing the past, but about responding to the future.

That is why generalization is not just a bonus property. It is tied to the reason machine learning is used at all. If there were no need to deal with new data, then a rule table or lookup table would often be enough without a machine-learning model.

What Counts As New Data¶

The phrase unseen data may sound like data from a completely unfamiliar world. But usually it means examples from the same problem that were not used in training.

This point is often misunderstood. Generalization usually does not mean it works unconditionally in any environment. It is first closer to within the same problem setting, it still holds up to some degree on examples it has not seen yet.

For example, in customer-churn prediction, it can be organized like this.

All four belong to the same domain, but from the model's perspective, there is a boundary between what it has seen and what it has not seen yet. Generalization is a story that happens exactly at that boundary.

To make that boundary clearer, it can be divided like this.

So generalization is not a declaration that it works on any data. It is closer to how much does it hold up on unseen examples of the same problem?

The range of generalization becomes easier to organize when you first ask how far can this still be treated as a new example of the same problem?

The range can be visualized like this.

flowchart TB
  A["same task<br/>same meaning"]
  B["same task<br/>new samples"]
  C["same task<br/>condition shift"]
  D["different task or meaning"]

  A --> B
  B --> C
  C --> D

This diagram divides the range that generalization points to. The most direct test of generalization happens on new samples of the same problem, and as you move toward condition shifts or entirely different tasks, it becomes harder to group everything under the same generalization claim.

• same task, same meaning is the region closest to the training data
• same task, new samples is the region where generalization is tested first
• same task, condition shift may still be the same problem, but it is a harder region
• once you reach different task or meaning, it is usually hard to group it immediately as the same generalization issue

The Relationship Between Overfitting And Generalization¶

Now it also becomes natural to connect why we need to bring back overfitting and underfitting from P4-5.1. Generalization ultimately asks does the model hold up on new data?

The overfitting discussed in P4-5.1 is a representative scene where generalization weakens.

So overfitting can be read as a direction that harms generalization. But generalization is not a concept that ends simply as the opposite of overfitting. More broadly, generalization points to the whole question of how much does the model hold up on new data?

• overfitting can weaken generalization
• underfitting can also weaken generalization
• generalization ultimately asks about the power to hold up on new data

Validation And Test Are Ultimately Tools For Seeing Generalization¶

The reason validation and test were separated in P4-4.2 was also generalization in the end.

• validation data: a tool for comparing which candidate is more likely to hold up on new data
• test data: a tool for checking one last time whether the final choice really holds up on new data

flowchart TB
  A["training data<br/>learn pattern"]
  B["validation data<br/>compare candidates"]
  C["test data<br/>final check"]
  D["future unseen data<br/>real use"]

  A --> B
  B --> C
  C --> D

This diagram shows that train, validation, and test all exist in the end to estimate future unseen data in advance. Generalization should be read not as if validation or test score itself were the final goal, but as part of the full flow that asks how much the model will hold up on actual unseen data in real use.

In this diagram, validation and test are not the destination. Both are intermediate tools used to estimate future unseen data.

Therefore, validation and test are not simply procedures for assigning scores. Before that, they are procedures for observing generalization indirectly.

If the same procedure is rewritten into a table, it becomes the following.

Detailed Learning Content¶

Generalization Means Robustness, Not Perfection¶

Readers may misunderstand generalization as it must be perfect on new data too. But generalization is not perfect replication. Usually, it asks does the model hold up above some usable level even on new data?

So generalization is closer to stability than to perfection.

Generalization is not the power to repeat memorized answers. It is the power to hold up even on similar problems seen for the first time.

Cases And Examples¶

Case 1. When A Recommendation Model Is Strong Only For Familiar Users¶

A content-service team is operating a recommendation model. The criteria people first used were signals such as recently watched genre, frequently clicked category, and items watched a lot by similar users.

On the training data, a model that follows those criteria well can achieve a high score. But when you move to a bundle of new users that is closer to actual service conditions, or to users whose recent preferences have changed, recommendation satisfaction drops noticeably. If the team had been looking only at training score, it could easily mistake the model for a sufficiently good one.

From the perspective of generalization, the question changes. What matters is not did it explain the old records well? but does it hold up even for similar users it has not seen yet? Validation data and test data should be read as tools for checking that point in advance.

The checkable result appears when you compare scores from a familiar user group and scores from a new user group. That lets you read how stably the model remains inside the same problem. If the gap is small, generalization is relatively preserved. If the gap is large, you need to check again whether the model adapted too strongly only to old patterns.

flowchart TD
  A["past-user recommendation data"]
  B["model fits familiar users well"]
  C["new or shifted users appear"]
  D["compare unseen-user performance"]
  E["small gap"]
  F["large gap"]
  G["generalization looks stable"]
  H["model may rely on old patterns too much"]

  A --> B --> C --> D
  D --> E --> G
  D --> F --> H

Example 1. Reading Generalization Again Through Recommendation Systems And Price Prediction¶

Now that the relationships among definition, new data, overfitting, validation, and test have been organized, they can be read again through actual scenes. All of the examples below connect to the same question.

Can this model hold up beyond the table currently in our hands?

If you rewrite it as a recommendation-system example, it can be read like this.

Price prediction is similar.

Example 2. Social-Phenomenon Data Can Be Read Through The Same Generalization Question¶

Generalization is not needed only in business services. The same question appears in models that read social phenomena.

For example, imagine a model that classifies online opinion or post flows.

Cases dealing with recurring social patterns such as traffic congestion or a rise in complaints are similar.

The common point of these examples is the same. The fact that a model saw many past cases is not sufficient by itself. Social phenomena change in expression, in conditions, and in the way people respond. So generalization can also be read as the question does it hold up to some degree even in new social scenes?

It also helps to note that the changes on the side of social phenomena are not just one type.

Practice And Examples¶

Reading The Generalization Question Through A Table¶

The following record shows how to read scores from the perspective of generalization. Because this is a scene for interpreting score combinations that have already been produced, it is more appropriate to see training score, validation score, and the gap in one table than to use Python code.

In this table, generalization gap is not a formal theoretical explanation. It is only a supporting expression that helps readers interpret the difference between training score and validation score. The judgment should not end with gap alone; the absolute levels of training score and validation score must be read together.

• model_A is relatively similar between training and validation.
• model_B is very high on training but much lower on validation.
• model_C has a small gap, but both scores are low.

So when reading generalization, looking only at the gap is not enough, and looking only at the level is not enough either. If the gap is small but both are low, as with model_C, it is hard to call it a model with good generalization. If only training is high and validation is much lower, as with model_B, overfitting should be suspected first.

The important interpretation here is the following.

• training score asks how well does it match the data it has already seen?
• validation score asks how much does it hold up on similar data it has not seen yet?

Generalization is closer mainly to the second question.

Now let us split the same model several different ways. The example below uses the same data and the same DecisionTreeClassifier, but changes the random_state of train_test_split and prints how much the test score and gap fluctuate.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

from statistics import mean, pstdev

from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(
    n_samples=220,
    n_features=8,
    n_informative=3,
    n_redundant=1,
    flip_y=0.08,
    class_sep=0.8,
    random_state=12,
)

results = []
for seed in range(6):
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.35, random_state=seed, stratify=y
    )
    model = DecisionTreeClassifier(max_depth=4, random_state=0)
    model.fit(X_train, y_train)
    train_score = accuracy_score(y_train, model.predict(X_train))
    test_score = accuracy_score(y_test, model.predict(X_test))
    results.append((seed, train_score, test_score, train_score - test_score))

for seed, train_score, test_score, gap in results:
    print(
        seed,
        "train=", round(train_score, 3),
        "test=", round(test_score, 3),
        "gap=", round(gap, 3),
    )

print(
    "test mean=", round(mean(row[2] for row in results), 3),
    "test std=", round(pstdev(row[2] for row in results), 3),
)

An example output is as follows.

1
2
3
4
5
6
7

0 train= 0.944 test= 0.623 gap= 0.321
1 train= 0.874 test= 0.805 gap= 0.069
2 train= 0.93 test= 0.714 gap= 0.216
3 train= 0.93 test= 0.766 gap= 0.164
4 train= 0.923 test= 0.74 gap= 0.183
5 train= 0.881 test= 0.766 gap= 0.115
test mean= 0.736 test std= 0.057

This output shows why generalization should not be closed with one score. Even with the same data and the same model, the score and the gap fluctuate depending on which samples went into the test side. So generalization asks not only what was the score on this split?, but also whether performance holds up to some degree across similar new-data splits.

Reading Generalization On Social Phenomena Through A Table¶

The next example simplifies the idea that a post-classification model can be strong on familiar expressions but become unstable when new expressions enter.

This table is not actual model-training code. It is a reading exercise for interpreting generalization. The key is that when the new-expression score falls, the model has not necessarily moved to a completely different task; even inside the same problem, robustness can weaken because expression changes.

• posts from last month has only a small difference between familiar and new expressions.
• new slang appears shows that the score can drop sharply when the expression changes even a little.
• sarcasm and indirect wording increase shows that when the surface sentence changes, the model may become even more unstable.

So in social phenomena too, generalization should be read more accurately as how much does it hold up under new expressions and changed conditions? rather than just it worked well on the past.

And the phrase hold up is important here. When new data enter, some difference and fluctuation always appear. Generalization asks whether the model does not completely collapse amid those fluctuations and still keeps a usable level.

Checklist¶

• Can you explain why generalization should be described not as raising training score but as holding up on new examples of the same problem?
• Can you distinguish which cases can be put directly into the range of generalization and which cases should be treated as a different problem?
• Can you explain that validation and test are ultimately devices for checking generalization?
• Can you explain that generalization is the property that lets a model work usefully even on data it has not seen yet?
• Can you explain that the real goal of machine learning is not training score itself, and that both overfitting and underfitting can weaken generalization?
• Can you explain that generalization means stable robustness on new data rather than perfect agreement?

Sources And References¶

• Google for Developers, Machine Learning Glossary, accessed 2026-07-19. https://developers.google.com/machine-learning/glossary
• scikit-learn developers, Cross-validation: evaluating estimator performance, scikit-learn User Guide, accessed 2026-07-19. https://scikit-learn.org/stable/modules/cross_validation.html
• Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, Jonathan Taylor, An Introduction to Statistical Learning, Springer, official website accessed 2026-07-19. https://www.statlearning.com/
• Ulrike von Luxburg, Bernhard Schoelkopf, Statistical Learning Theory: Models, Concepts, and Results, Max Planck Institute publication page, accessed 2026-07-19. https://is.mpg.de/publications/4179