P4-4.1 Training Data And Evaluation Data¶

Section ID: P4-4.1 Version: v2026.07.20

Chapter P4-3 showed how heuristics can narrow the model candidates to try first. That immediately raises an important question. How can you check whether that choice is actually acceptable?

In machine learning, it is not enough for a model to fit the data it has already seen. What we want is a model that makes usable judgments even on new data that will arrive later. So not all data are used only for learning. Some are left aside separately for evaluation.

Training data are the data used for learning. Evaluation data are the data used to check what the model learned. The most important first perspective is simple. If you take a test using only the worksheet you already studied, you can easily mistake your level of skill.

Scope Of This Section¶

This Section explains why data are split. The detailed distinction between validation data and test data is handled in P4-4.2. Here the goal is first to fix the difference between the data used for learning and the data left aside for evaluation.

Overfitting and generalization are treated in detail in P4-5. Metrics such as accuracy, precision, and recall are handled in P4-6. The focus here is the reason you must separate them before checking.

• Why should all data not be used only for training?
• What roles do training data and evaluation data play?
• Why is it risky when a model fits only data it has already seen?
• What kinds of misunderstandings does data splitting prevent?
• What should be treated carefully when the dataset is small?

Goals Of This Section¶

• You can distinguish the roles of training data and evaluation data.
• You can explain that training and evaluating on the same data can overestimate performance.
• You can understand that evaluation data are a proxy scene for estimating how the model may act on new data.
• You can explain why data splitting leads into model selection, overfitting, and generalization.
• You can keep a clear boundary that the detailed distinction between validation and test is handled in the next Section.

Learning Background¶

Understanding It First Through One Scene¶

Suppose a student memorized 100 math problems. If you give the same 100 problems again, the student can get a high score. But that alone does not prove the student can solve new problems.

A machine learning model is similar. The model looks for patterns in the training data. But if evaluation is also done on the same data, it becomes hard to tell whether the model learned a pattern or merely memorized particular examples.

This analogy is not perfect, but it shows the core point of data splitting. The purpose of the model is not to match examples it already saw again. It is to respond to examples it has not seen yet.

In practice, you first need to judge which split perspective should be used first in the current problem.

Main Learning Content¶

Data Are Split By Role¶

The most basic split is between the part used for learning and the part used for evaluation.

flowchart TB
  A["all available data<br/>collected examples"]
  B["training data<br/>model learns from this"]
  C["evaluation data<br/>model is checked here"]
  D["trained model<br/>patterns learned from training data"]
  E["performance estimate<br/>how well it may work on unseen data"]

  A --> B
  A --> C
  B --> D
  D --> C
  C --> E

In this diagram, evaluation data are the data not used directly during learning. The model is built from training data, then evaluation data are used to check whether the model also works on other examples.

Here the phrase evaluation data is used in a broad sense. In a real project, validation data and test data are often separated further. That distinction is handled in P4-4.2.

The same idea becomes more intuitive in a very small table.

In this example, the rule is learned from C01, C02, and C03, then checked on C04 and C05. If C04 and C05 are also used for training, the model has already seen them, so the meaning of evaluation becomes weak.

The same idea can be moved into code as follows. The example below splits the customer feature list X and churn labels y together using the same criterion, then checks which samples and labels enter the training and evaluation sides. In the result, inspect the training inputs/labels, evaluation inputs/labels, sample counts, and churn ratios for each bundle.

The key point is that train_test_split splits the inputs and labels together under the same criterion. After the split, it is important to build the habit of checking counts and label ratios before performance scores.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# This example splits input features X and target y into training and evaluation sets and checks label ratios.
from sklearn.model_selection import train_test_split

X = [
    [8, 0],  # recent purchases, support tickets
    [2, 3],
    [6, 1],
    [1, 4],
    [7, 0],
]
y = ["stay", "churn", "stay", "churn", "stay"]

X_train, X_eval, y_train, y_eval = train_test_split(
    X,
    y,
    test_size=0.4,
    random_state=42,
)

print("training inputs:", X_train)
print("evaluation inputs:", X_eval)
print("training labels:", y_train)
print("evaluation labels:", y_eval)
print("training sample count:", len(X_train))
print("evaluation sample count:", len(X_eval))
print("training churn ratio:", y_train.count("churn") / len(y_train))
print("evaluation churn ratio:", y_eval.count("churn") / len(y_eval))

An example output can be read like the following.

1
2
3
4
5
6
7
8

training inputs: [[6, 1], [8, 0], [1, 4]]
evaluation inputs: [[2, 3], [7, 0]]
training labels: ['stay', 'stay', 'churn']
evaluation labels: ['churn', 'stay']
training sample count: 3
evaluation sample count: 2
training churn ratio: 0.3333333333333333
evaluation churn ratio: 0.5

This code shows the most basic shape of not feeding the whole dataset into the model at once, but first splitting it into a training part and an evaluation part. X_train and y_train are used for learning, while X_eval and y_eval are left aside for checking the result.

The values printed together here are not model-performance metrics yet. They are instead the most basic inspection values that should be checked immediately after splitting the data.

• how the training sample count and evaluation sample count were divided
• whether the churn label ratio is skewed heavily to one side

Even these outputs alone help you understand more concretely what kind of environment the model is about to learn in.

In a more practical style, people usually split input columns and the answer column from a DataFrame first, then split the data. The example below uses the feature-column list feature_columns, the answer column target_column, and a tabular dataset df to check the shape of X_train and X_eval and the label ratios on the training and evaluation sides.

The key point is that a practical split should be read in the order of first separating the input-column bundle X and the answer column y. shape and label ratios are the basic outputs for checking a split result.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# This example splits input features X and target y into training and evaluation sets and checks label ratios.
import pandas as pd
from sklearn.model_selection import train_test_split

feature_columns = ["recent_purchases", "support_tickets", "days_since_login"]
target_column = "churned"

df = pd.DataFrame(
    [
        {"recent_purchases": 8, "support_tickets": 0, "days_since_login": 2, "churned": "stay"},
        {"recent_purchases": 4, "support_tickets": 1, "days_since_login": 5, "churned": "stay"},
        {"recent_purchases": 6, "support_tickets": 1, "days_since_login": 4, "churned": "stay"},
        {"recent_purchases": 1, "support_tickets": 4, "days_since_login": 21, "churned": "churn"},
        {"recent_purchases": 7, "support_tickets": 0, "days_since_login": 3, "churned": "stay"},
    ]
)

X = df[feature_columns]
y = df[target_column]

X_train, X_eval, y_train, y_eval = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
)

print("X_train shape:", X_train.shape)
print("X_eval shape:", X_eval.shape)
print("training label ratio:")
print(y_train.value_counts(normalize=True))
print("evaluation label ratio:")
print(y_eval.value_counts(normalize=True))

An example output looks like the following.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

X_train shape: (4, 3)
X_eval shape: (1, 3)
training label ratio:
churned
stay     0.75
churn    0.25
Name: proportion, dtype: float64
evaluation label ratio:
churned
stay    1.0
Name: proportion, dtype: float64

Here what matters is not the syntax itself, but the separation of roles. As long as you can read the order first separate X as input and y as answer, then separate them again into training and evaluation parts, that is enough. The shape and label-ratio outputs are the fastest checks for whether the split result became too skewed.

Data Splitting Must Also Fit The Problem¶

The phrase split the data does not always mean a random half-and-half split in the same way. The criterion for splitting also changes according to the nature of the problem.

For example, suppose shopping-mall data were collected from January to June.

In this case, learning from January through April and evaluating on May and June is closer to the real operational flow. By contrast, if part of the June data is cut out and mixed randomly into February and March for training, patterns that appeared only in the future can leak into past learning.

Detailed Learning Content¶

What Misunderstandings Appear If The Split Is Wrong¶

Even if the data are split, if the method does not match the problem, the result can still create false reassurance.

A very small churn example makes this problem clearer.

In such a case, even if the evaluation score is low, it becomes hard to tell whether the model is poor or whether the split itself was poor from the start. So after splitting, you must check not only the sample count but also the label distribution.

Why It Is Risky To Learn And Evaluate On The Same Data¶

The model looks for patterns in the data. But if the model is very flexible or the dataset is small, it can follow not only the general pattern but also accidental traces of the training data.

For example, suppose a churn dataset happened to contain many customers from a certain event period who churned. If the model trusts that trace too strongly, it may judge new customers incorrectly even when they are unrelated to that event.

If evaluation is done on the same data, that problem is hidden. The model receives a high score because it is matching traces it already saw. But if evaluation is done on data kept aside separately, it becomes more realistic to check whether the same judgment still works on other examples.

Training Score And Evaluation Score Are Read Differently¶

The score on training data and the score on evaluation data do not mean the same thing.

If the training score is high and the evaluation score is also high, that can be a relatively good signal. If the training score is high but the evaluation score is low, the model may have fit the training data too strongly. That problem is called overfitting and is treated in detail in P4-5.

By contrast, if both the training score and the evaluation score are low, the model may not have learned enough. That connects to underfitting, which is also handled in P4-5.

A short practical-style example can be read like the following.

This table does not imply a universal threshold. It is an example that shows how to read the gap between training score and evaluation score, rather than what the numbers must be.

Evaluation Data Are Not A Perfect Substitute For The Future¶

Evaluation data are a device for imitating data the model has not seen yet. But evaluation data do not guarantee perfect representation of the future.

For example, suppose shopping-mall data were collected from January to June, and part of them was left aside as evaluation data. That evaluation set shows customer behavior from the same broad period. But it may not perfectly represent a November discount season or customer behavior in the following year.

So data splitting is a necessary starting point, but not the whole story. Time change, sampling bias, the data-collection method, and service-policy changes must also be considered. The basic sense of sample and bias was already seen in the earlier probability-and-statistics recovery part, and in machine learning it reconnects again in P4-5 and P4-6.

Be More Careful When The Dataset Is Small¶

If the dataset is large enough, leaving some aside for evaluation still leaves a reasonable amount for learning. But when the dataset is small, the act of splitting itself becomes difficult.

When the dataset is small, the following problems appear.

• The training data shrink too much, so the model may not learn properly.
• The evaluation data become too small, so the score can swing.
• A single split result can be influenced heavily by chance.
• A specific category or rare case can gather only on one side.

Because of this, methods such as cross-validation are sometimes used. But this Section does not explain the full procedure of cross-validation. At this stage, it is enough to fix the point that the smaller the data, the more unstable the evaluation score can become. Cross-validation appears again in P4-4.2 and P4-8 as needed.

Cases And Examples¶

Case 1. When The Coupon-Response Score Looks Too Good¶

Suppose a marketing team wants to predict whether sending a coupon to a customer will lead to an actual purchase. The first human signals were things like was there a recent purchase, does the user visit the app frequently, and has the user used coupons before.

The problem is that because the team does not have much data, it wants to use the whole table together for both model learning and model evaluation. If this is done, the score can look high, but it becomes hard to know whether the model will still work well on new customers. It may simply have become good at matching order records it already saw.

Here data splitting changes the way judgment is made. If part of the customer records are left for learning and another part is left for evaluation, you can check whether the model behaves similarly even on customer records it sees for the first time. It also becomes easier to distinguish whether the criteria people considered important are truly repeated signals or merely traces that looked strong only in this particular table.

The checkable result is clear as well. If you compare the score from evaluation on the whole dataset with the score from separately held-out evaluation data, you can see whether a gap appears. If only the training score is high while the evaluation score falls a lot, that should be read as a signal that the interpretation without data splitting was too optimistic.

flowchart TD
  A["customer coupon table"]
  B["train and evaluate on the whole table"]
  C["score looks too good"]
  D["leave out an evaluation split"]
  E["train on one part only"]
  F["check unseen-customer score"]
  G["read whether optimism was justified"]

  A --> B --> C
  A --> D --> E --> F --> G

Cases And Examples¶

Misunderstandings That Data Splitting Prevents¶

Data splitting reduces the following misunderstandings.

Data splitting is not a device for lowering model performance. It is a device for reading performance more honestly.

Practice And Examples¶

Small Python Exercise¶

This Section does not yet need to train a model. The goal is to learn by hand what values should be checked first after splitting the data.

Exercise 1. Change test_size¶

The code below splits the same data using two different ratios. Using the customer feature list X, churn labels y, and two test_size values, it checks the training/evaluation sample counts and churn ratios under each ratio.

The key point is that as test_size grows, the evaluation data grow and the training data shrink. Changes in sample count and changes in label ratio should be read together.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# This example splits input features X and target y into training and evaluation sets and checks label ratios.
from sklearn.model_selection import train_test_split

X = [
    [8, 0],
    [2, 3],
    [6, 1],
    [1, 4],
    [7, 0],
    [3, 2],
    [9, 0],
    [2, 5],
]
y = ["stay", "churn", "stay", "churn", "stay", "churn", "stay", "churn"]

for ratio in [0.25, 0.5]:
    X_train, X_eval, y_train, y_eval = train_test_split(
        X,
        y,
        test_size=ratio,
        random_state=42,
    )

    print("test_size =", ratio)
    print("training sample count:", len(X_train))
    print("evaluation sample count:", len(X_eval))
    print("training churn ratio:", y_train.count("churn") / len(y_train))
    print("evaluation churn ratio:", y_eval.count("churn") / len(y_eval))
    print("-" * 30)

An example output can look like the following.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

test_size = 0.25
training sample count: 6
evaluation sample count: 2
training churn ratio: 0.3333333333333333
evaluation churn ratio: 1.0
------------------------------
test_size = 0.5
training sample count: 4
evaluation sample count: 4
training churn ratio: 0.25
evaluation churn ratio: 0.75
------------------------------

In this exercise, what matters is the split result, not a score. As test_size grows, the evaluation data increase and the training data decrease. When the dataset is very small, this difference feels even larger.

Exercise 2. Change random_state¶

Even with the same data, the split result can change if the shuffle criterion changes. The example below uses the same X, y, and different random_state values to compare the evaluation-label list and evaluation churn ratio under each seed.

The key point is that random_state is a reference value for reproducing a split result. With small data, even changing only the seed can make the evaluation composition swing substantially.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# This example splits input features X and target y into training and evaluation sets and checks label ratios.
from sklearn.model_selection import train_test_split

X = [
    [8, 0],
    [2, 3],
    [6, 1],
    [1, 4],
    [7, 0],
    [3, 2],
    [9, 0],
    [2, 5],
]
y = ["stay", "churn", "stay", "churn", "stay", "churn", "stay", "churn"]

for seed in [0, 7, 42]:
    X_train, X_eval, y_train, y_eval = train_test_split(
        X,
        y,
        test_size=0.25,
        random_state=seed,
    )

    print("random_state =", seed)
    print("evaluation labels:", y_eval)
    print("evaluation churn ratio:", y_eval.count("churn") / len(y_eval))
    print("-" * 30)

An example output can differ like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

random_state = 0
evaluation labels: ['stay', 'stay']
evaluation churn ratio: 0.0
------------------------------
random_state = 7
evaluation labels: ['stay', 'churn']
evaluation churn ratio: 0.5
------------------------------
random_state = 42
evaluation labels: ['churn', 'churn']
evaluation churn ratio: 1.0
------------------------------

This exercise shows why random_state should be written down. If you want to reproduce the same split result when the same code is run again, the reference value must be fixed.

Exercise 3. See What Problem Appears When The Data Are Skewed¶

The next example shows how easily the split result can swing when churn cases are few. It splits X and y with a sparse churn label and checks the training-label list, evaluation-label list, and the churn count in each bundle.

The key point is that in class-imbalanced data, label distribution can shift easily even with an ordinary random split. This is why label composition should be printed immediately after the split.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# This example splits input features X and target y into training and evaluation sets and checks label ratios.
from sklearn.model_selection import train_test_split

X = [[i] for i in range(10)]
y = ["stay", "stay", "stay", "stay", "stay", "stay", "stay", "stay", "stay", "churn"]

X_train, X_eval, y_train, y_eval = train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42,
)

print("training labels:", y_train)
print("evaluation labels:", y_eval)
print("training churn count:", y_train.count("churn"))
print("evaluation churn count:", y_eval.count("churn"))

An example output can be read like this.

1
2
3
4

training labels: ['stay', 'stay', 'stay', 'churn', 'stay', 'stay', 'stay']
evaluation labels: ['stay', 'stay', 'stay']
training churn count: 1
evaluation churn count: 0

In this exercise, one side can end up with almost no churn, or even none at all. In that state, it becomes very hard for the model to learn churn patterns or to evaluate them properly. That is why the next Section returns to stratified split.

Checklist¶

• Can you explain why using the same data for both learning and evaluation easily overestimates performance?
• Can you explain why the split method should differ in time-ordered data and rare-label data?
• Can you distinguish what training score and evaluation score each show?
• Can you explain that training data are used for learning, while evaluation data are left aside to check what the model learned?
• Can you explain that evaluation data are a proxy scene for estimating behavior on new data?
• Can you explain that if the dataset is small, even evaluation scores can swing, so they must be read more carefully?

Sources And References¶

• scikit-learn developers, Cross-validation: evaluating estimator performance, scikit-learn User Guide, accessed 2026-07-19. https://scikit-learn.org/stable/modules/cross_validation.html
• scikit-learn developers, train_test_split, scikit-learn API Reference, accessed 2026-07-19. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
• Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, Jonathan Taylor, An Introduction to Statistical Learning, Springer, official website accessed 2026-07-19. https://www.statlearning.com/