P4-4.2 Validation And Test¶

Section ID: P4-4.2 Version: v2026.07.20

P4-4.1 explained why data are divided into training data and evaluation data. Now the discussion moves one step further. The data used while choosing a model and the data used for the final one-time check do not play the same role.

If this difference is not separated, people start looking at test results repeatedly while choosing the model, and at that moment the test data can no longer play the role of data seen for the first time. That is why in practice evaluation data are often split again into validation data and test data.

This Section explains validation, test, and the difference between mid-process model-choice checks and the final check. Later Sections continue the current context through this handle, and the evaluation procedure after data splitting is connected again through this Section and the concept glossary.

Scope Of This Section¶

This Section explains the difference in role between validation and test. The calculation of metrics themselves is not covered in detail here. Metrics such as accuracy, precision, and recall are handled in P4-6.

This Section also explains the flow what should be looked at while choosing a model, but the full procedure of model selection and the discussion of baseline models return in P4-8. The ideas of overfitting and generalization are treated in more detail in P4-5.

• Why should validation data and test data be kept separate?
• When should each of them be used?
• Why is it a problem if you keep looking at test data in the middle?
• When the dataset is small, how should this distinction be read carefully?
• How does cross-validation connect to this structure?

Goals Of This Section¶

• You can explain the difference in role between validation and test.
• You can explain that validation data are for selection, while test data are for the final check.
• You can understand why repeated exposure to test data can distort the interpretation of results.
• You can explain that the distinction between validation and test becomes harder when the dataset is small.
• You can understand that cross-validation is not magic that replaces test data, but a method for making validation more stable within the available data.

Learning Background¶

Understanding It First Through One Scene¶

Think of a student preparing for university entrance exams by solving mock exams.

• practice problems: solved often during learning
• mock exam: used to compare which solution strategy is better
• real exam: used once at the end to check actual ability

Machine learning has a similar flow.

The biggest difference between validation data and test data is how often they are used in decision making.

Main Learning Content¶

Validation Data Help Selection¶

Validation data are used while choosing the model. For example, they are used for choices like the following.

• which to try first between logistic regression and decision tree
• whether tree depth should be 3 or 5
• whether changing preprocessing improved the result
• whether adding a feature actually helped

In other words, validation data are used in the middle of experiments to compare is this choice better than the previous one?

A very simple example can be read like this.

The important point in this table is not the number itself, but for what selection is this number being used? This is not yet the stage of publishing final performance. It is the stage of comparing candidates and narrowing them to one.

It becomes even easier if you translate it into a work scene. Suppose you are building an email-spam classifier.

• candidate 1: use word frequency only
• candidate 2: use word frequency + sender features
• candidate 3: candidate 2 + title-length feature

When choosing which of these three is better, you use validation data. This is not yet the stage of announcing the final model to deploy in the service. It is still the stage of finding the better candidate.

flowchart TB
  A["training data<br/>fit model"]
  B["validation data<br/>compare choices"]
  C["pick one model and setting"]
  D["test data<br/>final check once"]

  A --> B
  B --> C
  C --> D

In this diagram, validation data sit before the model is fixed. They appear repeatedly in the cycle change the experiment, check again, change again, check again.

Test Data Are For The Final Check¶

Test data are closer to a one-time final confirmation after the model has been chosen.

You can think about a flow like the following.

1. Train several models on the training data.
2. Compare them on the validation data and choose the most appropriate candidate.
3. Freeze that choice again.
4. Finally, check final performance on the test data.

The purpose of test data is to ask, after all of this selection process, how well does the model still work on data it sees for the first time? That is why test data should be opened as late as possible and should not be looked at repeatedly.

The same scene becomes clearer if you separate validation score and test score.

Here the difference between 0.81 and 0.76 is not strange. The candidate was chosen using the validation data, while the test data give a new-data check after that choice.

This difference is also where readers often misunderstand. If the validation score was higher and the test score dropped a little, that does not immediately mean something went wrong. It should instead be read as a signal that data for choosing and data for final checking play different roles.

The distinction between validation data and test data becomes clearer if you first ask is the current question for candidate selection, or for final confirmation?

Detailed Learning Content¶

Why You Should Not Keep Looking At The Test Set¶

If you keep opening the test data in the middle, you start changing choices to fit that result even without noticing.

Once this happens, the test data are no longer unseen data for the final check. In practice, they have been used like validation data.

The core principle is the following.

• validation data may be seen in the middle of experiments
• test data should usually be opened at the end
• if you keep changing selections after seeing the test set, the test set is also contaminated into a validation-like role

It becomes easier to remember if you compare them like this.

It also helps to look together at a flow that should not be done.

Cases And Examples¶

Case 1. Opening The Test Set Too Early In A Spam-Classification Experiment¶

Suppose a team operating a mail service is building a spam-classification model. Until now, people first filtered mail using rules such as are there many ad phrases, is it from a certain sender domain, or is the title excessively provocative.

Now the team wants a model that is better than the rule-based classifier. But if it keeps checking test data during experiments, a problem appears. For example, if Model A's test score looks disappointing, then features are added, the test score is checked again, settings are changed again, and so on. In that moment, the test set is effectively becoming comparison data.

This is exactly why validation data and test data are separated. Validation data are where you ask which candidate is better? Test data are where you ask is it now acceptable to check the chosen model one last time? Only when these roles are separated can the model-selection process avoid being dragged around by test results.

The checkable result appears in the experiment log. If the final model is chosen by candidate-specific validation scores and the test score is opened only once after that, you can read separately which numbers were used for selection and which number was used for the final check. By contrast, if the test score was recorded several times while changing selections, that test result is no longer a reliable basis for a final-confirmation claim.

flowchart TD
  A["spam dataset"]
  B["check test score early"]
  C["change features or settings again"]
  D["test becomes part of selection"]
  E["use validation for candidate choice"]
  F["freeze one final model"]
  G["open test once at the end"]

  A --> B --> C --> D
  A --> E --> F --> G

Separating The Roles Again With A Small Table¶

If you rewrite the customer-churn prediction example, it can be split like the following.

In this case, C04 and C05 may be seen several times in the middle of experiments. For example, when comparing Model A and Model B, both are checked on C04 and C05. But C06 and C07 should be opened once at the end.

It becomes even clearer when rewritten as questions.

• Should tree depth be 3 or 5? -> a question for validation data
• Can this model now be announced? -> a question for test data

When the question changes, the role of the data being used also changes.

Even with the same customer-churn table, once the question changes, the role of the data changes with it. The following diagram shows why the candidate-selection stage and the final-confirmation stage should not share the same data fragment.

flowchart TD
  A["customer churn table"]
  B["train split<br/>fit several models"]
  C["validation split<br/>choose depth / features / model"]
  D["final candidate"]
  E["test split<br/>open once at the end"]
  F["reported final result"]

  A --> B
  A --> C
  A --> E
  B --> C
  C --> D
  D --> E
  E --> F

Practice And Examples¶

Looking At The Split Structure Through Python¶

Keeping validation and test separate usually appears in code as a two-stage split. First split the whole data into training data and temporary evaluation data, then split that temporary evaluation data again into validation data and test data.

What to check in the example below is not model performance, but whether the roles of the three bundles remain separated even in code. In the output, inspect training, validation, and test sample counts, and validation/test label composition separately.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# This example separates training, validation, and test data to distinguish tuning from final checking.
from sklearn.model_selection import train_test_split

X = [[i] for i in range(12)]
y = ["stay", "churn", "stay", "stay", "churn", "stay", "stay", "churn", "stay", "stay", "churn", "stay"]

X_train, X_temp, y_train, y_temp = train_test_split(
    X,
    y,
    test_size=0.4,
    random_state=42,
)

X_val, X_test, y_val, y_test = train_test_split(
    X_temp,
    y_temp,
    test_size=0.5,
    random_state=42,
)

print("train size:", len(X_train))
print("validation size:", len(X_val))
print("test size:", len(X_test))
print("validation labels:", y_val)
print("test labels:", y_test)

An example output can be read like the following.

1
2
3
4
5

train size: 7
validation size: 2
test size: 3
validation labels: ['churn', 'stay']
test labels: ['stay', 'stay', 'stay']

This code does not split the data into three blocks at once directly. Instead, it first separates training and temporary evaluation data, then splits that temporary data again into validation and test. This two-step split is often easier to read.

Separating "Selection" And "Final Check" In A Table¶

This time, read a very small experiment log as a table. Because this is a scene for interpreting already recorded scores without actually training a model, a table that separates the roles of validation score and test score is more appropriate than Python code.

The key in this table is the order.

1. Compare the candidates by validation score.
2. Then choose one.
3. Finally, check the test score.

What you need to hold here is the order. Even in the same score table, validation is used for choosing and test is used for checking after the choice split into different roles.

Looking At A Bad Flow In A Table Too¶

The next example is a record meant to show what the problem is. It is not a good experiment procedure, but an example that reveals why you should not look at the test set repeatedly.

This table shows interpretation more than calculation. The moment you look at the test score first and then change the choice because of it, the test data begin to leave their role as data for final confirmation only.

How Validation And Test Should Be Read¶

When reading experiment results, separate the role before the number itself.

This Section does not yet treat metric types in detail. What matters now is that even when the number is the same, for example 0.82, its meaning changes depending on whether it is a validation score or a test score.

So when reading an experiment record, it becomes clearer to complete sentences like the following.

• Is this number a validation score used for model choice?
• Is this number a test score checked after the final choice?
• After seeing this number, was the model changed again?

If the answer to the third question is yes, then that number may no longer be only a number for final confirmation.

Be More Careful When The Dataset Is Small¶

If the dataset is not large enough, it becomes difficult to keep training, validation, and test all fully separate.

In such cases, cross-validation is sometimes used. Cross-validation is a method for repeating validation several times within the available data so that comparison becomes more stable. But that does not mean test data become unnecessary. In real projects, the validation structure must still be adjusted according to the dataset size and the task.

For example, if there are only 30 customer records, the following problems appear immediately.

The important point in this Section is not what ratio must be correct, but rather that the smaller the dataset is, the more carefully validation and test must be interpreted.

Checklist¶

• Can you explain why validation data are distinguished as for selection and test data as for final confirmation?
• Can you explain why repeatedly looking at test scores in the middle contaminates the test set into a validation-like role?
• Can you distinguish which questions should be asked to validation data and which should be left for test data?
• Can you explain that validation data are used to compare models and settings, while test data are used for the last check after the final choice?
• Can you explain that if selections keep changing to fit test results, the test set is also contaminated into a validation-like role?
• Can you explain that when the dataset is small, it becomes harder to secure both validation and test stably?

Sources And References¶

• scikit-learn developers, Cross-validation: evaluating estimator performance, scikit-learn User Guide, accessed 2026-07-19. https://scikit-learn.org/stable/modules/cross_validation.html
• scikit-learn developers, train_test_split, scikit-learn API Reference, accessed 2026-07-19. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
• Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, Jonathan Taylor, An Introduction to Statistical Learning, Springer, official website accessed 2026-07-19. https://www.statlearning.com/