P2-12.3 The Intuition of Preparing a Learning Dataset¶

Section ID: P2-12.3 Version: v2026.07.23

In P2-12.1, we read a DataFrame as a table-shaped data structure. In P2-12.2, we chose the needed columns from that table, filtered rows by condition, and checked summary values. Now the question moves one step further: what must be prepared to turn this table into a learning dataset that a model can read?

The important point here is that handling Pandas well and preparing a learning dataset well are not the same thing. The former is a table-manipulation skill. The latter is the work of deciding what inputs the model receives and what answers it should learn.

This Section explains the basic distinction among dataset, feature, target, validation, and data leakage. The representative explanation of DataFrame and table selection stays in P2-12.1, P2-12.2, and the concept glossary. Here, the focus is on how to reorganize that table into learning inputs and answers.

If Chapter 11 made a computable array shape, Chapter 12 is now the stage where we decide which columns to keep from the table and which to remove. The input and answer candidates organized here lead into the visualizations of Chapter 13 and the record organization of Chapter 14.

Core Criteria: The Intuition of Preparing a Learning Dataset¶

• You can explain learning-dataset preparation as reorganizing the original table into model inputs and answers.
• You can read one row as one sample and one column as a candidate feature or target.
• You can explain why X and y are separated.
• You can explain why train, validation, and test are separated.
• You can explain why data leakage distorts evaluation.
• You can explain that Pandas is used for reading data, selecting, filtering, creating columns, and basic checking, while training splits are often handled together with separate tools.

One Scene to Hold First¶

The first scene to hold in this Section is that one row is usually one sample, one column is a candidate feature or target, X is a bundle of input columns, and y is the answer column to predict.

If we look again at a small student table:

If this picture is held first, later expressions such as train_test_split, fit, and predict feel less vague.

Three Criteria¶

Do Not Train the Table As It Is; Reorganize It for the Question¶

The original table is often organized so that a person can read it easily, but it is not always in a shape that a model can learn from directly.

For example, consider the following table.

A person can ask several questions from this table.

• Do we want to predict the score?
• Do we want to classify pass/fail?
• Do we want to inspect trends by region?

Even with the same table, if the question changes, the structure of the learning dataset changes as well.

So dataset preparation is more accurately read not as organizing a table, but as drawing the boundary between inputs and answers again according to the question.

Separate X and y¶

The scikit-learn documentation typically uses X for the input feature matrix and y for the target. In the glossary, a feature is described as a quantity that represents a sample with numerical or categorical values, and features are described as columns in a data matrix. A sample is usually described as one feature vector, and a target as the dependent variable in supervised learning.

At this stage, it is enough to remember the following difference.

• X: the bundle of input columns the model will look at and use for judgment
• y: the answer column the model should predict

A small example helps.

Problem situation: when turning the student table into a classification problem that predicts pass/fail, you first need to divide it into input X and answer y. Input: a small DataFrame containing region, number of absences, score, and pass status for each student. Expected output: the input-column bundle X and the answer column y are separated. Concept to check: preparing a learning dataset is not using the original table unchanged, but dividing inputs and targets according to the problem definition.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# This example separates features X and target y from a DataFrame and prepares a learning dataset.
import pandas as pd

df = pd.DataFrame(
    {
        "student_id": ["S001", "S002", "S003", "S004"],
        "region": ["Seoul", "Busan", "Seoul", "Busan"],
        "absences": [1, 5, 0, 2],
        "score": [82, 45, 90, 73],
        "passed": ["yes", "no", "yes", "yes"],
    }
)

X = df[["region", "absences", "score"]]
y = df["passed"]

This code can be read as a classification problem that predicts passed.

• Each row is one student sample.
• region, absences, and score are input features.
• passed is the answer target.

Seen as a diagram, it looks like this.

flowchart TD
    raw["raw DataFrame<br/>student_id, region, absences, score, passed"]
    choose["choose target<br/>what are you trying to predict?"]
    split["split columns<br/>separate X and y"]
    x["X<br/>region, absences, score"]
    y["y<br/>passed"]

    raw --> choose --> split
    split --> x
    split --> y

The core point is that X and y are not fixed from the beginning. The problem definition comes first, and then the columns are divided.

This also keeps the perspective that connects directly to P2-11.2.

So dividing X and y does not only mean splitting column names. It also means deciding the shape of the array the model will read.

You can check the same separation with an input that has a few more rows. The input file is the same student-progress-samples.csv used in P2-12.2. One row is one student's learning record. Here, passed is the target, and some of the remaining columns are treated as input candidates.

Problem situation: I want to check the shapes of the input column group X, the answer column y, and a simple train/test candidate split from the original table. Input: a 36-row student-progress CSV, feature_columns, and target_column. Expected output: the shapes of X, y, encoded input columns, and train/test split candidates. Concept to check: Dataset preparation means deciding the boundary between input and answer in the table before model training, and separating rows for later evaluation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# This example separates features X and target y from a DataFrame and prepares a learning dataset.
from pathlib import Path
import pandas as pd

csv_path = Path("docs/assets/part-02/chapter-12/student-progress-samples.csv")
df = pd.read_csv(csv_path)

feature_columns = ["region", "study_hours", "absences", "practice_quizzes", "score"]
target_column = "passed"

X = df[feature_columns]
y = df[target_column]
X_encoded = pd.get_dummies(X, columns=["region"], dtype=int)

test_index = df.sample(frac=0.25, random_state=42).index
train_index = df.index.difference(test_index)

print("X shape:", X.shape)
print("y shape:", y.shape)
print("encoded columns:", list(X_encoded.columns))
print("train/test rows:", len(train_index), len(test_index))

The same code can be run as p2_12_3_dataset_split_preview.py. This does not train a scikit-learn model. The goal of Part 2 is first to check, at the Pandas level, which columns are inputs, which column is the answer, and which rows should be held out for later checking.

You Should Not Put Every Column In As-Is¶

Not every column in the table is suitable as model input. Usually, the following three groups must be separated.

1. columns that can be used directly for prediction
2. columns that need transformation
3. columns that should be excluded from the input

For example:

Here, categorical columns such as region may require a step that converts them into a numeric form depending on the model. Pandas get_dummies() is introduced as a function that converts categorical variables into dummy/indicator variables.

For example:

Problem situation: many models cannot read a string category such as region directly, so its representation must be changed. Input: input X made of the columns region, absences, and score. Expected output: X_encoded, where region has been expanded into multiple 0/1 columns. Concept to check: categorical columns may need to be re-expressed numerically before learning.

1
2
3
4
5

# This example separates features X and target y from a DataFrame and prepares a learning dataset.
X = df[["region", "absences", "score"]]
X_encoded = pd.get_dummies(X, columns=["region"])

print(X_encoded)

The core of this stage is not memorizing the encoding method. The more important question is can this column be read directly like a number, or must its representation be changed first?

If You Do Not Distinguish Identifiers and Answer Columns, You Easily Misread the Table¶

A common confusion is feeling that every visible column is a feature. But identifiers and targets play different roles from features.

For example:

• student_id is a marker that distinguishes each row.
• name is a human-readable name.
• passed may be the answer we want to predict.

If you include them in the input unchanged, two problems can appear.

1. The model may be drawn to accidental identifier information rather than general patterns.
2. Information too close to the answer may enter the input and inflate evaluation.

This intuition remains important later as well. A good dataset is not good because it has many columns. It is closer to a dataset that keeps only the columns appropriate to the problem definition.

Split Into Train, Validation, and Test¶

The next important step in dataset preparation is splitting. The scikit-learn train_test_split documentation describes it as a tool that randomly splits arrays or matrices into train and test subsets.

Here, it is enough to understand it like this.

• train: data the model actually learns from
• validation: data used to inspect settings and compare choices
• test: data used at the end as if it were being seen for the first time

Seen as a diagram:

flowchart TD
    full["full dataset"]
    train["train set<br/>data used for learning"]
    val["validation set<br/>data used to compare settings"]
    test["test set<br/>data used for final check"]

    full --> train
    full --> val
    full --> test

Why should we split it? Because performing well on data the model has already seen is not enough. What we want to know is does it work similarly on data it sees for the first time?

This perspective leads directly to overfitting, generalization, and evaluation metrics in Part 3.

Split First, and Learn Training-Time Transformations Only from Train¶

The scikit-learn common pitfalls documentation strongly warns against two mistakes.

• inconsistent preprocessing: the mistake of applying different preprocessing to train data and test data
• data leakage: the mistake of mixing information into the learning process that should be unavailable at prediction time

In particular, the documentation explains that test data should never be used to make choices about the model, and as a general rule advises not to call fit or fit_transform on test data.

At this stage, the following order is a good standard.

1. Choose input candidates and answer candidates from the full table.
2. Split them into train, validation, and test first.
3. For transformations that must be learned, such as means, standardization, encoding, or feature selection, establish the rule only from train.
4. Apply the same transformation to validation and test without learning from them.

If this order is broken, it becomes similar to letting the model peek at the answers in advance.

If you place the wrong flow and the safer flow side by side:

In this Section, we fix this judgment standard before implementation details.

It becomes clearer again in the following diagram.

flowchart TD
    full["full table"]
    choose["choose X and y"]
    split["split first<br/>train / validation / test"]
    fit_train["learn transform rule<br/>fit on train only"]
    apply_other["apply same transform<br/>only apply to validation and test"]
    bad["wrong order<br/>learn the rule from the full table first"]

    full --> choose --> split --> fit_train --> apply_other
    full -. avoid .-> bad

If you rewrite the wrong order and the safer order as questions:

View It Through a Case¶

Case 1. Predicting Pass/Fail While Accidentally Putting the Answer Column into the Input¶

Suppose a learner wants to build a model that predicts passed from a student table. The table contains student_id, region, absences, score, and passed together. At that point, it is tempting to put every visible column into X.

But that creates a problem. passed is the answer to predict, yet it also gets included in the input, and student_id may be only a marker that distinguishes students rather than something directly related to a general pattern. The model may end up peeking at the answer or being drawn to accidental identifier information instead of learning a real rule.

So in dataset preparation, you first decide what do we want to predict?, then separate y, and then choose again which features to keep and which columns to remove. After that, train, validation, and test should be split first, and transformations such as encoding or scaling that must learn a rule during training should be established only on the train side.

This case shows that dataset preparation is not simple table organization. Even with the same table, the boundary between X and y changes according to the question, and if you choose the split order incorrectly, the evaluation may become inflated. Pandas is the tool used in the front part of this process to choose columns and inspect structure, while fair learning evaluation is established only when the later split and transformation order are also correct.

The sentence the reader should especially hold here is the following.

• Separate X and y first, split into train / validation / test first, and then create transformation rules to learn only from train.

Pandas Handles the Front Part of the Preparation Process¶

Pandas is usually strong in the front part of dataset preparation.

• It reads raw sources such as CSV, Excel, and JSON.
• It chooses only the needed columns.
• It checks invalid values or strange formats.
• It distinguishes categorical columns from numeric columns.
• It performs basic column creation and transformation.

For example:

Problem situation: before model training, you want to inspect with your eyes what shape the input columns and answer column actually have. Input: separated X and y. Expected output: the first few rows of X and the first few values of y. Concept to check: Pandas is strong at inspecting table structure and checking column roles before learning.

1
2
3
4
5
6

# This example separates features X and target y from a DataFrame and prepares a learning dataset.
X = df[["region", "absences", "score"]]
y = df["passed"]

print(X.head())
print(y.head())

Or:

Problem situation: you need to check whether there are missing values and what type each column was read as. Input: the original DataFrame df. Expected output: the count of missing values for each column and the list of data types for each column. Concept to check: pre-learning inspection is not only about the values themselves, but also about missing values and type structure.

1
2
3

# This example separates features X and target y from a DataFrame and prepares a learning dataset.
print(df.isna().sum())
print(df.dtypes)

Code like this is closer to inspect the table before learning than to train the model.

By contrast, the actual split and learning pipeline is often handled together with tools such as scikit-learn.

For example:

Problem situation: after inspection is complete, you need to split the data into train and test to separate learning from evaluation. Input: input X, answer y, a split ratio, and a random seed. Expected output: X_train, X_test, y_train, and y_test split into learning and test sets. Concept to check: the table-manipulation stage and the learning-data split stage are connected, but they are different tasks with different roles.

1
2
3
4
5
6
7
8
9

# This example separates features X and target y from a DataFrame and prepares a learning dataset.
from sklearn.model_selection import train_test_split

X = df[["region", "absences", "score"]]
y = df["passed"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

The important point here is not memorizing the full function usage, but understanding that table manipulation and learning split are different stages that connect to each other.

In One Sentence, What Is Learning-Dataset Preparation?¶

If we reduce the core of this Section to one sentence, preparing a learning dataset is reassigning the roles of sample, feature, and target in the original table, and preserving the split order so that evaluation is not distorted.

Without this perspective, the longer the Pandas code becomes, the easier it is to lose track of why columns are being changed and divided. With this perspective, even before you know every preprocessing technique, you can still ask is this code choosing the input, choosing the answer, or preventing leakage?

Checklist¶

• Can you say in one sentence what you are trying to predict from the current table?
• Can you distinguish which column is y and which columns are candidates for X?
• Can you explain why it is risky to put identifier columns and answer columns directly into the input?
• Can you explain why train / validation / test are separated?
• Can you explain why preprocessing rules should be learned only from train?
• When you see X.shape = (4, 3) and y.shape = (4,), can you say what is the number of samples and what is the number of features?
• Can you explain that dataset preparation is not using the table as it is, but dividing it into X and y according to the problem definition?

Sources and References¶

• pandas Developers, pandas.get_dummies, pandas 3.0.4 documentation, checked on 2026-07-20. Used to confirm examples of converting categorical variables into dummy or indicator variables.
• scikit-learn Developers, Glossary, scikit-learn 1.9.0 documentation, checked on 2026-07-20. Used to confirm background for 1d/2d arrays, array-like inputs, estimator input conventions, and the terms X and y.
• scikit-learn Developers, train_test_split, scikit-learn 1.9.0 documentation, checked on 2026-07-20. Used to confirm the API for splitting arrays and matrices into train/test subsets and examples involving test_size and random_state.
• scikit-learn Developers, Common pitfalls and recommended practices, scikit-learn 1.9.0 documentation, checked on 2026-07-20. Used to confirm cautions about preprocessing order before and after train/test split and data leakage.