P7-2.3 Comparison-Experiment Practice¶

Section ID: P7-2.3 Version: v2026.08.01

Record experiment_variant, preprocessing_change, baseline_gap, fixed_error, new_error, and next_boundary_case. Comparing several variants makes it possible to distinguish a preprocessing problem from a missing data-boundary problem.

Separate failures before choosing a winner¶

Compare baseline, raw 1-NN, partial scaling, and z-score normalization on the same evaluation rows. A failure fixed after preprocessing points to scale, encoding, or missing-value handling. A failure that remains after normalization points to a need for more boundary cases or more useful features. Predictions that disagree across settings are ambiguous boundary cases.

The point is not to declare the highest score “best.” It is to narrow the kind of failure and choose the next action.

Learning questions and criteria¶

• Which evidence shows that a failure is fixed by a representation change rather than by a new data row?
• Which failures remain after z-score normalization and therefore need boundary evidence or another feature?
• Why can the highest aggregate score coexist with setting-sensitive cases?

You have completed the practice when you can identify the fixed evaluation rows, run all variants on the same stress rows, and propose one next experiment that changes either preprocessing, data coverage, or feature design—but not all three at once.

Use p7-2-churn-dataset.en.csv for training and basic evaluation and p7-2-stress-test.en.csv for additional failure-reading cases. A basic row is one subscription customer record; a stress row is an extra evaluation case for interpreting boundaries.

The flow makes the diagnostic decision explicit. It does not treat all disagreements as evidence against the model; it routes each observed transition toward a more specific question.

flowchart TD
  A[Run baseline, raw, partial-scale, and z-score variants] --> B[Inspect a fixed stress-evaluation row]
  B --> C{Raw wrong, z-score correct?}
  C -->|Yes| D[Preprocessing-fixable failure]
  C -->|No| E{Z-score still wrong?}
  E -->|Yes| F[Boundary data or feature question]
  E -->|No| G{Do settings disagree?}
  G -->|Yes| H[Setting-sensitive boundary case]
  G -->|No| I[Stable reference case]
  D --> J[Write one next action]
  F --> J
  H --> J
  I --> J

flowchart TD
  A[Fixed training rows and stress rows] --> B[Run four representation variants]
  B --> C[Compare accuracy and per-row transitions]
  C --> D[Separate fixed, remaining, and setting-sensitive failures]
  D --> E[Choose preprocessing, boundary data, or new feature question]
  E --> F[Record the next independent experiment]

Record the comparison¶

1. Evaluate baseline, raw 1-NN, partial scaling, and z-score 1-NN on the same rows.
2. Label each notable row as fixed by preprocessing, still remaining, or ambiguous.
3. Write whether the next change is preprocessing or boundary-data collection.

The original Korean practice code and chart asset produce the experiment table. The English manuscript preserves its learning decision: stress-01 is an example of a failure solved by preprocessing, stress-02 remains after normalization and needs boundary evidence, and stress-03 is sensitive to scale treatment and needs closer inspection.

Run the following code from the repository root to reproduce the four variants.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

import csv
from pathlib import Path
import numpy as np

train_path = Path("docs/assets/part-07/chapter-02/p7-2-churn-dataset.en.csv")
stress_path = Path("docs/assets/part-07/chapter-02/p7-2-stress-test.en.csv")
train_rows = list(csv.DictReader(train_path.open(encoding="utf-8")))
stress_rows = list(csv.DictReader(stress_path.open(encoding="utf-8")))
for row in train_rows + stress_rows:
    for column in ("unresolved_tickets", "days_since_login", "usage_minutes_30d", "label"):
        row[column] = int(row[column])
train_only = [row for row in train_rows if row["split"] == "train"]

def matrix(selected):
    return np.array([[row["unresolved_tickets"], row["days_since_login"], row["usage_minutes_30d"]] for row in selected], dtype=float)
def predict_1nn(train_x, train_y, test_x):
    predictions, nearest_ids = [], []
    for values in test_x:
        nearest_index = int(np.argmin(np.linalg.norm(train_x - values, axis=1)))
        predictions.append(int(train_y[nearest_index])); nearest_ids.append(train_only[nearest_index]["sample_id"])
    return np.array(predictions), nearest_ids

X_train, X_stress = matrix(train_only), matrix(stress_rows)
y_train = np.array([row["label"] for row in train_only]); y_stress = np.array([row["label"] for row in stress_rows])
baseline_label = int(np.bincount(y_train).argmax()); baseline = np.full(len(y_stress), baseline_label)
raw, raw_nearest = predict_1nn(X_train, y_train, X_stress)
partial_train, partial_stress = X_train.copy(), X_stress.copy()
partial_train[:, 2] /= 60; partial_stress[:, 2] /= 60
partial, partial_nearest = predict_1nn(partial_train, y_train, partial_stress)
mean, standard_deviation = X_train.mean(axis=0), X_train.std(axis=0)
if np.any(standard_deviation == 0): raise ValueError("A zero-standard-deviation feature cannot be z-score normalized.")
zscore, z_nearest = predict_1nn((X_train - mean) / standard_deviation, y_train, (X_stress - mean) / standard_deviation)

records = []
for row, base, raw_prediction, partial_prediction, z_prediction, raw_id, z_id in zip(stress_rows, baseline, raw, partial, zscore, raw_nearest, z_nearest):
    if raw_prediction != row["label"] and z_prediction == row["label"]: diagnosis = "fixed by preprocessing"
    elif z_prediction != row["label"]: diagnosis = "remaining boundary case after normalization"
    elif len({int(raw_prediction), int(partial_prediction), int(z_prediction)}) > 1: diagnosis = "setting-sensitive boundary case"
    else: diagnosis = "stable in current comparison"
    records.append({"sample": row["sample_id"], "actual": row["label"], "baseline": int(base), "raw": int(raw_prediction), "partial": int(partial_prediction), "zscore": int(z_prediction), "raw_nearest": raw_id, "z_nearest": z_id, "diagnosis": diagnosis})

print({"baseline": round(float((baseline == y_stress).mean()), 3), "raw": round(float((raw == y_stress).mean()), 3), "partial": round(float((partial == y_stress).mean()), 3), "zscore": round(float((zscore == y_stress).mean()), 3)})
for record in records:
    if record["sample"] in {"stress-01", "stress-02", "stress-03", "stress-04"}: print(record)

Interpret the comparison¶

Small synthetic experiments do not establish production performance. Their value is to turn a score into a concrete next data request or preprocessing check.

Stress-evaluation results and failure diagnoses¶

The shared stress evaluation makes four variants visible at once.

1
2
3
4
5
6
7
8

comparison summary = {
  'baseline accuracy': 0.361,
  'raw 1-NN accuracy': 0.861,
  'partial-scale 1-NN accuracy': 0.722,
  'z-score 1-NN accuracy': 0.833,
  'fixed by preprocessing': ['stress-01', 'stress-05', 'stress-13', 'stress-21', 'stress-29'],
  'remaining after normalization': ['stress-02', 'stress-03', 'stress-06', 'stress-14', 'stress-22', 'stress-30']
}

Raw 1-NN has the highest accuracy on this stress set. That does not make raw distance the universal choice: five rows are recovered by preprocessing, six remain after normalization, and eight rows vary by setting. The score and the failure diagnosis answer different questions.

The English chart presents these two layers together: the left panel is the four-variant score table; the right panel is the z-score-reference diagnosis distribution.

Read the result in this order.

1. Fact: raw 1-NN has the highest stress-evaluation accuracy here.
2. Failure diagnosis: normalization still leaves boundary cases and setting-sensitive rows.
3. Next action: preserve preprocessing checks for recovered rows; collect similar customers or new features for remaining and unstable rows.

The conclusion is not “always use z-score” or “always use raw distance.” It is “identify what kind of failure the present row represents.”

Read the 36-row stress result in layers¶

The stress set deliberately creates cases beyond the basic six-row evaluation. Its purpose is diagnostic coverage, not a new production score. Keep the following layers distinct.

The aggregate and the diagnosis can point in different directions without contradiction. Raw distance has the largest score here, while some particular rows are made correct only by preprocessing. A responsible record preserves both facts.

Four representative stress rows¶

| Row | Variant pattern | Diagnosis | First next action | | --- | --- | --- | | stress-01 | Raw retained; partial and z-score churn risk | Fixed by preprocessing | Check that scale treatment is repeatable. | | stress-02 | Raw correct; adjusted variants wrong | Remaining after normalization | Collect nearby churn-risk boundary examples. | | stress-03 | Raw correct; scale-adjusted variants wrong | Setting-sensitive retained boundary | Check feature sufficiency and labels. | | stress-04 | Most variants identify churn risk | Stable reference | Preserve as a regression example. |

Do not call stress-02 or stress-03 “noise” merely because their settings disagree. The disagreement is evidence that the current representation does not make the needed separation robustly.

Convert a diagnosis into a next data request¶

This is why “collect more data” is too vague. A good next-data request identifies the feature region, expected label, and failure transition that the new row is intended to test.

Add a second representation example¶

The same comparison discipline applies to action-unit sensor records. A raw tracking-error average may identify a one-off spike, while segment features and a baseline gap can identify repeated drift across comparable stages.

For the reused action-unit summary, E009, E011, and E012 are not raw-error flags but are segment-feature and baseline-gap candidates. E010 is a raw-error spike without the repeated segment signal. The appropriate action is different: reproduce E010, but investigate whether the repeated drift candidates share an upstream condition.

Choose a comparison axis before aggregating¶

This is another representation choice, not another classifier choice. Write which axis matches the project question before reporting an average across actions.

Independent experiments to try¶

1. Add two training rows near stress-02 while leaving scale rules unchanged.
2. Determine whether data coverage resolves the remaining boundary more directly than normalization.
3. Replace usage minutes with a stated behavioral feature such as session count.
4. Check whether stress-03 becomes stable across variants.
5. Change the partial-scale divisor from 60 to 30 and 120.
6. Record every row whose diagnosis changes; do not keep only the best accuracy.
7. Compare unequal actions using five-second bins and 25-percent progress bins.
8. State which axis answers the question and which information it discards.

Each is a separate experiment. Combining a new feature, new training rows, and a scale change would prevent a later reviewer from locating the source of a recovered or regressed result.

Project comparison log¶

1
2
3
4
5
6
7
8
9

training data version and fixed stress rows:
variant definitions and feature scales:
aggregate result for each variant:
fixed-by-preprocessing rows:
remaining-after-normalization rows:
setting-sensitive rows:
stable regression rows:
interpretation limited to the current comparison:
one next boundary row, feature, or preprocessing test:

The log makes a future score table interpretable. It also makes it possible to notice when a new variant improves a total score by trading one boundary failure for another.

Final learning check¶

• Did every variant use the same training and stress evaluation rows?
• Can you name a preprocessing recovery, a remaining boundary case, and a setting-sensitive case?
• Did you distinguish the highest aggregate score from the best diagnosis for one row?
• Did the next experiment change only one of scale, data coverage, feature choice, or comparison axis?
• Does the conclusion avoid claiming that any one representation is universally safest?

Keep score selection separate from action selection¶

A team may still need to select one variant for a constrained pilot. That selection should be documented separately from the failure diagnosis.

The highest score can inform a pilot decision. It cannot erase the list of diagnostics that made other variants useful. In particular, a raw-distance choice should retain tests that would expose scale dominance in the next data slice.

A safe comparison sentence¶

On the fixed 36-row synthetic stress set, raw 1-NN has the highest displayed accuracy, 0.861. Z-score preprocessing recovers five raw failures but leaves six boundary cases and the settings disagree on eight rows. The next iteration will retain all variants, collect cases near the unresolved boundaries, and test one stated feature change before selecting a general rule.

This sentence contains an aggregate fact, a sample-level diagnosis, and a bounded next action. It does not claim that raw or z-score is universally correct.

Inspect the source of a disagreement¶

When variants disagree, inspect four pieces of evidence before changing labels or features:

1. The raw feature values and units for the stress row.
2. The nearest training IDs under each representation.
3. The expected label and the label guideline used for that row.
4. The exact transformation, including any divisor or training-derived statistic.

The first two identify how the geometry changed. The third prevents a disputed business rule from being mistaken for a model issue. The fourth makes the run reproducible. If any field is missing, classify the row as “needs review” rather than as proof that a representation is defective.

Boundary-data request example¶

For a row like stress-02, a useful request is not “collect more churn data.” A more useful request is: “Collect both retained and churn-risk customers with similar ticket count, inactive days, and usage minutes to stress-02; retain payment outcome and recent session count if available.” This request names the region and the potential separating information.

For a setting-sensitive retained row like stress-03, collect nearby retained cases as well. Gathering only one label can make a nearest-neighbor boundary look artificially clear.

What remains outside this practice¶

The exercise does not estimate uncertainty across many random splits, choose a production decision threshold, or compare families of models. It also does not establish that the three provided features are ethically or operationally appropriate for a retention decision. Those questions require a broader data, governance, and evaluation process.

Its narrower value is to practice a reusable sequence:

1
2

fixed comparison rows → multiple stated representations → per-row transitions
→ failure diagnosis → one next preprocessing, data, feature, or axis question

Use this sequence before increasing model complexity. A larger model can still be evaluated through the same fixed rows, transitions, limits, and next-question record.

Reviewer checklist¶

If a comparison omits one of these fields, add it before naming a winner. The missing field is usually the information needed to tell a promising score from a reliable project decision.

A practical experiment sequence¶

Use the results in this order rather than tuning every setting at once.

1. Reproduce the default run. Verify the four accuracies and the named representative stress rows.
2. Choose one diagnosis. Select either a preprocessing recovery, a remaining boundary, or a setting-sensitive row.
3. State one intervention. Change one divisor, add one documented training region, or add one feature definition.
4. Keep the original stress rows. They are the regression evidence for the new run.
5. Compare transitions, not only the new score. List recoveries, regressions, and unchanged unresolved rows.
6. Update the next question. If a boundary remains, request more specific data instead of repeating the same scale change.

This sequence is slow enough to be explainable and fast enough to guide a small project. It replaces “try another model” with an experiment whose result can rule out at least one hypothesis.

Example next-iteration log¶

1
2
3
4
5
6
7

iteration: stress comparison, feature-scale review
unchanged evidence: 12 training rows and the 36 named stress rows
observed issue: stress-01 recovered by scaling; stress-02 remains unresolved
intervention: add a documented pair of retained/churn rows near stress-02
regression rows to retain: stress-01, stress-02, stress-03, stress-04
result fields: per-variant accuracy, neighbor ID, prediction transition, diagnosis
decision boundary: do not choose a new default until regressions are reviewed

The log records what was deliberately held fixed as well as what changed. That is what lets a future reader distinguish an experiment from an accumulation of untraceable improvements.

Final reminder¶

The goal is not to make every row agree across every setting. Some disagreement is valuable because it identifies where current data, features, or evaluation rules are too weak to support a confident decision. Preserve that disagreement as a target for the next comparison.

Keep the variant definitions with every score table. Keep the fixed stress-row IDs with every diagnosis. Keep raw and transformed feature units visible to reviewers. Keep a recovered row and a regression-risk row in the next test set. Keep a remaining boundary case open until new evidence changes it. Keep comparison-axis choices explicit for sensor summaries. Keep the selected pilot rule separate from the diagnostic evidence. Keep the conclusion limited to the documented synthetic comparison.

A comparison record for the next iteration¶

For example: stress-01 is misread as retained under raw distance, but becomes churn risk after scaling. Stress-02 and stress-03 change in different directions by variant, showing that normalization is not automatically safer for every boundary. The next iteration should collect boundary-region customer cases and consider a feature such as payment-failure count, rather than only tuning preprocessing further.

Extend the comparison to action-unit sensor records¶

The same reasoning applies to the synthetic action-unit summary in p7-action-unit-summary.en.csv.

The current example separates one-off event E010 from repeated segment-structure changes in E009, E011, and E012. The priority is not merely raising the raw-error threshold. It is retaining segment features and baseline gaps together so repeated drift has evidence.

1
2
3
4

E009: raw flag false, segment flag true, baseline-gap flag true → missed by raw errors but found by segment features
E010: raw flag true, segment flag false, baseline-gap flag false → possible one-off spike; reproduce it
E011: raw flag false, segment flag true, baseline-gap flag true → repeated drift candidate
E012: raw flag false, segment flag true, baseline-gap flag true → repeated drift candidate

Choose an axis before comparing actions¶

This is not a request for another model. It is a representation choice: decide whether the present question concerns elapsed seconds or comparable action stages before aggregating the sensor records.

Try changes directly¶

1. Add two training cases similar to stress-02; determine whether data coverage helps more directly than normalization.
2. Replace usage minutes with another behavioral feature such as average session count; determine whether stress-03 separates more clearly.
3. Change the partial scale divisor from 60 to 30 and 120; record how sensitive the variant is.
4. Compare unequal actions by five-second windows and by 25% progress windows; state which axis fits the question.

The decision criterion is not the highest score alone. A failure removed after preprocessing suggests a scale issue; a failure surviving every setting suggests that data boundaries or the question definition need review.

Checklist¶

Final handoff¶

Keep every experiment setting beside its outcome. Compare recovered, remaining, and newly wrong rows. Preserve the named boundary case for the next trial. State which preprocessing component changed. Do not treat one improved metric as a complete explanation. Keep the original test condition fixed. Record the next data need.

Sources and references¶

The customer and stress records are synthetic practice data created for this book.