P7-4.2 Decomposing a Failed Result Again¶

Section ID: P7-4.2 Version: v2026.08.01

When a result fails, decompose it before replacing the model. Separate data issues, representation issues, optimization signals, metric choice, and individual error patterns. A failure record should narrow the next check rather than assign blame to one component prematurely.

The practical sequence is baseline comparison, token coverage and OOV inspection, then a search for repeated error patterns. This order prevents a low score from becoming an unsupported demand for a larger model.

Learning questions and criteria¶

• Which evidence distinguishes a baseline limit from an input-representation limit?
• Why is low vocabulary coverage a review signal rather than an automatic error label?
• What smallest follow-up test would separate normalization, missing data, and model-capacity hypotheses?

You should be able to write one failure record with a confirmed observation, a limited interpretation, and a next probe.

A failure is several possible questions¶

Write a confirmed fact, a limited interpretation, and a next test. Do not describe a failed result as proof that a model family is inherently unsuitable without inspecting these layers.

For the routing practice data, retain the same train/test split as P7-4.1. The purpose is not to claim production performance; it is to make the evidence needed for a failure diagnosis visible.

Read token coverage beside the routing result¶

Keep the P7-4.1 support-routing data fixed. Token coverage is the fraction of an evaluation sentence that occurs in the training vocabulary. Low coverage is a review signal, not an automatic error label.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import csv
from pathlib import Path
import numpy as np

rows = list(csv.DictReader(Path("docs/assets/part-07/chapter-04/p7-4-support-routing-dataset.en.csv").open(encoding="utf-8")))
train_rows = [row for row in rows if row["split"] == "train"]
test_rows = [row for row in rows if row["split"] == "test"]
def tokens(text): return text.split()
vocab = sorted({token for row in train_rows for token in tokens(row["text"])})
index = {token: position for position, token in enumerate(vocab)}
def vectorize(selected):
    matrix, token_sets, oov_sets = np.zeros((len(selected), len(vocab))), [], []
    for row_number, row in enumerate(selected):
        token_list = tokens(row["text"]); token_sets.append(token_list); oov = []
        for token in token_list:
            if token in index: matrix[row_number, index[token]] += 1
            else: oov.append(token)
        oov_sets.append(oov)
    return matrix, token_sets, oov_sets
X_train, _, _ = vectorize(train_rows)
y_train = np.array([int(row["label"]) for row in train_rows])
profiles = np.vstack([X_train[y_train == label].sum(axis=0) for label in (0, 1)])
X_test, token_sets, oov_sets = vectorize(test_rows)
for row, token_list, oov, vector in zip(test_rows, token_sets, oov_sets, X_test):
    scores = profiles @ vector; prediction = int(scores.argmax()); actual = int(row["label"])
    coverage = round((len(token_list) - len(oov)) / len(token_list), 3) if token_list else 0
    print({"sample": row["sample_id"], "tokens": token_list, "oov": oov, "coverage": coverage, "refund_score": int(scores[0]), "delivery_score": int(scores[1]), "prediction": prediction, "actual": actual, "review_needed": coverage < .5 or prediction != actual})

Two evaluation samples have low coverage, but they lead to different outcomes. In the current CSV, 평가-05 has coverage 0.200 and is a refund request predicted as delivery: its key intent word 캔슬 is out of vocabulary while the remaining shipping term receives score. 평가-07 has coverage 0.333 but remains correctly routed because known refund-related terms are strong.

The figure makes that distinction visible. Green bars are correct and red bars are incorrect; the dashed line is a review threshold, not an error boundary.

The next stages are representation normalization, adding data for a missing expression family, and only then reconsidering a larger model structure.

The two-sample flow rejects the tempting but incorrect shortcut: “both are low coverage, so both have the same cause.”

flowchart TD
  A[Find two samples with coverage 0.333] --> B[Quick conclusion: call them the same failure]
  A --> C[Check correctness and key OOV terms]
  C --> D[Test-05: incorrect; intent word is OOV]
  C --> E[Test-07: correct; strong intent words remain]
  D --> F[Prioritize normalization and review before a structural change]
  E --> F

Interpret coverage and score together¶

Coverage answers “how much of this sentence is represented in the training vocabulary?” It does not answer “which team should receive this request?” Scores show which known words each class profile can use. An incorrect low-coverage sample may have a missing intent word, competing known words, or both.

Do not erase the correct low-coverage case from the log. It is evidence that coverage is not a classifier metric by itself, and it can reveal regressions after a vocabulary or normalization change.

Use a complete failure split¶

The broader decision flow keeps coverage in the right place. It comes after the baseline comparison and before a stronger claim about representation or model limits.

flowchart TD
  A[Find an incorrect or risky sample] --> B{Does the baseline also fail this kind?}
  B -->|Yes| C[Check baseline limits or label distribution]
  B -->|No| D{Is coverage low or is a key term OOV?}
  C --> D
  D -->|Yes| E[Check normalization, tokenization, and data coverage]
  D -->|No| F{Do similar expressions fail repeatedly?}
  E --> F
  F -->|Yes| G[Consider representation or model limits]
  F -->|No| H[Record a boundary case or label ambiguity]

When the baseline fails the same type of request, inspect labels, data distribution, and the baseline rule first. When the baseline is adequate but a key word is OOV, try a reversible representation change first. Only a repeated failure pattern with adequate coverage is stronger evidence for a feature or model limitation.

Project record and controlled checks¶

Record the following fields for each review sample:

1
2
3
4
5
6
7
8

sample identifier:
tokenization rule:
known tokens and OOV tokens:
coverage:
class scores, prediction, and actual label:
baseline result:
confirmed observation:
smallest next probe:

Then run controlled changes one at a time.

1. Replace 캔슬 with 취소 in test-05 and compare coverage, scores, and prediction.
2. Replace 스케줄 with 일정 in test-07; record whether the preventive-risk status changes even when the prediction remains correct.
3. Add a new unseen synonym without changing the model. Explain why a lower coverage result supports a representation hypothesis, not immediately a capacity hypothesis.
4. Pick one bar below the threshold and write both its observation and the causal claim that the data does not justify.

The next section tests a normalization rule directly. Keep this section’s before-state values so that any improvement can be attributed to a specified representation change rather than a vague “model improvement.”

Turn a coverage result into a failure record¶

The code output is most useful when it is compressed into an auditable record rather than copied as a long console listing.

1
2
3
4
5
6
7
8
9

task: route an inquiry to refund or delivery
tokenization: split on spaces
evaluation reference: seven fixed inquiries
coverage review threshold: below 0.500
incorrect sample: 평가-05
preventive review sample: 평가-07
confirmed observation: 평가-05 is low coverage and predicted delivery instead of refund
limited interpretation: a missing cancellation expression and known shipping words may compete
next probe: normalize a documented synonym, then rerun the same rows

The record has a deliberate boundary. It reports what occurred in the current representation; it does not assert that a tokenizer, data collector, or model architecture is the sole cause.

Distinguish three review statuses¶

Coverage alone assigns none of these labels. It tells the reviewer where an input representation may deserve attention. Prediction and validated label evidence determine whether a row is an error.

Read the two low-coverage inquiries side by side¶

The comparison provides a small counterexample to a tempting rule: “low coverage means low quality.” 평가-07 has low coverage but a correct route. The more defensible rule is: “low coverage requires a sample-level review, especially when it coincides with an error.”

Separate representation, data, and model questions¶

One incorrect result can support several hypotheses. Make them testable by stating what changes and what stays fixed.

The order is practical rather than universal. A reversible representation test is inexpensive and directly connected to the observed OOV. If it fails, the record gives a clearer reason to test data expansion or another representation.

Avoid changing several explanations at once¶

Inspect baseline and coverage in the right order¶

Baseline comparison answers a different question from coverage. A majority-label baseline shows what happens when the input text is ignored. Coverage shows what the selected text representation can recognize. Neither alone measures the quality of a routing policy.

This ordering prevents “the baseline is weak” from becoming “the model needs to be larger,” and prevents “coverage is low” from becoming “the predicted label is wrong.”

A reusable retrospective sentence¶

평가-05 was routed to delivery although its stored label is refund. Under space-splitting tokenization, its coverage is 0.200 and the cancellation expression is out of the training vocabulary while a shipping-document term remains known. 평가-07 is also low coverage but stays correct, so coverage alone is not the explanation. First apply one documented normalization rule and compare both named references; then decide whether missing mixed-intent training examples or a different representation needs testing.

This sentence contains a fact, a limited interpretation, a counterexample, and a next experiment. It deliberately stops before claiming that normalization will fix the failure.

Review fields to add to an evaluation report¶

These fields are useful even when a project uses subword tokenization or an embedding model. The exact definition of “known token” changes, but the need to document input coverage, a decision rule, and error references remains.

Practice questions before P7-4.3¶

1. Which facts make 평가-05 a representation review case rather than only a metric decrease?
2. Why does 평가-07 need to remain after a proposed correction?
3. What result would weaken the normalization hypothesis?
4. Which evidence would justify adding mixed-intent training examples?
5. Which elements must remain fixed to compare a model-family change fairly?

Answer the questions with the stored rows, not with an architecture name. P7-4.3 performs the normalization comparison and reports recovered as well as unchanged cases.

Use a decision procedure after every failure¶

The procedure below makes the diagnosis order explicit. It is not a guarantee that every failure has one cause.

1. Confirm the evaluation row, actual label, split, and prediction.
2. Compare the model with the stated baseline on the same rows.
3. Inspect tokens, known coverage, OOV terms, and class scores.
4. Classify the row as error, preventive risk, stable reference, or label question.
5. State one representation, data, policy, or model hypothesis as a conditional sentence.
6. Choose one reversible or smallest practical probe.
7. Rerun every retained reference row and report recovered, unchanged, and newly wrong IDs.
8. Update the hypothesis only after recording the transition.

The point of the sequence is not bureaucracy. It protects the project from a common failure pattern: changing preprocessing, training data, and classifier together, then attributing an aggregate score change to the most attractive explanation.

Example before-and-after report format¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

before representation: split on spaces, no synonym normalization
before primary error: 평가-05, coverage 0.200, predicted delivery, actual refund
before preventive case: 평가-07, coverage 0.333, predicted refund, actual refund
changed component: one documented expression normalization rule
after primary error: record coverage, scores, and prediction
after preventive case: record coverage, scores, and prediction
recovered IDs:
unchanged error IDs:
newly wrong IDs:
bounded conclusion:
next question:

The blank fields are intentional. Do not fill them with a prediction about the outcome before the run. A useful experiment report distinguishes an expectation from an observed transition.

What each outcome would mean¶

This table is a guardrail against declaring success from one recovered row. A controlled improvement includes the failures it introduces.

Connect failure records to project decisions¶

The decisions have different owners in a real project. A data curator, domain policy owner, and modeling engineer may each need to review a different evidence field. Keeping the record structured makes that handoff possible.

A compact learning review¶

Before leaving this section, a learner should be able to complete the following four statements.

1. 평가-05 is a confirmed error because its stored label and prediction disagree; its coverage is a supporting representation signal.
2. 평가-07 is a preventive review case because low coverage does not prevent a correct prediction.
3. A normalization test changes representation, so labels, evaluation IDs, and the reporting rule must remain fixed.
4. A recovery after normalization supports a narrow expression-mismatch explanation; it does not prove that the overall routing problem is solved.

These statements link the numerical output to a defensible next action. They are more useful than memorizing an arbitrary coverage threshold.

Limits of a vocabulary-coverage check¶

Coverage is intentionally simple in this exercise: it counts space-separated tokens that occurred in the training vocabulary. Production text systems may use subword units, character models, embeddings, spelling correction, or multilingual normalization. Those choices can reduce the number of unseen units, but they do not remove the need to inspect whether the representation preserves the domain meaning of a request.

Therefore, use coverage to prioritize reading, not to automate acceptance or rejection. The appropriate follow-up depends on the error’s input evidence, domain policy, and regression record.

Final handoff to the normalization test¶

Keep 평가-05 and 평가-07 visible in the next report. P7-4.3 changes one explicitly documented expression rule and compares the original text with the normalized text. Its result should be read as a test of that rule, not as proof that every OOV expression has been solved.

Final project retrospective¶

The routing evaluation contains one confirmed error and one correct low-coverage reference. The error has coverage 0.200 and lacks the cancellation expression in the current vocabulary, while known shipping evidence remains. The correct low-coverage reference demonstrates that coverage is a review signal rather than a correctness rule. The next run changes one documented normalization rule, retains every evaluation ID, and reports recovered, unchanged, and newly wrong cases before making another diagnosis.

Use this structure for a later project as well:

• State the observed row-level facts.
• Name the representation or data signal without overclaiming cause.
• Preserve a counterexample or preventive reference.
• Specify one changed component and the fixed evaluation evidence.
• End with the question that remains after the run.

The threshold is a review convention for this exercise, not a universal operating rule. Document why it was chosen before using it to prioritize work. If the threshold changes, retain the earlier review list so that the change is visible. The review convention cannot replace label validation or error analysis. It only determines which stored rows are read first.

Checklist¶

Sources and references¶

Keep the review threshold with the stored rows. Separate confirmed errors from rows selected for preventive review. Change one diagnosis component at a time. Preserve the fixed evaluation condition. Record the next question before choosing a remedy. Do not let a priority signal become a causal claim. Keep unresolved label evidence visible. Report new regressions as well as recoveries.

This section uses internal practice examples.