P3-4.5 How Well Does the Sample Set We Collected Represent the Overall Operating Situation¶

Section ID: P3-4.5 Version: v2026.07.24

Once the sample unit has been fixed as something like one full action or one recent segment, one more question remains that is easy to miss. How well does the sample set we collected represent the overall operating situation? Even if the table itself is well organized, if the cases in it were gathered only from a specific process mode, a specific time period, or a specific equipment state, then the table may fail to describe the overall operating scene evenly. Choosing the sample unit correctly and having a sample bundle that evenly represents the whole situation are not the same thing.

What Must the Representativeness of the Sample Bundle Be Distinguished From¶

The problem of representativeness asks, separately from whether the definition of one sample is correct, what operating range the sample bundle actually covers.

In other words, the definition of one sample and the representativeness of the sample bundle are different problems.

Representative Scenes Where Representativeness Becomes Weak¶

Even if all samples are the same unit of one full action, representativeness can still differ depending on from which range they were gathered.

So even if the number of samples is large, representativeness can still be weak when the covered conditions are narrow.

Four Things Worth Writing Down First¶

In Part 3, what matters more than formal sampling theory at this stage is to write down the following four things first.

These notes are not for proving generalization later. They are for first making visible what the current table does represent and what it still does not represent.

A Small Diagram¶

flowchart TD
    A[Current sample set]
    A --> B[Mostly day shift]
    A --> C[Mostly machine M1]
    A --> D[Only one high-load case]
    A --> E[Only one after-maintenance case]

    B --> F[Coverage is uneven]
    C --> F
    D --> F
    E --> F

This diagram shows that even if every sample unit is consistently one full action, the operating range it covers can still be tilted to one side. In other words, the point of this section's example is not to read many raw table values, but to identify first which conditions are overrepresented and which conditions are nearly empty.

Why This Problem Has to Come After the Sample Unit¶

The problem of representativeness can only be read after the sample unit has first been fixed. If it is still unclear whether one row means a time-point record or one full action, then questions such as how many night-shift actions are there? or how many high-load-condition samples are there? cannot even be counted properly.

So the order is as follows.

1. First decide what will count as one sample.
2. Then check what condition range those samples actually cover.

Only after these notes are left behind can we later read results together with from what condition range was this sample bundle obtained?, and avoid missing what operating conditions were barely seen? In that sense, the problem of representativeness is close to the problem of first writing down what the current sample bundle covers and what it misses.

Small Code Example¶

Problem situation: even if all sample units are correctly aligned as one full action, check which conditions the actual sample bundle is tilted toward.

Input: the action-sample table stored in p3_4_5_sample_coverage.csv and the minimum observation criterion minimum_count. This table contains shift, load_mode, machine_id, and maintenance_phase.

Expected output: a coverage summary showing which conditions were seen a lot and which were nearly empty. If minimum_count changes, the number of conditions marked as representativeness gaps also changes.

Concept to check: the correctness of one sample definition and the even representativeness of the sample bundle over the whole operating range are different problems. A representativeness judgment needs an observation criterion.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

# This example checks how well collected samples represent the overall operation by category and time window.
import csv
from collections import Counter
from pathlib import Path

minimum_count = 9
preview_sample_count = 8

input_path = Path("docs/assets/part-03/chapter-04/p3_4_5_sample_coverage.csv")
coverage_scopes = ["shift", "load_mode", "machine_id", "maintenance_phase"]

with input_path.open(newline="", encoding="utf-8") as file:
    samples = list(csv.DictReader(file))

coverage_summary = []
for scope in coverage_scopes:
    counts = Counter(sample[scope] for sample in samples)
    ordered_counts = sorted(counts.items(), key=lambda item: (-item[1], item[0]))
    most_seen, most_seen_count = ordered_counts[0]
    least_seen, _ = sorted(counts.items(), key=lambda item: (item[1], item[0]))[0]
    under_minimum = sum(1 for count in counts.values() if count < minimum_count)
    coverage_summary.append(
        {
            "scope": scope,
            "most_seen": most_seen,
            "count": most_seen_count,
            "least_seen": least_seen,
            "unique_conditions": len(counts),
            "under_minimum_conditions": under_minimum,
        }
    )

print("1) raw sample coverage table")
for sample in samples[:preview_sample_count]:
    print(
        f"{sample['event_id']}: shift={sample['shift']}, "
        f"load_mode={sample['load_mode']}, machine_id={sample['machine_id']}, "
        f"maintenance_phase={sample['maintenance_phase']}"
    )
print(f"... {len(samples) - preview_sample_count} more event-level samples")
print()
print(f"2) coverage summary when minimum_count = {minimum_count}")
for item in coverage_summary:
    print(
        f"{item['scope']}: most_seen={item['most_seen']} ({item['count']}), "
        f"least_seen={item['least_seen']}, "
        f"unique_conditions={item['unique_conditions']}, "
        f"under_minimum_conditions={item['under_minimum_conditions']}"
    )

Expected output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

1) raw sample coverage table
E01: shift=day, load_mode=normal, machine_id=M1, maintenance_phase=stable
E02: shift=day, load_mode=normal, machine_id=M1, maintenance_phase=stable
E03: shift=day, load_mode=normal, machine_id=M1, maintenance_phase=stable
E04: shift=day, load_mode=normal, machine_id=M1, maintenance_phase=stable
E05: shift=day, load_mode=normal, machine_id=M1, maintenance_phase=stable
E06: shift=day, load_mode=normal, machine_id=M1, maintenance_phase=stable
E07: shift=day, load_mode=normal, machine_id=M1, maintenance_phase=stable
E08: shift=day, load_mode=normal, machine_id=M1, maintenance_phase=stable
... 28 more event-level samples

2) coverage summary when minimum_count = 9
shift: most_seen=day (26), least_seen=night, unique_conditions=2, under_minimum_conditions=0
load_mode: most_seen=normal (25), least_seen=low, unique_conditions=3, under_minimum_conditions=2
machine_id: most_seen=M1 (22), least_seen=M2, unique_conditions=3, under_minimum_conditions=2
maintenance_phase: most_seen=stable (28), least_seen=after-maintenance, unique_conditions=2, under_minimum_conditions=1

What matters in this example is not a classification technique, but making visible at a glance what the current table sees a lot of and what it barely sees. The value to manipulate here is minimum_count. When minimum_count = 9, some scopes such as shift have all conditions above the criterion, while scopes such as load_mode, machine_id, and maintenance_phase have some conditions marked as representativeness gaps. If this value is lowered, the gaps decrease; if it is raised, more conditions are marked as insufficient. That is how we can explain with both numbers and a table why even with 36 samples, representativeness can look different by condition.

When reading this table, three things should be checked together. Can this table explain the time, mode, and equipment range from which it collected samples? Can we write down the conditions that were barely seen? And later, when reading evaluation scores, can we also bring back to mind this range of representativeness? Only when notes like these are attached does the sample table become not just an organized table, but a table that also records what operating range it represents.

A representativeness gap also appears later in model evaluation. The next example uses the same CSV, takes the first 24 rows as the training bundle, and the last 12 rows as the checking bundle. The training bundle is dominated by normal and stable conditions, while the checking bundle contains more low and after-maintenance conditions. Here we make a reduced label, needs_review, which is 1 for high load or after-maintenance conditions, and compare a simple baseline with a small decision tree.

Problem situation: We want to see which conditions receive the errors of a baseline and a model when the training bundle is not representative.

Input: The same p3_4_5_sample_coverage.csv, categorical condition columns, and the reduced label needs_review.

Expected output: Condition distribution in the training/checking bundles, baseline and decision-tree accuracy, and the number of errors by load_mode.

Concept to check: If we look only at one overall accuracy number, the operating conditions that were barely seen can stay hidden, so representativeness gaps should be read together with condition-level errors.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

# This example checks where baseline and model errors concentrate when training coverage is biased.
import pandas as pd
from pathlib import Path
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.tree import DecisionTreeClassifier

input_path = Path("docs/assets/part-03/chapter-04/p3_4_5_sample_coverage.csv")
samples = pd.read_csv(input_path)

# This is a reduced label for this section's observation. A real operational label needs separate review.
samples["needs_review"] = (
    samples["load_mode"].eq("high") | samples["maintenance_phase"].eq("after-maintenance")
).astype(int)

train = samples[samples["event_id"].between("E01", "E24")]
test = samples[samples["event_id"].between("E25", "E36")]
features = ["shift", "load_mode", "machine_id", "maintenance_phase"]

preprocess = ColumnTransformer(
    [("category", OneHotEncoder(handle_unknown="ignore"), features)]
)
models = {
    "dummy": DummyClassifier(strategy="most_frequent"),
    "tree": DecisionTreeClassifier(random_state=0, max_depth=3),
}

print("train coverage")
print(train.groupby(["load_mode", "maintenance_phase"])["event_id"].count())
print()
print("test coverage")
print(test.groupby(["load_mode", "maintenance_phase"])["event_id"].count())
print()

for name, estimator in models.items():
    model = make_pipeline(preprocess, estimator)
    model.fit(train[features], train["needs_review"])
    predicted = model.predict(test[features])
    result = test.assign(
        predicted=predicted,
        error=lambda df: df["predicted"].ne(df["needs_review"]),
    )
    print(f"{name} accuracy:", accuracy_score(test["needs_review"], predicted))
    print("errors by load_mode:", result.groupby("load_mode")["error"].sum().to_dict())

Expected output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

train coverage
load_mode  maintenance_phase
high       stable                4
normal     after-maintenance     2
           stable               18
Name: event_id, dtype: int64

test coverage
load_mode  maintenance_phase
high       after-maintenance    1
           stable               1
low        after-maintenance    2
           stable               3
normal     after-maintenance    3
           stable               2
Name: event_id, dtype: int64

dummy accuracy: 0.4166666666666667
errors by load_mode: {'high': 2, 'low': 2, 'normal': 3}
tree accuracy: 0.75
errors by load_mode: {'high': 0, 'low': 3, 'normal': 0}

If we look only at overall accuracy, the decision tree appears better than the baseline. But errors by load_mode shows that errors remain in the low condition. That condition was absent from the training bundle and appears for the first time in the checking bundle. So this output makes us ask first not how well did the model score?, but which conditions were barely seen before evaluation? Representativeness checking is a table check before training a model, and it is also a condition check we must return to when reading model evaluation.

Fixing the sample unit correctly does not automatically mean that the sample bundle represents the whole operating situation. That is why, in Part 3, the time range, mode range, equipment range, and remaining gaps should all be written down together.

Sources and Further Reading¶

• Google for Developers, Machine Learning Glossary: labeled example. Because the example unit has to be fixed before we can ask what set of examples represents the current problem, it strengthens this section's starting point that the definition of one sample and the representativeness of the sample bundle should be read separately. https://developers.google.com/machine-learning/glossary / Accessed: 2026-07-20
• W3C, PROV-Overview. Because it explains that provenance and activity context should be preserved together, it provides the higher-level frame that to explain the range of representativeness, we must be able to trace from what time period, what machine, and what operating mode the current sample bundle came. https://www.w3.org/TR/prov-overview/ / Accessed: 2026-07-20
• NIST/SEMATECH e-Handbook of Statistical Methods, What are Variables Control Charts?. Because it explains that when current performance is compared with past performance, the samples should come from the same essential conditions, it provides a general basis for the claim that before sample count, the operating conditions covered by the data must first be organized. https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc32.htm / Accessed: 2026-07-20