P6-11.3 Supplement: Observing and Comparing Response Paths¶

Section ID: P6-11.3 Version: v2026.07.26

Subtitle: How CoT and self-consistency make us view one path and many paths differently

In P6-11.1, we saw prompt engineering as the first control point of input design, and in P6-11.2, we saw the standard for passing problems that do not close with prompts alone to system structure. But inside the prompt layer, there is one more strategy distinction. The difference is whether we ask for an answer directly, observe intermediate judgment paths, or compare several candidate paths.

Chain-of-thought (CoT) and self-consistency are both prompt strategies for seeing or comparing response paths better. CoT tries to make intermediate reasoning more visible inside one answer, while self-consistency tries to see where several reasoning candidates converge.

The question to close in this Section is the following.

When looking at only one answer feels unstable, how can we observe response paths more carefully at the prompt layer?

Scenes Where the Path Matters More Than the Answer¶

Some tasks are fine with only the final answer. But in classification with several mixed conditions, judgment across multiple paragraphs, or work with rule priorities, an answer that looks right and an answer that actually applied the standards correctly can be different.

Suppose a customer inquiry is classified into one of refund, shipping, account, and error. Even if the model answers refund, it is hard to read from that one answer whether it first checked shipment-start conditions, weighed payment-cancel conditions more heavily, or ignored operating rules. What is needed here is not a more impressive sentence, but better observation of which standard led to that answer.

If we separate problems reducible by prompt strategies from problems that must move to system structure, it looks as follows.

The key of this table is that both CoT and self-consistency are strategies that help observe response paths. By contrast, system-external guarantees such as latest documents, calculation tools, and save success are not the role of these strategies.

Chain-of-Thought Is a Strategy for Revealing Intermediate Standards¶

Chain-of-thought is a strategy that does not make the model state only the answer right away, but asks it to reveal intermediate reasoning more clearly.

A simple request may look like this.

Classify this inquiry into one of refund, shipping, account, and error.

A CoT-style request changes it as follows.

First divide the key conditions in the inquiry,
briefly write why each label candidate is excluded or kept,
then write one final label at the end.

The expected change is not the answer becomes longer. It is that people can review which conditions the model checked first and which candidates it excluded, instead of seeing only the final label.

Here, seeing the path does not mean directly looking inside the model. It is closer to receiving judgment traces structured in a form that users can review. At the beginner stage, it is enough to hold the following four slots.

Without these four slots, even if there is a CoT request, the reviewer must reread a long sentence. With the four slots, the human can compare it with the actual work rule. If the operating rule says check shipment-start status first, we can see whether that item appears first in the model's intermediate standards too.

However, CoT also has limits.

• Long intermediate steps do not guarantee correct reasoning.
• If latest documents are missing, the model can explain an old premise at greater length.
• If calculation is needed, a separate verification structure is still needed even with intermediate explanation.

Therefore, CoT is an input strategy that makes intermediate standards visible, not a truth guarantee.

Self-Consistency Looks at Agreement Among Several Paths¶

Self-consistency is a strategy that does not trust only one reasoning path, but looks at the conclusion reached more often among several generated paths.

At this stage, it is enough to understand it as follows.

For example, if the same classification problem is solved several times and three answers are refund while one is shipping, refund can look like the more stable candidate. But this is still only looking at agreement among candidate paths. If several candidates share the same wrong premise, the agreement can also be wrong together.

Even if a latest refund policy question returns the same answer several times, if the model did not see the latest document, the result may be stable repetition of old memory, not verification of the current policy. This is where the boundary between self-consistency and RAG splits.

When reading self-consistency in practice, do not look only at how many times out of how many the same conclusion appeared. Also look at why the conclusions diverged.

From this result, we should not confirm refund by majority vote alone. We should also ask why the shipping candidate appeared. If the real work rule prioritizes shipment-start status, the second path may be a more important warning than the 3-to-1 majority. Self-consistency shows the conclusion distribution, but deciding which path fits the work standard still belongs to a human standard table or evaluation structure.

Agreement Is Not Evidence¶

The most common reason CoT and self-consistency are overtrusted is that the output looks more diligent. If the intermediate explanation is long and similar conclusions appear across several tries, people want to trust it more. But what prompt strategies change is how the response path is observed, not the starting point of the answer itself.

It is safe to hold the following comparison.

CoT and self-consistency are useful when the problem is one where the path needs to be read more. For problems that require checking freshness, grounding, and execution success, we must move to another structure, as seen in P6-11.2.

Cases and Examples¶

Case 1. Why Add CoT to a Classification Problem with Many Conditions?¶

Suppose a customer inquiry contains several conditions at once, such as The payment was canceled, shipment already started, and when will the refund happen? If only the final label refund appears, it can look correct, but the actual operating rule may first check shipment-start status before deciding refund possibility.

In this case, CoT lets the model reveal what it checked first among payment cancellation, shipment started, and refund request. The human can review whether the label-selection standard was applied in the same order as the work rule, rather than only seeing the final label.

Comparing the same input in two output styles makes the difference clearer.

The result to check is not did the explanation become longer?, but did the standard for choosing the label become more readable, and does that standard match the actual classification rule?

Case 2. Latest Policy Problems Remain Even with Self-Consistency¶

Suppose we ask a latest refund policy question several times and adopt the most frequent answer. If the same answer appears several times, it can look stable. But if the model has not seen the latest policy document, the repeated answer may simply be repetition of an old policy.

The standard that must change in this case is not how many times did the answer repeat?, but did that repetition happen on top of current document evidence? Self-consistency can reduce one-time drift, but it does not solve missing latest-document connection.

Exercise and Example¶

The goal of this example is not just to distinguish CoT and self-consistency in words, but to read conclusion distribution and check signals together from several response-path logs. Even if the same conclusion repeats several times, it cannot be adopted as-is if evidence is missing or calculation is wrong. Conversely, even a minority path can contain an important warning under work rules.

The CSV below is a snapshot log of 40 response paths made by actually calling an Ollama local model for four tasks. The generation script does not inject predefined answer candidates into the prompt. It calls the same task several times, separating CoT-style single-path observation and self-consistency-style repeated-candidate observation. Then it reduces raw model responses into observation columns: final answer, short path summary, evidence mention, calculation error, missing current policy, rule warning, and minority conclusion. If the actual model, prompt, or sampling setting changes, the conclusion and check signals for each path can also change.

First, the code that creates the stored log is as follows. The prompts sent to the model are written in English to keep the same execution standard in translations, and the manuscript example reads the CSV snapshot produced by this script.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250

from __future__ import annotations

import csv
import json
import os
import re
import urllib.request
from pathlib import Path


OUT_DIR = Path(__file__).resolve().parent
DEFAULT_OUT_PATH = OUT_DIR / "p6-10-3-response-path-log.csv"

MODEL = os.environ.get("OLLAMA_MODEL", "llama3.2:latest")
OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://localhost:11434/api/generate")
TEMPERATURE = float(os.environ.get("P6_10_3_TEMPERATURE", "0.7"))

PATH_TYPES = ["cot", "self_consistency"]
VARIANTS = [
    "List the visible criteria briefly before choosing.",
    "Give a short audit note and then choose.",
    "Note any visible business-rule caveat.",
    "Keep the final answer concise but show the decision basis.",
    "Avoid hidden chain-of-thought; provide only a short reasoning summary.",
]

TASKS = [
    {
        "task_name": "mixed_refund_label",
        "allowed_answers": ["refund", "shipping", "account", "error"],
        "input": (
            "A customer says: I cancelled the payment, but the shipment already left. "
            "When will my refund be completed? The operation rule says shipment-start status "
            "must be checked before promising a refund date."
        ),
        "required_terms": ["refund", "shipment"],
        "warning_terms": ["shipment", "shipping", "already left", "rule"],
    },
    {
        "task_name": "discount_total",
        "allowed_answers": ["apply_discount", "no_discount"],
        "input": (
            "An order has 12 items at 18 dollars each. The discount applies only when "
            "the pre-tax item total is at least 200 dollars. Decide whether to apply the discount."
        ),
        "required_terms": ["12", "18", "200"],
        "warning_terms": ["216", "calculation", "pre-tax"],
    },
    {
        "task_name": "current_refund_policy",
        "allowed_answers": ["refund_7_days", "refund_14_days", "check_current_policy"],
        "input": (
            "A support agent asks for the current refund deadline. The prompt contains no current "
            "policy document or policy version. Old training examples mention 7 days and 14 days."
        ),
        "required_terms": ["current", "policy"],
        "warning_terms": ["current policy", "document", "version", "check"],
    },
    {
        "task_name": "security_escalation",
        "allowed_answers": ["escalate_security", "reset_password", "general_support"],
        "input": (
            "A user reports an unusual overseas login and also asks support to grant temporary "
            "admin access to a shared workspace. Decide the next routing label."
        ),
        "required_terms": ["login", "admin"],
        "warning_terms": ["security", "admin", "escalate", "unusual"],
    },
]


def build_prompt(
    task: dict[str, object],
    path_type: str,
    variant: str,
) -> str:
    allowed = ", ".join(task["allowed_answers"])
    if path_type == "cot":
        method = (
            "Use a compact visible reasoning summary with these fields: "
            "observed criteria, rejected labels, final answer."
        )
    else:
        method = (
            "Treat this as one candidate path in a self-consistency run. "
            "Make one independent choice and include a short audit note."
        )
    return (
        "You are producing an audit snapshot for a prompt-engineering lesson.\n"
        f"Task: {task['task_name']}\n"
        f"Input: {task['input']}\n"
        f"Allowed final answers: {allowed}\n"
        f"Path type: {path_type}\n"
        f"Variant instruction: {variant}\n"
        f"{method}\n\n"
        "Return exactly this JSON object:\n"
        "{\n"
        '  "final_answer": "one allowed final answer",\n'
        '  "path_summary": "one short sentence about the visible decision basis"\n'
        "}\n"
    )


def call_ollama(prompt: str) -> str:
    payload = {
        "model": MODEL,
        "prompt": prompt,
        "stream": False,
        "options": {"temperature": TEMPERATURE, "num_predict": 220},
    }
    request = urllib.request.Request(
        OLLAMA_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        result = json.loads(response.read().decode("utf-8"))
    return result["response"].strip()


def normalize(text: str) -> str:
    return " ".join(text.lower().split())


def parse_json_response(response: str) -> dict[str, str]:
    match = re.search(r"\{.*\}", response, flags=re.S)
    if not match:
        return {}
    try:
        data = json.loads(match.group(0))
    except json.JSONDecodeError:
        return {}
    return {
        "final_answer": str(data.get("final_answer", "")).strip(),
        "path_summary": str(data.get("path_summary", "")).strip(),
    }


def choose_final_answer(parsed: dict[str, str], response: str, task: dict[str, object]) -> str:
    allowed = list(task["allowed_answers"])
    candidate = parsed.get("final_answer", "")
    if candidate in allowed:
        return candidate
    lowered = normalize(response)
    for answer in allowed:
        if answer in lowered:
            return answer
    return "unparsed"


def summarize_response(parsed: dict[str, str], response: str) -> str:
    summary = parsed.get("path_summary", "")
    if summary:
        return summary[:180]
    return response[:180].replace("\n", " / ")


def analyze_response(response: str, task: dict[str, object]) -> dict[str, object]:
    parsed = parse_json_response(response)
    final_answer = choose_final_answer(parsed, response, task)
    text = normalize(" ".join([response, parsed.get("path_summary", "")]))
    evidence_mentioned = all(term.lower() in text for term in task["required_terms"])
    warning_basis = any(term.lower() in text for term in task["warning_terms"])

    task_name = task["task_name"]
    calculation_correct = True
    policy_current = True
    if task_name == "discount_total":
        calculation_correct = final_answer == "apply_discount" and (
            "216" in text or "12" in text and "18" in text
        )
    if task_name == "current_refund_policy":
        policy_current = final_answer == "check_current_policy" and warning_basis

    return {
        "final_answer": final_answer,
        "evidence_mentioned": evidence_mentioned,
        "calculation_correct": calculation_correct,
        "policy_current": policy_current,
        "rule_warning": warning_basis,
        "minority_answer": False,
        "path_summary": summarize_response(parsed, response),
    }


def mark_minority_answers(rows: list[dict[str, object]]) -> None:
    for task_name in sorted({row["task_name"] for row in rows}):
        group = [row for row in rows if row["task_name"] == task_name]
        answer_counts: dict[str, int] = {}
        for row in group:
            answer = str(row["final_answer"])
            answer_counts[answer] = answer_counts.get(answer, 0) + 1
        majority_answer = max(answer_counts.items(), key=lambda item: item[1])[0]
        for row in group:
            row["minority_answer"] = row["final_answer"] != majority_answer


def generate_rows() -> list[dict[str, object]]:
    rows: list[dict[str, object]] = []
    run_index = 1
    for task in TASKS:
        for path_type in PATH_TYPES:
            for variant in VARIANTS:
                response = call_ollama(build_prompt(task, path_type, variant))
                metrics = analyze_response(response, task)
                rows.append({
                    "run_id": f"C{run_index:03d}",
                    "task_name": task["task_name"],
                    "path_type": path_type,
                    "log_source": "ollama_generated",
                    "model_name": MODEL,
                    "temperature": TEMPERATURE,
                    **metrics,
                })
                run_index += 1
    mark_minority_answers(rows)
    return rows


def write_rows(rows: list[dict[str, object]], path: Path) -> None:
    fieldnames = [
        "run_id",
        "task_name",
        "path_type",
        "log_source",
        "model_name",
        "temperature",
        "final_answer",
        "evidence_mentioned",
        "calculation_correct",
        "policy_current",
        "rule_warning",
        "minority_answer",
        "path_summary",
    ]
    with path.open("w", encoding="utf-8", newline="") as file:
        writer = csv.DictWriter(file, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(rows)


def main() -> None:
    out_path = Path(os.environ.get("P6_10_3_OUT_PATH", str(DEFAULT_OUT_PATH)))
    write_rows(generate_rows(), out_path)
    print(f"wrote {out_path}")


if __name__ == "__main__":
    main()

If Ollama is installed and a local model is available, .venv/bin/python docs/assets/part-06/chapter-10/p6_10_3_generate_response_path_log.py can be run to create a new log with the same format. The numbers included in the manuscript are a snapshot obtained by running llama3.2:latest with a specific setting. If it is run again, conclusion distributions and check-signal counts can change, and that difference itself shows why self-consistency and log observation are needed.

• Response path log: p6-10-3-response-path-log.csv

One row is one response path. The core columns are task_name, path_type, log_source, model_name, temperature, final_answer, evidence_mentioned, calculation_correct, policy_current, rule_warning, and minority_answer. path_type distinguishes whether this is CoT-style single-path observation or a self-consistency-style repeated candidate. What should be checked here is not only conclusion majority vote, but whether missing evidence, calculation error, missing current policy, work-rule warning signals, and minority conclusions outside the majority remain together. In particular, path_summary is not model-internal reasoning itself. It is a path summary reduced to a reviewable level.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

# Read response-path logs and compare conclusion distribution
# together with check signals.
import csv
from pathlib import Path

log_path = Path("docs/assets/part-06/chapter-10/p6-10-3-response-path-log.csv")


def to_bool(value):
    return value.lower() == "true"


def read_rows(path):
    with path.open(encoding="utf-8", newline="") as file:
        rows = list(csv.DictReader(file))
    for row in rows:
        for column in [
            "evidence_mentioned",
            "calculation_correct",
            "policy_current",
            "rule_warning",
            "minority_answer",
        ]:
            row[column] = to_bool(row[column])
    return rows


def summarize_task(rows, task_name):
    group = [row for row in rows if row["task_name"] == task_name]
    answer_counts = {}
    for row in group:
        answer_counts[row["final_answer"]] = answer_counts.get(row["final_answer"], 0) + 1
    majority_answer, majority_count = max(answer_counts.items(), key=lambda item: item[1])
    return {
        "answer_counts": answer_counts,
        "majority_answer": majority_answer,
        "majority_ratio": round(majority_count / len(group), 2),
        "missing_evidence": sum(not row["evidence_mentioned"] for row in group),
        "calculation_error": sum(not row["calculation_correct"] for row in group),
        "stale_policy": sum(not row["policy_current"] for row in group),
        "rule_warning": sum(row["rule_warning"] for row in group),
        "minority_answer": sum(row["minority_answer"] for row in group),
    }


rows = read_rows(log_path)
tasks = sorted({row["task_name"] for row in rows})

print("[dataset]")
print("run_count =", len(rows))
print("task_count =", len(tasks))
print("log_sources =", sorted({row["log_source"] for row in rows}))
print("models =", sorted({row["model_name"] for row in rows}))
print("temperatures =", sorted({row["temperature"] for row in rows}))
print()

for task_name in tasks:
    print(f"[{task_name}]")
    summary = summarize_task(rows, task_name)
    for key, value in summary.items():
        print(key, "=", value)

The example output can be read as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

[dataset]
run_count = 40
task_count = 4
log_sources = ['ollama_generated']
models = ['llama3.2:latest']
temperatures = ['0.7']

[current_refund_policy]
answer_counts = {'check_current_policy': 7, 'refund_7_days': 2, 'refund_14_days': 1}
majority_answer = check_current_policy
majority_ratio = 0.7
missing_evidence = 1
calculation_error = 0
stale_policy = 3
rule_warning = 8
minority_answer = 3
[discount_total]
answer_counts = {'apply_discount': 10}
majority_answer = apply_discount
majority_ratio = 1.0
missing_evidence = 7
calculation_error = 6
stale_policy = 0
rule_warning = 10
minority_answer = 0
[mixed_refund_label]
answer_counts = {'error': 10}
majority_answer = error
majority_ratio = 1.0
missing_evidence = 0
calculation_error = 0
stale_policy = 0
rule_warning = 10
minority_answer = 0
[security_escalation]
answer_counts = {'escalate_security': 10}
majority_answer = escalate_security
majority_ratio = 1.0
missing_evidence = 5
calculation_error = 0
stale_policy = 0
rule_warning = 10
minority_answer = 0

In this result, mixed_refund_label, discount_total, and security_escalation all have a majority conclusion ratio of 1.0. But security_escalation still has 5 cases with missing evidence, so the fact that every conclusion is the same does not mean enough reviewable standards remain. discount_total also converges entirely to apply_discount, but many paths do not leave enough calculation evidence. current_refund_policy mostly converges to check_current_policy, but minority conclusions that chose old refund periods and missing current-policy signals still remain. Here, rule_warning separately shows whether the response retained a signal that should be reviewed under work rules, and minority_answer shows whether a conclusion different from the majority existed.

When the same log is shown as a chart, it becomes clearer that conclusion agreement and observed check signals are different axes. Even if the upper bar is high, if the lower check signals are also high, the answer should not be adopted only because it repeated often. The lower bars are not response counts, but the sum of several check columns. If one response has both missing evidence and a rule warning, both signals are added together, so the bar height should be read not as how many answers failed, but as how many signals remain for reviewers to revisit.

Values readers can directly change in this example are the log rows themselves and the check-signal standards. For example, if rule_warning is made stricter, only warnings that are important under real work rules can remain among response paths. If paths where policy_current is False are all excluded, we can also check how the self-consistency majority changes. Through this manipulation, we confirm that CoT and self-consistency are not answer-guaranteeing technologies. They are strategies that help observe and compare response paths better.

Now mark whether the first thing to check in each scene is CoT, self-consistency, or system structure rather than a prompt strategy. The key is first choosing why the output is unstable.

Explanation:

The point of this exercise is not to bundle CoT and self-consistency as stronger prompts. CoT makes one path more readable, while self-consistency compares several paths. But evidence and execution guarantees are still separate structural problems.

Checklist¶

• Can you explain CoT as a strategy that makes intermediate reasoning paths more visible?
• Can you explain self-consistency as a strategy that looks at agreement among several reasoning candidates?
• Can you distinguish the fact that an intermediate explanation is long or a conclusion repeats from guarantees of latest evidence, calculation verification, and tool execution?
• Are you ready to read automatic prompt optimization in P6-11.4 as a prompt experiment-loop strategy, not a response-path strategy?

Sources and References¶

• Jason Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, arXiv, 2022, accessed 2026-07-19.
• Xuezhi Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, arXiv, 2022, accessed 2026-07-19.
• Pranab Sahoo et al., A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications, arXiv, 2024, accessed 2026-07-19.