P6-17.2 Division of Work Between Automatic Evaluation and Human Evaluation¶

Section ID: P6-17.2 Version: v2026.07.31

Record evaluation division of labor as automatic_check, human_review, repeatable_rule, context_judgment, escalation_case, and review_note. This distinction keeps automatic evaluation, which is strong at repeatable checks, and human evaluation, which is strong at contextual judgment, from looking like substitutes for each other.

Subtitle: How are repeated checks and contextual judgment divided between automatic evaluation and human evaluation?

Once LLM evaluation axes are set, the next standard is deciding not to handle every evaluation in the same way. Items that can be checked mechanically and repeatedly, such as format errors, are easy to send first to automatic evaluation. Items that need contextual judgment, such as interpretation, nuance, and actual helpfulness, still need human evaluation.

Automatic evaluation is fast and repeatable, while human evaluation is better at context and quality sense. In real operations, the two are often used together. The evaluated target here is not the internal LLM algorithm, but the actual output produced by the model, such as answers, summaries, and guidance messages.

This also connects back to why we covered harnesses and MCP earlier. Output candidates, inputs, evidence documents, and tool-call records must remain so automatic evaluation can repeatedly check the same conditions and humans can later reread the same run. In other words, evaluation division of work is an operational structure for treating LLM outputs as reviewable service records.

Division of work between automatic and human evaluation¶

The core questions are:

What is automatic evaluation strong at?
What still requires human evaluation?
Why do structures that mix both often appear?

The core is not is automatic evaluation better, or are people better. Even for the same output, some items are first marked for correction by an automatic grader, while other items move to a human review queue. Evaluation division of work moves away from one person reading the whole quality judgment end to end. It automates repeatable checks and leaves ambiguous judgments as review packets that people can read.

Evaluation item	Easier for automatic evaluation	What human evaluation must still leave
Format compliance	JSON format, required fields, length limit, source presence	Whether the reading order is actually natural even though the format is valid
Groundedness	Source link, citation location, banned unsupported-expression detection	Whether the cited sentence actually supports the answer's conclusion
Helpfulness	Required item presence, next-action wording	Whether the user can actually understand the next action
Safety	Banned expressions, risky keywords, policy-violation candidates	Whether contextual misunderstanding or overconfidence remains

So the central question is how should the same evaluation axes be divided between automatic graders and human review.

Separating repeated checks from contextual judgment¶

You should be able to explain the difference between automatic evaluation and human evaluation, distinguish repeatability from contextual judgment, explain why operations mix the two, and split evaluation results into candidates that need automatic fixing and candidates that need human review.

Rather than memorizing long definitions, it is safer to use the standard of why the same answer branches into stopped first by an automatic grader, sent to a human review queue, or left as a review packet.

First visible answer state	Following evaluation route	Why the route splits this way
Format error, missing required field, basic criterion failure	Automatic fix needed	This is a mechanical failure to fix before a person reads the full answer.
Format is valid, but interpretation, nuance, or risk judgment is ambiguous	Human review queue	Automatic checks alone cannot fully read misunderstanding risk and helpfulness.
Automatic criteria pass, but contextual judgment remains	Human review packet	The output candidate and review questions must be bundled so a person can reread it in context.

This table makes evaluation division easier to understand as a branching structure of where should this candidate stop and where should it go, not as a comparison of who is better.

What automatic evaluation is strong at¶

Automatic evaluation is usually strong at:

fast repetition
comparing many samples
detecting regressions
checking format
checking items with clear criteria

For example, it is good for checking:

whether JSON format is valid
whether required fields exist
whether retrieved document sources are attached
whether a certain score became worse than the previous version

Automatic evaluation is strong at repetition and comparison.

What human evaluation is strong at¶

Human evaluation is usually strong at:

understanding context
judging subtle quality differences
noticing awkward expressions
judging real work fit
judging social risk and misunderstanding potential

For example:

the answer is formally valid but actually creates misunderstanding
the explanation is fluent but does not fit the reader level
evidence exists, but the wording is too categorical

These are easy to miss with automatic evaluation alone.

Human evaluation is better at context and nuance.

Why automatic evaluation alone is not enough¶

Automatic evaluation is strong when criteria are clearly defined, but it is hard for it to replace all quality judgment.

For example, automatic evaluation can often check:

whether a specific keyword exists in a document
whether a format constraint was followed
whether a score crossed a threshold

But it has limits in judging:

whether the answer is actually trustworthy and usable
whether the wording can mislead the user
whether it fits the organization's real purpose

Why human evaluation alone is also not enough¶

If evaluation relies only on people, other problems appear.

It is slow.
It is costly.
Standards can differ across reviewers.
It is hard to repeat frequently.

Human evaluation has depth, but as operations scale, it is hard to sustain without automation.

Why both are used together¶

In practice, the two are usually mixed like this.

Automatic evaluation first catches large-scale regressions and format errors.
Human evaluation checks important samples and subtle quality issues.
Automatic criteria are reinforced again when needed.

Automatic evaluation and human evaluation are therefore more similar to a division of work than to replacements for each other.

In operational flow, this division can be read more briefly like this.

Evaluation route checked first	What it catches best	Where it goes next
Automatic grader based on a fixed evaluation set	Regressions, format errors, basic groundedness signals	Human review or automatic-fix-needed state
Human review	Ambiguous interpretation, nuance, actual helpfulness, high-risk samples	Fix, hold, or approve decision

Automatic evaluation is strong at repeated comparison, and human evaluation is strong at interpreting ambiguous candidates and high-risk samples. This is why practical external curricula often keep automatic evaluation sets and human review together.

Before and after deployment, the same questions can be run through an automatic evaluation set to find regressions and format errors first, while human evaluation checks whether those changes matter to real user experience. Automatic evaluation is a broad filter, and human evaluation interprets the meaning of the remaining candidates.

Simplified once more:

flowchart LR
  A["candidate outputs"]
  B["automatic gate"]
  C["review packet"]
  D["human context judgment"]

  A --> B
  B --> C
  C --> D

The key point in this figure is that automatic evaluation does not finish all judgment first. It reduces the candidates that need human review and helps decide final action faster.

Drawn very simply¶

flowchart TD
  A["system outputs"]
  B["automatic checks"]
  C["human review"]
  D["combined quality decision"]

  A --> B
  A --> C
  B --> D
  C --> D

The key point of this diagram is that actual quality judgment does not end through only one path.

Another important point is that automatic and human evaluation results become inputs to operational judgment. The end of evaluation division is not simply quality judgment complete; it is more similar to a state where candidates have been sorted into continue or stop paths.

Judgment left by evaluation division	Question read again in operations
Whether the automatic grader passed	Can this candidate move to full human reading?
Whether human review is needed	Which path should it move to: fix, hold, or approval?
Human review packet	What questions should a person use when rereading it?
Failure or review reason	Which cause should failure handling trace first?

Automatic and human evaluation are not procedures that simply say good or bad. They are baselines for splitting candidates into operational routes.

Cases and examples¶

The focus of these cases is not what is correct, but which failures are filtered automatically and which failures must be read by people to the end.

Case 1. Evaluating RAG answers¶

Suppose a RAG answer includes both a source link and a cited span. If links and format are attached well, it is easy to feel first that there is evidence. Automatic checks can quickly confirm is there a source and is the format valid, but a link does not mean the answer actually reflects the document's meaning. For example, the document may contain an exception clause while the answer reads only one main sentence and states always possible.

In that case, automatic checks may pass while the actual user guidance is wrong. This interpretation error appears only when a person reads the document and answer together. Automatic evaluation checks is evidence attached, while human evaluation checks was the evidence read correctly. The criterion changes from is there a source link to does the source actually support the interpretation.

This matters in real operations because passing automatic checks can create too much comfort. If links and citation formatting are present, many teams first feel that the answer is grounded. But what automatic checks see well and what people must read to the end are different. Automatic checks filter surface structure quickly, while exception interpretation and contextual overstatement require reading the source and answer together.

Answer state	What automatic evaluation can see first	What human evaluation must read to the end
Link and citation format both exist	Source presence, format pass	Is there no categorical claim that omits an exception clause?
Cited paragraph exists	Paragraph reference is not empty	Does the paragraph actually support the answer condition?
Answer is natural and short	Length and format are fine	Did shortening remove a key limitation?

The misunderstanding this table corrects is expecting that if a source link exists, the interpretation is also mostly correct.

Case 2. Evaluating AI agent execution records¶

Even if an AI agent produced a final answer, the execution record may contain signals such as repeated search, failed tool calls, or repeated reads of the same document. Automatic evaluation can quickly mark whether these signals exist. But whether the signal is actually a problem or a necessary confirmation step requires a person to read the record together.

For example, if similar searches are repeated several times even after enough documents were found in the first search, automatic evaluation can leave a repeated call signal. But whether that repetition was an unnecessary detour or a needed recheck of ambiguous evidence depends on the execution flow. The criterion changes from did a final answer appear to does the automatic signal become a human-review question.

The result to check in this case is not cost calculation or operation optimization itself. That judgment is handled more directly in P6-17's operational constraints. Here, we only need to hold the division of work: automatic evaluation first gathers abnormal signals from the execution record, and human evaluation rereads those signals in context.

Execution record state	Signal automatic evaluation catches well	Question human evaluation should ask next
Similar searches are repeated	Repeated-call signal	Was it necessary evidence checking or an unnecessary detour?
Failed tool call remains	Failure signal	Did it repeat the same failure or recover through another path?
Final answer exists	Success flag	Did success hide intermediate risk signals?

The important standard is not wrapping up evaluation with only success/failure. Automatic evaluation gathers abnormal signals, and human evaluation interprets whether those signals are actual quality problems.

Case 3. Customer-support answer¶

Suppose a customer-support answer is fast, has no banned terms, and follows format. It is easy to feel that if automatic evaluation passed everything, it is almost enough. But if the actual sentence sounds too cold or shifts responsibility to the customer, service quality can still be poor. Conversely, even if the sentence is polite, the customer can still be confused if the action order is awkward.

For example, if the answer explains the document-submission route for a long time before saying whether a refund is available, the facts may be right but the customer may still not understand the answer. This does not show up well in format checks; people need to read it. In customer-support scenes, automatic evaluation is a basic guardrail, while human evaluation checks misunderstanding risk and tone quality.

Situation	Good target for automatic evaluation first	What human evaluation must read to the end
RAG answer evaluation	Source presence, format match, citation notation	Was the exception clause interpreted correctly?
Agent execution record	Repeated-call signal, failure signal, success flag	Is the signal a real quality issue or a necessary contextual check?
Customer-support answer	Banned terms, format, length, response speed	Tone, misunderstanding risk, next-action clarity

Scenes where automatic and human evaluation should split¶

A common misunderstanding is leaning to one side: automatic is better because it is faster, or humans are more accurate, so automation matters less. In real operations, the core is separating what should be filtered automatically first from what must be left to people to the end.

If you suspect this	First question to ask
`Does a person really need to see this?`	Can it be repeatedly checked by an automatic grader, like format, length, or source presence?
`Automatic checks passed, so why is it still unsettling?`	Did a person reread nuance, misunderstanding risk, and actual helpfulness?
`Why do some answers get fixed immediately while others go to people?`	Is the boundary between automatic-fix-needed and human-review-queue defined?

The standard to learn first is simple. Automatic evaluation is stronger at quickly filtering repeatable surface criteria; human evaluation is stronger at judging ambiguous interpretation and actual helpfulness to the end. In operation, they should be read as division of work, not competition.

Practice and example¶

The goal of the example is to see that automatic evaluation and human evaluation have different roles through different check items. Instead of looking at one answer, we compare several LLM output candidates and ask what does automatic evaluation stop first through repeated checks and what moves to a review packet for a person to read.

The example uses the English evaluation-routing candidate CSV p6_16_2_eval_routing_cases_en.csv. One row is one LLM output candidate that can appear in operation. model_output is the candidate answer, and source_marker, required_action, format_marker, max_length, and banned_terms are criteria the automatic grader checks repeatedly. The CSV does not contain prewritten human risk labels or answer labels.

The automatic grader names continue from P6-17.1's evaluation axes. source_marker_grader maps to groundedness, required_action_grader maps to helpfulness, format_grader and length_grader map to format compliance, and banned_terms_grader maps to safety. This mapping also appears in the code's GRADER_AXIS_MAP.

The output shows code-grader results for each candidate, optional LLM-as-a-judge results, human review packets, and a routing summary. The key point in the code is that it does not let code replace human evaluation. The code first checks repeatable surface criteria and then creates candidate sentences plus review questions for candidates that pass.

The local LLM judge is attached only to sample candidates when Ollama is running. Ollama is not used here to teach a specific product workflow; it is only a local way to show that an LLM can also be attached as one type of automatic grader. The prompt passed to the LLM judge stays in English and includes code-grader signals such as required action exists and banned expression absent. This keeps the LLM judge from looking like a replacement for human evaluation. It should be read as a supporting automatic grader that refers to repeatable check results.

The operational judgment criteria to read together are:

Check item	Why it is needed
Automatic grader pass status	To see whether basic guardrails such as format, length, evidence hint, and banned terms pass first
Automatic grader fix note	To separate mechanical fixes needed before human review
Human review packet	To let people read evidence interpretation, helpfulness, tone, and omissions even after automatic pass
Automatic-fix-needed status	To separate mechanical failures before people read the answer
Routing summary	To clearly record how automatic-fix-needed and human-review queues split

from __future__ import annotations

import csv
import json
import os
import re
import urllib.error
import urllib.request
from collections import Counter
from pathlib import Path
from pprint import pprint


OUT_DIR = Path(__file__).resolve().parent
CSV_PATH = OUT_DIR / "p6_16_2_eval_routing_cases_en.csv"
OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://localhost:11434/api/chat")
OLLAMA_TAGS_URL = OLLAMA_URL.replace("/api/chat", "/api/tags")
OLLAMA_MODEL = os.environ.get("AIBOOK_OLLAMA_MODEL", "llama3.2:latest")
USE_LLM_GRADER = os.environ.get("AIBOOK_USE_LLM_GRADER", "1") != "0"
OLLAMA_TIMEOUT_SECONDS = float(os.environ.get("AIBOOK_OLLAMA_TIMEOUT", "20"))

HUMAN_REVIEW_RUBRIC = [
    "Does the source signal actually support the answer's conclusion?",
    "Can the user immediately understand the next action?",
    "Is the tone or overconfidence risky in context?",
    "Did the answer omit an important condition or exception while shortening?",
]

SAMPLE_CASE_IDS = {"case_001", "case_002", "case_007", "case_008"}
AUTO_GRADE_PASS_THRESHOLD = 1.0
GRADER_AXIS_MAP = {
    "source_marker_grader": "groundedness",
    "required_action_grader": "helpfulness",
    "format_grader": "format_compliance",
    "length_grader": "format_compliance",
    "banned_terms_grader": "safety",
}


def split_terms(value: str) -> list[str]:
    return [term.strip() for term in value.split("|") if term.strip()]


def grade_to_bool(value: bool) -> float:
    return 1.0 if value else 0.0


def run_automatic_graders(row: dict[str, str]) -> dict[str, float]:
    output = row["model_output"]
    banned_hits = [term for term in split_terms(row["banned_terms"]) if term in output]
    return {
        "source_marker_grader": grade_to_bool(row["source_marker"] in output),
        "required_action_grader": grade_to_bool(row["required_action"] in output),
        "format_grader": grade_to_bool(output.endswith(row["format_marker"])),
        "length_grader": grade_to_bool(len(output) <= int(row["max_length"])),
        "banned_terms_grader": grade_to_bool(not banned_hits),
    }


def automatic_grade(grades: dict[str, float]) -> float:
    return sum(grades.values()) / len(grades)


def grade_fix_note(grades: dict[str, float]) -> str:
    failed = [name for name, grade in grades.items() if grade < 1.0]
    if not failed:
        return "-"
    failed_with_axes = [f"{name}({GRADER_AXIS_MAP[name]})" for name in failed]
    return "Fix the low automatic-grader items first: " + ", ".join(failed_with_axes)


def ollama_is_available() -> bool:
    if not USE_LLM_GRADER:
        return False
    try:
        with urllib.request.urlopen(OLLAMA_TAGS_URL, timeout=0.4):
            pass
    except (OSError, urllib.error.URLError):
        return False
    return True


def build_llm_grader_prompt(row: dict[str, str], grades: dict[str, float]) -> str:
    return f"""You are an automatic grader for an LLM answer.

Return exactly one JSON object.
The JSON object must have two keys: score and reason.
The score must be one of these numbers: 1.0, 0.5, 0.0.
The reason must be one short English sentence.
The reason must mention a concrete phrase in the candidate answer or a concrete missing signal.
Do not copy placeholder words from this instruction.

Use one of these scores:
- 1.0: the answer is grounded in the required source signal, gives the required next action, and is useful without overclaiming.
- 0.5: the answer is partly useful, but an important nuance, source connection, or next action is weak.
- 0.0: the answer misses a critical action, is unsupported, includes a banned expression, blames the user, or sounds unsafe.

Deterministic code grader observations:
- source_marker_present: {grades["source_marker_grader"] == 1.0}
- required_action_present: {grades["required_action_grader"] == 1.0}
- format_ok: {grades["format_grader"] == 1.0}
- length_ok: {grades["length_grader"] == 1.0}
- banned_expression_absent: {grades["banned_terms_grader"] == 1.0}

If required_action_present is false or banned_expression_absent is false, do not give 1.0.
If banned_expression_absent is false, the reason must name the banned expression from the banned expressions list.
If required_action_present is false, the reason must say that the required next action signal is missing.

Candidate answer:
{row["model_output"]}

Required source signal: {row["source_marker"]}
Required next action signal: {row["required_action"]}
Banned expressions: {row["banned_terms"]}
"""


def parse_llm_grader_json(raw_text: str) -> dict[str, object]:
    match = re.search(r"\{.*\}", raw_text, flags=re.DOTALL)
    if not match:
        return {"score": 0.0, "reason": "LLM grader did not return JSON."}
    try:
        data = json.loads(match.group(0))
    except json.JSONDecodeError:
        return {"score": 0.0, "reason": "LLM grader returned invalid JSON."}

    score = float(data.get("score", 0.0))
    score = min(1.0, max(0.0, score))
    reason = str(data.get("reason", "")).strip()
    return {"score": score, "reason": reason[:180] if reason else "No reason returned."}


def fallback_reason_from_code_graders(row: dict[str, str], grades: dict[str, float]) -> str:
    if grades["required_action_grader"] < 1.0:
        return f"Required next action signal '{row['required_action']}' is missing."
    if grades["banned_terms_grader"] < 1.0:
        banned_hits = [
            term for term in split_terms(row["banned_terms"]) if term in row["model_output"]
        ]
        return f"Banned expression appears in the answer: {', '.join(banned_hits)}."
    if grades["source_marker_grader"] < 1.0:
        return f"Required source signal '{row['source_marker']}' is missing."
    return "The LLM judge did not return a reason, so code-grader observations remain the primary record."


def run_llm_grader(row: dict[str, str], ollama_available: bool) -> dict[str, object]:
    grades = run_automatic_graders(row)
    if not ollama_available:
        return {
            "available": False,
            "model": OLLAMA_MODEL,
            "score": None,
            "reason": "Local LLM judge is disabled or not running, so this optional grader was skipped.",
        }

    payload = {
        "model": OLLAMA_MODEL,
        "stream": False,
        "format": "json",
        "messages": [
            {"role": "system", "content": "You are a strict automatic grader. Return only JSON."},
            {"role": "user", "content": build_llm_grader_prompt(row, grades)},
        ],
        "options": {"temperature": 0},
    }
    try:
        request = urllib.request.Request(
            OLLAMA_URL,
            data=json.dumps(payload).encode("utf-8"),
            headers={"Content-Type": "application/json"},
        )
        with urllib.request.urlopen(request, timeout=OLLAMA_TIMEOUT_SECONDS) as response:
            raw_text = json.loads(response.read().decode("utf-8"))["message"]["content"]
    except (OSError, urllib.error.URLError, KeyError, json.JSONDecodeError) as error:
        return {
            "available": False,
            "model": OLLAMA_MODEL,
            "score": None,
            "reason": f"LLM grader call failed: {error}",
        }

    parsed = parse_llm_grader_json(raw_text)
    reason_source = "llm"
    reason = str(parsed["reason"])
    if reason == "No reason returned.":
        reason = fallback_reason_from_code_graders(row, grades)
        reason_source = "code_grader_fallback"
    return {
        "available": True,
        "model": OLLAMA_MODEL,
        "score": parsed["score"],
        "reason": reason,
        "reason_source": reason_source,
    }


def llm_grader_alignment_note(grades: dict[str, float], llm_result: dict[str, object]) -> str:
    if not llm_result["available"]:
        return "The LLM judge was not used in this run."
    failed_code_graders = [name for name, grade in grades.items() if grade < 1.0]
    llm_score = llm_result["score"]
    if llm_score == 1.0 and failed_code_graders:
        return "The LLM judge missed code-grader failure signals. Recheck routing with code graders and human review packets."
    if llm_score != 1.0 and not failed_code_graders:
        return "Code graders passed, but the LLM judge left quality concern. Human review should reread evidence and usefulness."
    return "The LLM judge and code graders mostly pointed in the same direction."


def first_human_review_focus() -> str:
    return "The automatic graders passed. A person now reads context, usefulness, and tone to the end."


def human_review_packet(row: dict[str, str], grades: dict[str, float]) -> dict[str, object]:
    return {
        "candidate": row["model_output"],
        "automatic_grade": round(automatic_grade(grades), 2),
        "first_focus": first_human_review_focus(),
        "rubric": HUMAN_REVIEW_RUBRIC,
    }


def route_case(row: dict[str, str]) -> dict[str, object]:
    grades = run_automatic_graders(row)
    score = automatic_grade(grades)
    auto_pass = score >= AUTO_GRADE_PASS_THRESHOLD
    route = "human_review_queue" if auto_pass else "fix_with_automatic_grader_first"

    return {
        "case_id": row["case_id"],
        "domain": row["domain"],
        "source_row": row,
        "automatic_grade": round(score, 2),
        "auto_pass": auto_pass,
        "grader_scores": grades,
        "grader_fix_note": grade_fix_note(grades),
        "human_review_packet": human_review_packet(row, grades) if auto_pass else {},
        "route": route,
    }


def load_reports() -> list[dict[str, object]]:
    with CSV_PATH.open(newline="", encoding="utf-8") as csv_file:
        rows = list(csv.DictReader(csv_file))
    return [route_case(row) for row in rows]


def summarize_reports(reports: list[dict[str, object]]) -> dict[str, object]:
    route_counts = Counter(str(report["route"]) for report in reports)
    auto_pass_count = sum(bool(report["auto_pass"]) for report in reports)
    return {
        "case_count": len(reports),
        "auto_pass_count": auto_pass_count,
        "auto_fail_count": len(reports) - auto_pass_count,
        "automatic_fix_first_count": route_counts["fix_with_automatic_grader_first"],
        "human_review_queue_count": route_counts["human_review_queue"],
        "route_count": dict(sorted(route_counts.items())),
    }


def main() -> None:
    reports = load_reports()
    ollama_available = ollama_is_available()
    print("[summary]")
    pprint(summarize_reports(reports))
    print("[llm_grader]")
    pprint(
        {
            "enabled": USE_LLM_GRADER,
            "available": ollama_available,
            "model": OLLAMA_MODEL,
            "note": "Only sample cases call the optional local LLM judge to keep the example short.",
        }
    )
    print()

    sample_reports = [
        report for report in reports if str(report["case_id"]) in SAMPLE_CASE_IDS
    ]
    for report in sample_reports:
        print("=" * 80)
        print(report["case_id"], "/", report["domain"])
        print("automatic_grade =", report["automatic_grade"])
        print("auto_pass =", report["auto_pass"])
        print("grader_scores =")
        pprint(report["grader_scores"])
        print("grader_fix_note =", report["grader_fix_note"])
        print("llm_grader =")
        llm_result = run_llm_grader(report["source_row"], ollama_available)
        pprint(llm_result)
        print(
            "llm_grader_alignment =",
            llm_grader_alignment_note(report["grader_scores"], llm_result),
        )
        print("human_review_packet =")
        pprint(report["human_review_packet"])
        print("route =", report["route"])


if __name__ == "__main__":
    main()

The example output can be read like this when the optional local LLM judge is disabled or unavailable.

[summary]
{'auto_fail_count': 20,
 'auto_pass_count': 16,
 'automatic_fix_first_count': 20,
 'case_count': 36,
 'human_review_queue_count': 16,
 'route_count': {'fix_with_automatic_grader_first': 20,
                 'human_review_queue': 16}}
[llm_grader]
{'available': False,
 'enabled': False,
 'model': 'llama3.2:latest',
 'note': 'Only sample cases call the optional local LLM judge to keep the '
         'example short.'}

================================================================================
case_001 / refund
automatic_grade = 1.0
auto_pass = True
grader_scores =
{'banned_terms_grader': 1.0,
 'format_grader': 1.0,
 'length_grader': 1.0,
 'required_action_grader': 1.0,
 'source_marker_grader': 1.0}
grader_fix_note = -
llm_grader =
{'available': False,
 'model': 'llama3.2:latest',
 'reason': 'Local LLM judge is disabled or not running, so this optional '
           'grader was skipped.',
 'score': None}
llm_grader_alignment = The LLM judge was not used in this run.
human_review_packet =
{'automatic_grade': 1.0,
 'candidate': 'The refund request handling time is 14 days. Based on the '
              'notice please send the order number so we can help with intake.',
 'first_focus': 'The automatic graders passed. A person now reads context, '
                'usefulness, and tone to the end.',
 'rubric': ["Does the source signal actually support the answer's conclusion?",
            'Can the user immediately understand the next action?',
            'Is the tone or overconfidence risky in context?',
            'Did the answer omit an important condition or exception while '
            'shortening?']}
route = human_review_queue

The first thing to notice is that the code does not imitate human judgment. A candidate such as case_002, which is missing the required next action, and a candidate such as case_007, which includes a banned expression, stop at the automatic grader before human review. By contrast, a candidate such as case_001, which passes repeated check criteria, is not approved immediately. It moves to the human review queue with a packet that a person can read.

Another value to watch is reason_source when the optional LLM judge is available. If reason_source is llm, the local LLM judge produced the reason directly. If it is code_grader_fallback, the LLM judge's reason was empty or weak, so code-grader observations reinforced the reason. This distinction matters because the LLM judge itself is also something to review. Automatic grading does not end with one LLM judgment; it records code-check signals and LLM-judge signals together and leaves room for conflict.

automatic and human evaluation routing

This chart shows that the first row splits candidates into automatic-grader pass and fail, while the second row sends failed candidates to automatic fix and passed candidates to the human review queue. An automatic pass is therefore not an approval result. It is an intermediate state organized so a person can read further.

The same result can be grouped briefly by operational route.

Candidate	First visible state	Why it goes this route	Follow-up action
`case_001`	Human review queue	It passed automatic graders, but evidence interpretation, usefulness, and tone still need human reading.	Bundle into a review packet and send to a person
`case_002`	Automatic fix needed	It has the notice signal, but the next action, order number request, is missing.	Fix the output candidate and rerun automatic grading
`case_007`	Automatic fix needed	It has required signals, but includes a banned expression such as `always possible`.	Revise the categorical expression and rerun automatic grading

The result to check in this example is that automatic evaluation quickly checks repeatable conditions such as format, length, evidence hint, and banned terms, while human evaluation separately checks actual helpfulness, misunderstanding risk, tone quality, and interpretation of policy exceptions. In operation, automatic graders do not replace the final human judgment. They make the set of candidates humans read smaller and more reproducible.

Readers can try these adjustments in the example.

Remove or add next-action guidance in the CSV's model_output and see how required_action_grader changes.
Change banned_terms or candidate wording and see how banned_terms_grader changes.
Change AIBOOK_OLLAMA_MODEL and see how the optional LLM judge's score, reason, and reason_source change.
Adjust max_length and see how the length limit works at the automatic-grader stage before human review.
Change HUMAN_REVIEW_RUBRIC and see how the review packet attached to automatically passed candidates changes.

Human review remaining after automatic grading¶

The previous example is not a complete implementation that imitates human evaluation in code. It shows that the same output becomes a different work state after passing through automatic graders and human review packets. The core point is that increasing automatic checks and reducing human review are not the same goal. They are a division of work for catching different failures.

In one line, automatic evaluation and human evaluation are not a relationship of which is superior; they are an operational division of work that separates which candidates should be mechanically fixed first and which candidates should be read in context by people.

The more important point is that filter many cases quickly and check important errors to the end cannot be solved through the same review path. Automatic and human evaluation should therefore be read as an operational division that decides where a candidate stops and how far it moves up.

This division matters because it:

extends P6-17.1's evaluation axes from what to check to how to check
turns evaluation from a score table into a reviewable operational process
connects the same output candidates to records and failure handling that can be reviewed repeatedly
sets criteria for later verification procedures

Checklist¶

You should be able to explain that automatic evaluation is strong at repetition and comparison, while human evaluation is strong at context and subtle quality judgment.
You should be able to say that operations can shake with automatic evaluation alone or human evaluation alone, so the two should be used together.
You should know that quality judgment does not stop here, but continues into how candidates are left as records and review routes.

Sources and Further Reading¶

OpenAI, Getting started with datasets, OpenAI API Docs, accessed 2026-07-24.
OpenAI, Graders, OpenAI API Docs, accessed 2026-07-24.
OpenAI, Evaluate agent workflows, OpenAI API Docs, accessed 2026-07-24.
Yupeng Chang et al., A Survey on Evaluation of Large Language Models, arXiv, 2023, accessed 2026-07-24.