Read results and calibrate evaluators

An aggregate score is the beginning of analysis, not the conclusion.

Eval results depend on the population, questions, verdict definitions, reviewers, and evidence available. A pass rate can change because behavior improved, because traffic changed, or because the rubric and evaluator changed.

Read results from individual evidence outward

Open the batch from Batches on the evals page. A recurring Eval's schedule page lists its batches; filter them by In progress, Overdue, or Closed. A batch has two tabs:

• Responses shows one row per trace or span and one column per question, with a count of answers received. Each cell combines the evaluators assigned to that question. Open a row to see the subject and each evaluator's answer.
• Report holds an AI-written summary of what the batch's answers found. Select Generate report to create it; generation and report access require the Eval's creator, a project admin, or an organization owner or admin with access to this project.

1. Open representative subjects

Review:

• an obvious pass;
• an obvious failure;
• a boundary case;
• a missing-evidence case;
• any high-impact or surprising verdict.

Confirm that the response applies the rubric to the intended trace or span.

2. Read by question

Per-question results show which standard is moving. A single combined score can hide that correctness improved while completeness or tone declined.

Look for:

• one verdict dominating almost every case;
• unusually high abstention;
• reviewers using rating anchors differently;
• one failure category concentrated in a workflow or release.

3. Compare evaluators

Compare only on shared subjects. Account for incomplete responses and different assignments before interpreting rates.

4. Read trends over comparable batches

Recurring Eval batches are comparable only when the population, sampling, rubric, and evaluator behavior remain sufficiently stable. Record changes to any of them.

Neatlogs does not chart Eval results across batches. To follow a trend, open each batch from the schedule page or export each batch's responses and compare them side by side.

Understand agreement

Agreement measures whether reviewers return compatible verdicts on the same evidence.

Neatlogs does not calculate agreement for Evals. Export the batch responses (see the FAQ below); each exported row names the evaluator and whether it is a person or an AI evaluator, so you can compute these views on shared subjects.

High agreement is not automatically good. Two reviewers can share the same blind spot.

Investigate disagreement by case

For each important disagreement, ask:

1. Did both reviewers see the same evidence?
2. Is the question asking one decision?
3. Are verdict boundaries explicit?
4. Is an authoritative policy missing?
5. Did the AI use irrelevant content or ignore missing evidence?
6. Did the human apply an unstated exception?
7. Does the case belong to a slice absent from calibration?

Classify the cause before changing the evaluator.

Calibrate without overfitting

Use one set of cases to write and tune the evaluator, then test the revision on a held-out set containing normal traffic and difficult examples.

Re-run calibration after:

• changing the rubric or verdict schema;
• changing the AI model or evaluator prompt;
• adding or removing tools;
• changing workflow behavior;
• updating a relevant business policy;
• seeing a new production failure mode.

Decide when unattended review is justified

Require:

• stable performance on representative subjects;
• acceptable false-pass behavior on high-impact failures;
• honest handling of missing evidence;
• controlled tool and data access;
• repeatable results after changes;
• periodic human sampling;
• a route for exceptions and disputed cases.

Automation can differ by question. A stable formatting check may run unattended while a policy exception remains human-reviewed.

Turn results into action

A failed verdict is evidence, not a diagnosis.

Open the subject and reviewer reasoning. Search for related failures. When the pattern is actionable, investigate the cause. If the failure becomes a precise repeatable condition, create a detection for ongoing production tracking.

Frequently asked questions

Is aggregate agreement enough?

No. Check important questions, workflows, customer groups, languages, and failure categories separately.

Should old verdicts change after a rubric update?

Preserve historical results. Submitted answers cannot be edited, and a dispatched batch keeps the questions it was sent with. Run the revised rubric in a new Eval and compare it with the earlier batches.

Can results be exported?

Yes. On a batch's Responses tab, open Export and choose CSV or JSON. The file contains the current view, including any filters, with one row per answer: subject, evaluator, evaluator kind, question, value, and submission time. Export becomes available once the first response arrives. A generated Report can be copied as Markdown with Copy markdown or saved with Save as PDF. Where the batch list offers AI report, it opens the saved report viewer.

Report generation runs asynchronously; you can leave the page and return later. Use Regenerate to refresh an existing report. If the permission check cannot load, use its retry control; a temporary loading error is separate from a confirmed denial of access.

What if two human reviewers disagree?

Resolve the standard or add adjudication. AI agreement is not an automatic tiebreaker.

How often should calibration be repeated?

Repeat it whenever the evaluator, evidence, workflow, or business standard changes, plus periodic checks during stable operation.

Next step

Turn a failed Eval into an investigation →