Evals and evaluators

An Eval answers a question that telemetry alone cannot settle: did this behavior meet our standard?

For a refund assistant, a trace can show which policy was retrieved and what answer was returned. An Eval can judge whether that answer applied the policy correctly, explained the decision clearly, and avoided unsupported claims.

The value of an Eval is not the form itself. It is the repeatable connection between a defined population, relevant evidence, a precise question, a reviewer, and a comparable verdict.

In the product, select evals, then create eval. The Dataset steps (Type, Schedule, Conditions) set the subject and population, Evaluators chooses who answers, Form holds the questions and response formats, and Review confirms the setup. An Eval is human, AI, or mixed depending only on which evaluators you add.

Follow one Eval from question to result

Suppose customers report that a refund assistant sometimes cites policies that were not active on the date of the request.

1. Define the question: “Did the answer use the refund policy effective on the request date?”
2. Choose the subject: the complete trace, because the judgment needs the request, retrieved context, and final answer.
3. Choose the population: a representative sample of refund traces from the last two weeks.
4. Choose a verdict: Pass, Fail, or Cannot determine.
5. Ask policy specialists to review an initial set and explain difficult cases.
6. Configure an AI evaluator with the clarified standard and the evidence it may read.
7. Send some subjects to both people and AI to measure agreement.
8. Review disagreements before allowing the AI evaluator to handle a larger share.
9. Investigate failed results and watch the pass rate after changes.

This sequence turns a vague complaint into a measurable standard without pretending that the first rubric or model judgment is automatically correct.

Choose the right subject

The subject is the exact trace or span presented for judgment. Choose Trace-level (one trace, one row) or Span-level (one step's own input and output, one row) in the Type step.

Use a complete trace when the answer depends on the final outcome, several operations, or coordination across the run. Use a span when one operation contains all required evidence.

Choosing a subject that is too small hides relevant evidence. Choosing one that is too broad makes review slower and may distract the evaluator with unrelated details.

Choose a population that represents the decision

An Eval can use explicitly selected existing traces or spans, all evidence matching a set of filters (Evaluate all matching), a sample of that matching set (Sample a subset), or matching evidence collected in the future. The Schedule step decides which: Just once reviews existing evidence now or at a set time, while Daily, Weekly, Biweekly, or Monthly collects new matching evidence each period.

The population should match the decision you intend to make:

• use recent production failures to understand a known problem;
• sample normal traffic to estimate overall quality;
• filter by workflow or environment to compare like with like;
• include edge cases to test a proposed change;
• collect future runs to monitor a standard over time.

A high score on an easy or narrow sample does not establish production quality. Keep the selection, filters, time range, and sampling method visible when reading aggregate results.

Write questions that can be applied consistently

“Was this good?” leaves every reviewer to invent a different standard. A useful question identifies the behavior, the evidence, and the boundary between verdicts.

Weak:

Was the refund answer accurate?

Stronger:

Based on the request date and the policy text included in the trace, did the answer apply the policy that was active on that date?

When ambiguity is legitimate, represent it. Cannot determine is often safer than forcing Pass or Fail when the required policy or tool result is missing.

Keep one decision per question

Do not combine correctness, tone, completeness, and safety into one verdict. A failure would be difficult to interpret, and two reviewers could disagree while noticing different problems.

Separate questions make the results actionable:

• Did the answer apply the correct policy?
• Did it explain the decision in language the customer could understand?
• Did it avoid exposing sensitive account information?

Choose a verdict that supports the next decision

The response format determines what the result can tell you.

Reasoning is valuable for calibration and investigation, but it does not replace a structured verdict when you need comparable results.

Human evaluators establish the standard

People bring policy knowledge, customer context, and the ability to recognize when the rubric itself is incomplete. Use human review to:

• define acceptable and unacceptable examples;
• resolve policy ambiguity;
• review high-impact or unusual cases;
• explain why near-boundary cases differ;
• audit automated judgments over time.

Human review is not automatically consistent. Two specialists can interpret the same question differently. That disagreement is useful evidence that the rubric, available context, or verdict choices need work.

AI evaluators apply the standard at scale

AI evaluators are reusable definitions managed under AI evaluators on the evals page. Neatlogs supports two AI judge types, plus deterministic code where enabled. Evaluators run on the managed Neatlogs model by default, so you do not need your own provider key; see Configure an AI evaluator.

LLM judge

An LLM judge makes a single-shot judgment from the evidence supplied with the subject, with no tools. It returns the configured verdict and, when Include reasoning is on, an explanation. Use it when the trace or span already contains everything needed.

Agent judge

An Agent judge is a multi-step agent that can use tools to gather evidence. A new evaluator starts as an agent judge. On its Evidence step you grant it sources one at a time, such as Neatlogs data or connected tools; access is read-only and each source is opt-in. Use it when the judgment genuinely requires evidence outside the subject, such as a current policy source or related production runs.

More tool access is not automatically better. Every source should have a clear role in the rubric, and the evaluator should not receive broad access merely because it is available.

Code evaluator

Where enabled, a Code evaluator returns deterministic scores from Python or JavaScript. It is not available in every deployment, and an Eval that uses one must be AI-only. Use it for checks that need no judgment, such as format or arithmetic validation.

Calibrate before unattended review

Calibration tests whether the evaluator applies your standard to real evidence, including difficult cases.

Human examples
    ↓
Explicit rubric and verdicts
    ↓
AI evaluator tested on the same subjects
    ↓
Disagreements reviewed
    ↓
Rubric, evidence, or evaluator adjusted
    ↓
Larger-scale review with continued sampling

Send selected subjects to both human and AI reviewers by assigning the same questions to both in Evaluators. Neatlogs records each reviewer's answers; compare them yourself and inspect:

• overall agreement;
• agreement by question and verdict;
• false passes and false failures;
• performance on important edge cases;
• reasoning that relies on missing or irrelevant evidence;
• changes after model, prompt, or rubric updates.

Agreement alone is not proof of quality. Reviewers can agree because they share the same blind spot or because the sample is too easy.

Read results in context

Read individual verdicts before relying on an aggregate. Then compare results by question, subject type, evaluator, workflow, model, environment, and time range where those dimensions are available.

A falling pass rate can indicate a real regression, but it can also reflect a harder traffic mix, a changed sampling filter, a stricter rubric, or a different evaluator. Keep the evaluation setup attached to the result.

What happens when an Eval fails

A failed verdict is evidence of a standard not being met. It is not a root-cause analysis.

Start with the underlying trace or span. Search for related failures, check whether the rubric was applied correctly, and open an investigation when the behavior is actionable. Group recurring evidence into one incident instead of creating a separate ticket for every failed subject.

Failed Evals can also reveal a new repeatable condition. Once the failure mode is understood precisely, create a detection to track where it appears in future production evidence.

Frequently asked questions

Can one question go to both a person and an AI evaluator?

Yes. Reviewing the same subjects is the most direct way to measure agreement and calibrate the AI evaluator.

Can I evaluate existing and future runs?

Yes. Existing evidence can be selected explicitly or through filters and sampling. Future Evals collect matching evidence over a configured window or recurring schedule.

Is a thumbs-up vote an Eval?

No. Voting is lightweight feedback on one piece of evidence. An Eval defines a population, questions, reviewers, and structured results that can be compared.

Should every question use the same evaluator?

No. Assign each question to the reviewer with the required knowledge. One Eval can mix human reviewers and AI evaluators.

What if the evaluator cannot determine the answer?

Treat missing evidence as a valid outcome when the rubric allows it. Do not force a confident verdict from an incomplete trace or unavailable policy source.

Next step

Create your first Eval →