Review Evals as a human

Human review establishes how the written standard applies to real production evidence. The goal is not to answer quickly. It is to make a decision that another qualified reviewer could understand and reproduce.

Before opening the first item

Confirm:

• what decision the Eval supports;
• whether the subject is a trace or one span;
• which evidence the question allows;
• what each verdict means;
• how to handle missing evidence and exceptions;
• when reasoning is required.

If those rules are unclear, ask the Eval owner before reviewing a large batch.

Open your assigned work

Open evals and select Assigned to me, then open the Eval. The review page shows one item at a time, with your progress (submitted items out of the total, plus the due date) and a strip of items marked Current, Submitted, or Not answered yet. Move between items with the previous and next arrows or the Up and Down arrow keys.

If you were invited by email and have no Neatlogs account, open the link in the invitation instead. It goes to a review page for that Eval that needs no sign-in and expires after 7 days.

Review each subject in the same order

1. Read the question first

Reading the output first can cause you to invent a standard around the case. Start with the question, rubric, and verdict definitions.

2. Confirm the subject

Check whether you are judging the complete run or one operation. Do not penalize a retrieval span for a later model decision, or judge end-to-end success from one tool call.

3. Inspect only the required evidence

Start with the evidence presented with the subject. A trace-level item shows the conversation: the input, the intermediate steps (collapsed until you expand them), and the output. A span-level item shows that step's own input and output, labeled Arguments and Result for a tool call. When the rubric allows it and the answer requires more context, select Open trace to open the full trace in a new tab.

4. Choose the configured verdict

Apply the stated boundary. Do not change the meaning of a rating because the previous item was unusually good or bad.

Questions marked Required must be answered before you can submit. Your answers save as a draft automatically (Draft saved) until you submit.

5. Add reasoning when it adds value

Explain:

• policy-dependent decisions;
• boundary cases;
• missing or contradictory evidence;
• surprising failures;
• judgments likely to help rubric or AI calibration.

Routine obvious verdicts do not need an essay.

6. Submit

Select Submit & next to record the item and move on. On the last item the button reads Submit, and Submit final response? asks you to confirm. In the app, submitted answers are final.

Read the question before the output, then judge only the configured subject with the permitted evidence.

Handle ambiguity honestly

Do not force every case into Pass or Fail when the form provides a more honest outcome.

Keep the batch comparable

• Apply the same rating anchors throughout.
• Use only the evidence allowed by the rubric.
• Do not infer customer facts that are not present.
• Record recurring rubric defects instead of compensating differently on every item.
• Take a break when fatigue begins changing your threshold.
• Avoid reviewing only failures or only easy items when the batch is intended to represent production.

Respond to Slack and email notifications

Depending on the Eval's settings, you may be notified by email or by a Slack direct message when work is assigned or a deadline approaches. Choose which of these you receive under Settings → Account → Notifications (Evaluation assignments and Evaluation updates). Eval responses cannot be submitted from Slack, and a Slack reply does not record an answer. Follow the link to the review page and answer there.

Escalate a rubric problem

Pause and tell the owner when:

• a legitimate outcome has no verdict option;
• the question combines several independent judgments;
• required business context is absent;
• examples contradict the written rule;
• reviewers interpret a rating anchor differently.

Preserve completed responses as historical results. The owner should revise future review rather than silently changing what earlier verdicts meant.

Finish the batch

Before submitting the last item:

1. check the item strip for anything still marked Not answered yet;
2. review any Cannot determine or Not applicable cases;
3. flag rubric defects and repeated failure patterns;
4. confirm that each item shows as Submitted;
5. tell the owner if the sample appears systematically incomplete or biased.

Frequently asked questions

Can I change a submitted answer?

Not in the app. Until you submit, you can change answers freely; they are saved as a draft. Once submitted, its answers are final, and hovering the disabled submit button shows Already submitted. If a submitted answer is wrong, tell the Eval owner. Guest reviewers using an emailed link can resubmit while the link is valid.

Why can I see only some questions?

Questions are assigned per reviewer. You see only the questions the Eval's creator assigned to you.

Do I need to inspect every span?

No. Inspect the evidence needed to apply the rubric.

What if I disagree with another reviewer?

Compare the evidence and rubric. The disagreement may reveal a missing boundary or example rather than one careless reviewer.

What if the form itself is unclear?

Pause and tell the Eval owner. Do not normalize ambiguity through inconsistent answers.

Next step

Read and compare Eval results →