Add human and AI reviewers

Reviewer selection is part of the Eval design. A precise question can still produce weak results when it is assigned to someone without the required context or to an AI evaluator that has not been calibrated.

Assignments are made per question. One Eval can combine policy specialists, product teammates, engineers, LLM judges, and agent judges.

You add reviewers on the Evaluators step of the New eval wizard, then choose who answers each question on the Form step. Every evaluator answers every question by default; select a question's evaluator chip (it reads Everyone) to narrow it in Assigned evaluators. Reviewers see only the questions assigned to them.

Start from the knowledge required

For each question, write down:

1. which evidence contains the answer;
2. which expertise is required;
3. how often the question will be asked;
4. what happens if the verdict is wrong;
5. whether the standard is already stable.

When to use a person

Choose human review when:

• the policy or quality standard is still being defined;
• customer or business nuance changes the answer;
• the evidence is incomplete or contradictory;
• the case is rare, high impact, or safety-sensitive;
• you need examples and explanations for AI calibration;
• a wrong automated verdict would trigger consequential action.

People are not automatically consistent. Give them the same rubric, evidence boundary, and verdict choices. Use disagreement to improve the standard.

Add people inside or outside your organization

On the Evaluators step, type a name or email address. Suggestions come from your organization's members, and you can also enter any email address.

• Members of your organization review in the app. Being assigned is enough to answer that Eval, even for someone with only Project Viewer access or no access to the project.
• People without a Neatlogs account become guest reviewers. They receive an emailed link to a review page that needs no sign-in, covers only that Eval (for a recurring Eval, its current batch), and expires after 7 days. Guest reviewers can revise a submitted answer while the link is valid.

Creating an Eval requires Project Member or Project Admin access. After the Eval is created, only its creator can change its reviewers, deadlines, or status.

When to use an AI evaluator

Choose an AI evaluator when:

• the rubric is explicit and stable;
• the same judgment must be applied repeatedly;
• the required evidence can be supplied or retrieved safely;
• representative human answers exist for testing;
• errors can be sampled and corrected.

Start with an LLM judge when the subject already contains the answer. Use an Agent judge only when the verdict requires external context or several evidence-gathering steps.

Do not grant tools that the question does not require.

Use dual assignment for calibration

Assign the same question and subjects to both qualified people and the AI evaluator.

Shared subjects
    ├── Human verdicts
    └── AI verdicts
             ↓
       Compare by question,
       failure type, and slice

Review disagreements individually. The human may be wrong, the AI may be wrong, the rubric may be ambiguous, or the evidence may be insufficient.

Dual assignment is most useful on a representative calibration sample. It does not need to continue for every subject once the evaluator is stable, but periodic shared review should remain.

Assign the reviewer mix

A practical progression:

Different questions can be at different stages. One stable correctness question may use AI while a new policy-exception question remains human-only.

Set a realistic review window

Match the deadline to:

• number of assigned subjects and questions;
• complexity of the evidence;
• expected reasoning depth;
• reviewer availability;
• urgency of the decision.

Choose 1 day, 3 days, 1 week, 2 weeks, or a custom date. A one-time Eval has one deadline (Set the deadline to fill this eval). An Eval that collects future traces gets a review window that starts after each send (Set eval window). A rushed, incomplete review is not made reliable by a short SLA.

Auto-reminders (No reminders, Every day, Halfway through, One day before deadline, or Custom) currently go out for one-time Evals only. For recurring Evals, follow up with reviewers directly.

Choose the review surface

The web review queue is the canonical surface. Teammates open their work from Assigned to me on the evals page; guest reviewers use the page their email links to. Both show the assigned subject, question, evidence, and response controls.

Under Send notifications via, choose Email, Slack, or both. Slack notifications arrive as direct messages and are available only after Slack is connected under Settings → Organization → Integrations. Reviews are not answered inside Slack: the messages link back to Neatlogs. See Review from Slack.

Track completion without changing history

Use batch progress to distinguish dispatched, completed, and outstanding work. A batch shows as Pending (sent, no answers yet), In progress, Overdue (the deadline passed with answers missing), or Closed. Follow up with reviewers or adjust future scheduling rather than silently rewriting submitted responses.

The Eval's creator can move deadlines with Change deadline on a batch (or Edit deadlines on a recurring Eval's schedule page), and end review with Close batch or Close eval. A closed batch cannot be reopened yet.

When a reviewer repeatedly skips one question, check whether the evidence, rubric, or assignment is the real problem.

Frequently asked questions

Does every question need the same reviewer?

No. Assignment is per question.

Can several AI evaluators answer the same question?

Yes, when you want to compare them on shared subjects. Avoid duplicate unattended judgments that serve no decision.

Can a guest review without project membership?

Yes. Enter their email on the Evaluators step. Someone without a Neatlogs account gets an emailed review link scoped to that Eval, valid for 7 days. An ordinary project URL does not grant access.

What happens when a reviewer misses the deadline?

A one-time Eval closes when its deadline passes, keeping the answers that arrived. A recurring Eval's batch is marked Overdue and stays incomplete while the schedule continues. The Eval's creator can use Change deadline or adjust the next review window.

When should a question remain human-only?

Keep human control when the standard is unsettled, the impact is high, or AI behavior is not reliable on important slices.

Next step

Create and test an AI evaluator →