Create your first Eval
Turn a quality question into a review batch that produces comparable, traceable verdicts.
An Eval is a structured review of traces or spans. Use it when “this looks wrong” needs to become a repeatable judgment that people and AI evaluators can answer consistently.
At the end of this guide, reviewers will have a bounded set of production evidence, specific questions, and a clear verdict format.
The first Eval should answer one real product decision. For example: Can we trust refund answers to use the policy that was active on the request date? Everything in the Eval, from subject selection to verdict format, should support that decision.
Start with the decision
Write the decision the results should support before you create the form:
Can we trust refund answers to follow the policy in effect on the request date?Then turn that decision into questions that can be answered from the selected evidence.
| Weak question | Stronger question | Why it is stronger |
|---|---|---|
| Was this good? | Did the final answer apply the policy effective on the request date? | Names the standard and evidence |
| Was retrieval correct? | Did the retrieval span return at least one policy valid on the request date? | Defines the operation and success condition |
| Is the tone okay? | Does the answer explain the denial without blaming the customer? | Describes observable behavior |
Split correctness, tone, citation quality, and task completion when they require different evidence or expertise. One question should represent one judgment.
Choose the subject
| Evaluate | Use when | What one example contains |
|---|---|---|
| Trace-level | Quality depends on the end-to-end interaction | The first message in and the final answer out; one row per trace |
| Span-level | You are judging one model, tool, retrieval, or agent step | That step's own input and output; one row per span |
Choose the smallest subject that still contains everything required to answer the question. Evaluating a whole trace to judge a single retrieval result creates extra reviewer work; evaluating one span to judge end-to-end task completion hides necessary context.
Build the Eval
You need Project Member or Project Admin access to create an Eval. A Project Viewer can answer Evals assigned to them but cannot create one; ask a project admin for access.
Open evals and select create eval. The New eval wizard opens with four steps: Dataset, Evaluators, Form, and Review. If you left a setup unfinished, choose Resume draft or Start fresh.
Dataset, Type: choose what one example is. Select Trace-level or Span-level.
Dataset, Schedule: choose when it runs. Under How often?, choose Just once or a cadence: Daily, Weekly, Biweekly, or Monthly. Then choose Immediately or At a set time.
- Just once and Immediately reviews traces that already exist.
- Just once and At a set time collects matching traces from the week before that time, including ones that arrive in the meantime, and sends them to evaluators at that time.
- A cadence creates a schedule that sends a new batch every interval until you stop it. Starting Immediately also sends matching traces from the previous interval right away.
Dataset, Conditions: choose the population and sample size. Add conditions on fields such as Workflow, Tags, Metadata, Detections, User ID, Cost, or Latency. Span-level Evals also offer Span type and Span name.
For existing traces, set Duration (how far back to search) and How many to evaluate: All matching, or Evaluate the most recent with a count (100 by default). For traces that have not arrived yet, set the Sampling rate and, optionally, Cap total (one-time) or Cap per batch (recurring). Select Preview samples to open examples the conditions would select.
Evaluators: choose who answers. Type a teammate's name or any email address to add people, and attach saved AI evaluators. When people are assigned, set the deadline (for Evals that collect future traces, the review window after each send), Auto-reminders, and whether to Send notifications via Slack or Email. Auto-reminders currently go out for one-time Evals only. See Add human and AI reviewers for reviewers outside your organization.
Form: build the questions. Add questions yourself, start from Templates, or describe what you need in Ask AI to draft questions… and select Generate. Every evaluator answers every question by default; select a question's evaluator chip (it reads Everyone) to choose specific people or AI evaluators in Assigned evaluators. Select Preview to see the form as reviewers will.
Review: confirm and create. Confirm & send summarizes Eval details (a name and an AI-generated description you can edit), Scope, Form, Evaluators, and Settings. Select Create eval. When setup finishes, Neatlogs opens the new Eval.
The Eval form turns a general quality concern into questions reviewers can answer against the same evidence.
Choose a verdict format
The answer format determines what you can compare later.
| Answer type | Best for | Example |
|---|---|---|
| Multiple choice or Dropdown | Clear pass/fail or one category | Valid policy / Expired policy / Insufficient evidence |
| Linear scale or Rating | Ordered quality levels with defined anchors | 1 = contradicts policy; 3 = incomplete; 5 = fully supported |
| Checkboxes | Several independent failure modes may apply | Missing citation, wrong version, unsupported exception |
| Multiple choice grid or Checkbox grid | The same judgment across several rows | One row per policy clause |
| Short answer or Paragraph | Context and reasoning | Why the case is ambiguous |
New questions are Required by default; turn that off for optional reasoning. Prefer a structured verdict plus optional reasoning. Free text alone is difficult to aggregate and calibrate.
Write reviewer guidance
For every question, specify:
- the evidence the reviewer may use;
- what each answer means;
- how to handle missing information;
- whether partial success passes;
- one positive and one negative example when the boundary is subtle.
Use an explicit Insufficient evidence or Not applicable option when it is a legitimate outcome. Forcing uncertain cases into pass or fail corrupts the standard you later use to calibrate AI evaluators.
Test before dispatching
On the Conditions step, select Preview samples to open traces the conditions would select. On the Form step, select Preview to read the form as a reviewer will. Check the form against at least three cases:
- an obvious pass;
- an obvious failure;
- a boundary or missing-evidence case.
If two qualified reviewers could reasonably choose different answers, improve the question or rubric before scaling the review.
Know what gets created
- A one-time Eval produces one review batch, listed under Batches on the evals page.
- A recurring Eval produces a schedule, listed under Schedules, and a new batch on each cadence.
- An unfinished setup is kept under Drafts so you can resume it.
- Assigned reviewers find their work under Assigned to me.
- Reviewer answers remain attached to their subject and question. A batch's Responses tab shows one row per trace or span and one column per question.
Avoid common first-Eval mistakes
| Mistake | Better approach |
|---|---|
| Combining correctness, tone, and completeness in one question | Create one question per independent judgment |
| Selecting only known failures | Include passes, failures, and boundary cases |
| Evaluating a span with a trace-level question | Choose the smallest subject that contains all required evidence |
| Using free text as the only result | Add a structured verdict and optional reasoning |
| Assigning AI before establishing the standard | Start with qualified human examples |
| Forcing missing evidence into Fail | Add an explicit insufficient-evidence outcome |
| Comparing batches with different populations as one trend | Keep filters, sampling, and schedule visible |
Know when the Eval is ready
Before dispatch, confirm that:
- every question represents one decision;
- reviewers can answer from the allowed evidence;
- verdict boundaries and missing-data behavior are explicit;
- the selected population represents the decision;
- at least one pass, failure, and boundary case has been previewed;
- assignments and review window are realistic.
If two qualified reviewers still need to invent different rules, revise the Eval before scaling it.
Frequently asked questions
Can one Eval use both people and AI evaluators?
Yes. By default every evaluator answers every question, and you can narrow each question to specific people or AI evaluators. Sending the same question to both lets you compare their answers. The exception is a Code evaluator, where it is enabled: an Eval that uses one cannot also have people assigned.
Should the first Eval use an AI evaluator?
Usually begin with people on a small, representative sample. Once the standard and edge cases are clear, test an AI evaluator against those answers.
Can I change a dispatched batch?
Only partly. The Eval's creator can move a deadline with Change deadline or end the batch with Close batch (Close eval on a one-time Eval). The questions and population of a dispatched batch stay as sent, and submitted answers are final. To change the questions or conditions, create a new Eval rather than rewriting completed review history.
How large should the first sample be?
Large enough to include obvious passes, failures, and boundary cases, but small enough to review carefully. Coverage matters more than a round number.