Create and test an AI evaluator
Configure a reusable LLM judge or agent judge with instructions, an output format, and only the evidence it needs.
An AI evaluator is useful only when its verdict is repeatable enough to support a decision. Build it from a standard people already understand, then test disagreement before assigning it unattended production work.
To start, open evals, select AI evaluators, then select New evaluator. You can also select Create new evaluator while choosing evaluators for an Eval. The wizard walks through Type, Instructions, Model & output, Evidence (agent judges only), Try it, and Review.
Creating, editing, testing, or deleting an evaluator requires Project Member or Project Admin access. A Project Viewer can see evaluators but not change them; ask a project admin for access.
Choose the judge type
- LLM judge: a single-shot judgment from the model, with no tools. It sees the subject and your instructions. Use this by default.
- Agent judge: a multi-step agent that uses tools to gather evidence before judging. Use it when the subject alone is insufficient.
A new evaluator starts as an Agent judge. Switch to LLM judge on the Type step when the trace or span already contains everything the verdict needs.
| Requirement | Better starting point |
|---|---|
| The trace or span contains every required fact | LLM judge |
| The verdict requires a current policy or connected source | Agent judge |
| The condition is exact and deterministic | Detection or code rule rather than an AI evaluator |
| The standard is not settled | Human review before either AI option |
Where it is enabled, the Type step also offers Code evaluator, which scores with deterministic Python or JavaScript. It is not available in every deployment, and an Eval that uses one must be AI-only: it cannot also have people assigned.
Write the rubric
The Instructions step is one free-text field, pre-filled with a scaffold you replace. State the desired behavior, decision criteria, edge cases, and what does not count as success. Write instructions as if briefing a careful human reviewer.
Define the verdict schema
On Model & output, choose what the evaluator returns:
- With Custom output fields off, it returns a simple verdict: an overall verdict and an overall score.
- With Custom output fields on, you define each field. Field types are Text (short), Text (long), Number / scale, Rating, Single choice, Multiple choice, and Dropdown. Select Edit as JSON to edit the fields directly.
- Include reasoning asks the evaluator to explain each verdict. It helps review and debugging but adds latency and cost.
Use structured fields for comparison and reserve free text for reasoning that cannot be represented cleanly.
When you assign the evaluator to questions on an Eval form, it answers those questions, using each question's answer type, instead of its own output fields. Write the rubric so it applies cleanly to the questions you plan to assign.
Choose available evidence
Grant only what the question needs. The Evidence step appears for agent judges only, and each source is opt-in:
| Source | What it allows |
|---|---|
| Neatlogs data | Read traces, spans, and detections from this project. On by default for a new agent judge. |
| Connected tools & MCP | Read-only actions from apps connected under Settings → Account → Tools & MCP |
| Custom MCP server | Your own MCP server, added on the Custom MCP tab of Settings → Account → Tools & MCP with Available for evaluator grants turned on. Only reviewed read-only endpoints work for evaluation today; other servers fail with an "is not supported for read-only evaluation" error. |
| HTTP fetch | Read-only GET requests to the hosts you list under Allowed hosts, such as api.example.com or *.example.com |
Under Advanced limits, Max tool calls sets a hard stop per run (1 to 50) and Max budget (USD) caps spend per run.
Connect tools when required
Connect apps, or add a custom MCP server on the Custom MCP tab, under Settings → Account → Tools & MCP, then grant them on the evaluator's Evidence step. A custom MCP server must have Available for evaluator grants turned on before an evaluator can use it. Connections there are per user: each teammate connects their own accounts, and tools run with that person's permissions. A connected tool is not automatically available to every evaluator. See Tools & MCP.
Choose the model
The model picker on Model & output offers Neatlogs (managed, marked Default) and, under Your providers, any provider keys added under Settings → Project → AI Providers. You do not need your own provider key to run an evaluator. See Model providers and keys.
For an LLM judge, the Decision model toggle sends trace input and output, evidence, and grading criteria to the external provider TypeSafe for extra processing beyond the selected model. New evaluators use your saved preference, defaulting to on if you have not set one. Check the toggle and turn it off if your data policy does not allow that processing.
Test against a real trace
Use the Try it step to run the current configuration before you save. Choose Select a sample to pick a recent trace or span from the project, or Custom data to type your own input and output. Include difficult positive and negative cases, not only the example used to write the rubric.
Test with real evidence and inspect the structured verdict and reasoning before assigning the evaluator to questions.
Inspect the verdict and reasoning
Check whether the verdict follows the schema, cites or uses the intended evidence, handles missing data honestly, and applies the rubric rather than substituting generic quality preferences. For an agent judge, the result also shows how many tool calls it made.
Assign the evaluator
On Review, give the evaluator a name and description, then select Create evaluator. Attach it to an Eval on the Evaluators step, then choose which questions it answers on the Form step.
A saved evaluator opens as a read-only detail page. To test it again, select Edit and return to Try it. Re-test after changing its instructions, model, output, or tool access.
Build a calibration set
Test at least these cases before unattended use:
| Case | What it tests |
|---|---|
| Obvious pass | The evaluator recognizes the intended behavior |
| Obvious failure | It catches the defined failure mode |
| Boundary case | Rating anchors and exceptions are precise |
| Missing evidence | It can abstain rather than invent context |
| Adversarial wording | Subject content cannot override the rubric |
| Human disagreement | The standard itself may need clarification |
Compare the evaluator with qualified human answers. Investigate disagreement by case; one aggregate agreement number can hide a systematic error on the failures you care about.
Diagnose evaluator failures
| Symptom | Likely cause | Better response |
|---|---|---|
| Verdict is plausible but ignores the rubric | Instructions or response contract are too weak | Rewrite the rubric around explicit criteria |
| Correct cases fail when evidence is missing | No abstention behavior is defined | Add a missing-evidence outcome |
| Reasoning uses irrelevant trace content | Evidence scope is too broad | Limit the subject or available fields |
| Tool-based verdicts vary unexpectedly | External context or tool path is unstable | Narrow tool access and inspect tool results |
| Aggregate agreement is high but critical failures pass | Calibration underrepresents the important slice | Add held-out high-risk cases |
| Subject text changes evaluator instructions | The instruction boundary is weak | Separate untrusted content and test adversarial cases |
Know when it is ready for assignment
The evaluator is ready for limited use when it:
- returns the configured schema reliably;
- applies the rubric on obvious and boundary cases;
- handles missing evidence honestly;
- has acceptable false-pass behavior on important failures;
- uses only the intended evidence and tools;
- has been compared with qualified human review;
- has a plan for continued sampling after assignment.
Frequently asked questions
Which model should I choose?
Start with the Neatlogs default, then compare quality, latency, and cost on representative evidence. Other models appear once a provider is configured under Settings → Project → AI Providers.
Can an evaluator write to connected systems?
No. Evaluator tool access is read-only: connected apps expose only read actions, and HTTP fetch sends GET requests. Still grant the narrowest evidence the question needs.
Does a playground verdict count as a review?
No. Try it runs test the evaluator without saving anything or submitting an Eval response.
When should I use an agent judge?
Only when a correct verdict requires evidence beyond the supplied subject or several retrieval steps.
What happens to results if I delete an evaluator?
Evals that already used it keep their past results.