AI Evaluators
Configure an LLM or agent judge with your own rubric and verdict schema to review traces automatically, alongside or instead of human reviewers.
An AI evaluator is a reusable judge you configure once (a rubric plus a verdict schema) and then assign to answer specific questions on an evaluation, the same way you'd assign a teammate. Every trace can be reviewed automatically instead of just the ones your team has time to open.
Two kinds of judge
When you create an evaluator, you choose how it reviews:
- LLM as judge: a single-shot judgment from the model. No tools, just the trace and your rubric. Fast, and the right default for most questions.
- Agent as judge: a multi-step agent that can use tools to gather evidence before judging. Use this when answering the question means inspecting span data more closely, calling a tool, or checking something external before deciding.
An agent judge can draw on:
- Neatlogs's own tools: reading traces, spans, and detections in your project. Available by default, no setup required.
- Connected integrations and MCP servers: whatever you've already connected under Settings → Tools & MCPs.
- Your own custom MCP server: point the judge at a self-hosted MCP server.
- Allowlisted HTTP fetches: read-only GET requests, restricted to hosts you explicitly list. A request to any host not on the allowlist is refused.
Not sure which to pick: start with an LLM judge. Move to an agent judge only for questions the model can't answer from the trace alone.
Rubric and verdict schema
An evaluator has two parts:
- The rubric: instructions telling the judge how to evaluate, written the same way you'd brief a human reviewer.
- The verdict schema: the fields the judge must fill in for every item it reviews: a label, a type (rating, multiple choice, yes/no, free text, and so on), and whether it's required. This is the same shape as questions on a human evaluator's form, so the two are interchangeable when you assign them.
Testing before you assign it
Before an evaluator reviews anything for real, try it in the playground: run it against a real trace from your project, or against your own typed input and output, and see the verdict it produces. You can do this while still drafting the evaluator, or later against an evaluator you've already saved. Playground runs never count as a real review.
Assigning it
AI evaluators are assigned the same way human reviewers are: per question, on the evaluation's form. A question can go to a person, to an AI evaluator, or to both. Mix and match per question rather than committing an entire evaluation to one reviewer type.
See Human Evaluators for the reviewer side of the same form.