Evals
Route traces to reviewers, human or AI, with a structured form, then aggregate the results into quality metrics.
Some quality questions can't be answered by a rule or a classifier: they need judgment. Evals let you route real traces to reviewers with a structured form, collect their judgments, and aggregate the results. It's how you turn "this output felt off" into something measurable.
An evaluation has the same shape no matter who's answering it:
- Name and describe the evaluation.
- Choose a trace source: score traces (or specific spans) already captured, or score new ones as they arrive over a collection window.
- Build the form: a set of questions spanning ratings, linear scales, multiple choice, checkboxes, dropdowns, grids, and free text. Start from a template, describe it in plain language and let Neatlogs generate it, or build one from scratch.
- Assign reviewers to each question. This is the part that's new: a question can be answered by a teammate, by an AI evaluator you've configured, or both. Assignment is per-question, not just per-evaluation, so a single form can mix a person's judgment on one question with an AI evaluator's judgment on another.
- Launch. Items are dispatched on your schedule; each reviewer, human or AI, only sees the questions assigned to them.
Results are aggregated the same way regardless of who answered: overall summaries, per-question breakdowns, and per-batch views for recurring evaluations, exportable as CSV or JSON.
The two kinds of reviewer
Human Evaluators
Send traces to your team with a custom form and collect their judgments.
AI Evaluators
Configure an LLM or agent judge with your own rubric to review traces automatically.
Evals complements the lightweight voting you do while debugging in Traces. Use voting for quick, ad-hoc signal as you browse; use Evals for structured, assigned review against a defined rubric.