Create a detection
Define a known production condition, test it against real traces, and turn matches into a useful signal.
A detection answers: where did this known condition occur? It labels matching spans or traces so you can filter, trend, investigate, and optionally alert on them.
Detections are best for conditions you can describe before the next run arrives. They do not decide whether the overall behavior met a subjective quality standard. That is an Eval.
Define the condition first
Write examples before choosing an implementation:
Match: a tool call returns success=false or a 5xx status.
Do not match: the tool returns an empty but valid result.
Evidence: tool output and status attributes.If you cannot name the evidence that proves a match, investigate more cases before automating it.
For example, “bad refund answer” is not yet a detection condition. “The final output mentions policy v2 after its retirement date” is closer, but it still needs a reliable request date and policy version field or text pattern.
Choose a detection type
| Type | Level | Use it for | Example |
|---|---|---|---|
| Regex | Span | Deterministic text patterns in a span's input or output | Secret prefix, known error phrase, malformed identifier |
| Conditions | Span | Span fields such as status, name, latency, and tokens, combined with AND/OR logic | Error status and latency above a threshold |
| PII | Span | Selected personal-data types: Email Address, Phone Number, Credit Card, IP Address, URL | An email address in a model output |
| Classifier | Span, or trace when the classifier supports it | A built-in semantic classifier with a score | Jailbreak attempts or agent refusals |
| Decision Model | Span or trace | A plain-language criterion that a model checks, without training | The tool failed to complete the requested action |
| Trace Rules | Trace | Match, count, order, aggregate, and absence checks across the complete trace | Tool called more than five times in one trace |
| Code | Span or trace | A Python or JavaScript detect() function that returns true on a match | Business-specific validation |
Choose the simplest deterministic type that expresses the condition. Use Classifier or Decision Model only when the boundary is semantic, and Code only when built-in rules cannot represent it clearly.
Create and test the detection
Open Detections and start a new detection. Select detections in the sidebar, then select create detection. To describe the condition in plain English instead, select Copilot: it drafts the detection in a chat and shows a preview for you to review before anything is created.
Choose the level. Select Span-level when one operation contains the evidence. Select Trace-level when the rule depends on the whole execution or relationships across spans.
Choose the type. The cards depend on the level: span level offers Regex, Conditions, Classifier, Decision Model, PII, and Code; trace level offers Trace Rules, Decision Model, Code, and Classifier when a built-in classifier supports traces. Use the decision table above.
Set the scope. For span-level rules, choose All spans, Run on types (for example LLM, tool, or MCP spans), or Run on names (specific span names from your traces). Trace-level rules evaluate the complete trace and skip this step.
Configure and preview. Add patterns, conditions, PII types, a classifier, a criterion, trace rules, or code. Most types show a preview of recent spans or traces marked Match or No match while you configure them. PII has no preview step.
Review and create. Enter a Name that describes the condition, an optional Description (or select Autofill to suggest both), and choose a Detection group. Check the level, type, scope, and configuration, then select Create detection.
Create a detection from the Detections page, then validate it on evidence before relying on its matches.
Scope at the right level
Use span level for statements such as:
- this tool returned an error;
- this model output contains a secret pattern;
- this retrieval step exceeded the latency limit.
Use trace level for statements such as:
- the agent called the same tool repeatedly;
- the run completed without ever calling the required validator;
- total tool-call count exceeded the workflow limit.
A broad scope makes testing look successful because it finds many obvious matches. A useful scope finds the exact operational condition and excludes nearby normal behavior.
Validate with positives and negatives
Review more than the first matching row:
| Test set | What it reveals |
|---|---|
| Clear positive examples | Whether the rule catches the intended failure |
| Clear negative examples | Whether normal behavior is mislabeled |
| Near misses | Whether boundaries and thresholds are correct |
| Different workflows or environments | Whether the scope leaks beyond its intended population |
A match remains attached to the underlying trace evidence so it can be reviewed rather than trusted as an unexplained label.
Name and group it for operations
Prefer names that describe the matched condition, such as Refund policy version missing, over vague names such as Bad response. Groups should help readers interpret the signal consistently across the project.
After creating the detection, observe its baseline frequency before creating an alert. Traffic mix, a new workflow, or improved instrumentation can change volume without indicating a quality regression.
What happens after creation
A new detection is active as soon as you create it. Neatlogs evaluates each trace in the background after the trace completes, so matches appear shortly after a run finishes. The detection never blocks or modifies the running agent.
Creating or enabling a detection also starts a backfill over traces from the last 14 days. The detection page shows Backfilling historical data with its progress until it finishes. Changing the rule or scope of an existing detection clears its earlier matches and backfills again.
Matches appear on the affected spans or traces, on the detection's Flagged spans or Flagged traces tab, and in detection trends on the detections list and the detection's Overview tab.
To stop a detection, open its menu and select Pause (keeps past results but stops checking new traces) or Disable (hides past results and stops checking new traces). Use the Active, Paused, and Disabled filters on the list to find them again.
Creating, editing, pausing, or deleting a detection requires the Project Member or Project Admin role on the project. If you are a Project Viewer, ask a project admin or an organization owner or admin to make the change.
Coding agents connected to the Neatlogs MCP server can also preview and create detections.
Diagnose a weak detection
| Symptom | Likely cause | Better response |
|---|---|---|
| Too many normal runs match | Scope or condition is too broad | Add negative examples and narrow the rule |
| Known failures do not match | Required evidence is missing or the pattern is too strict | Inspect fields and near misses |
| Matches cluster in an unrelated workflow | Span names, types, or project scope are too broad | Restrict the eligible population |
| Classifier scores are difficult to interpret | The semantic boundary or threshold is unclear | Review scored examples and clarify the condition |
| Code rule is hard to maintain | Built-in rules may express the same condition | Prefer Conditions, Regex, or Trace Rules when possible |
| Frequency changes after instrumentation work | The measurement changed | Compare field coverage before calling it a regression |
Know when the detection is ready
Rely on it operationally when:
- the condition is stated in observable terms;
- the level and scope contain all required evidence;
- clear positives match;
- clear negatives do not match;
- near misses behave as intended;
- the name describes the condition rather than an assumed cause;
- someone knows how matches will be reviewed.
Observe its normal production rate before attaching a paging alert.
Frequently asked questions
Is PII detection the same as redaction?
No. Detection labels evidence that appears to contain selected PII. Redaction controls what is captured, stored, or displayed.
Can I change the level or type later?
No. Level and type are fixed once a detection exists, and Duplicate keeps them too. Create a new detection with the level and type you need, test it, then pause or disable the old one. You can edit the scope, configuration, name, description, and group of an existing detection with Edit.
Why does preview behavior differ by type?
Regex, Conditions, Trace Rules, and Code are deterministic, so a preview shows exactly which recent spans or traces match. Classifier and Decision Model results come from a model, so review several matches and non-matches rather than trusting one sample. PII has no preview in the wizard: review its first matches on the detection page after the backfill.
Should every detection create an alert?
No. Alert only when a match rate or threshold requires timely action. Many detections are more useful as filters and trends.