Detections, incidents, and alerts

Understand how detections label repeated production behavior, incidents organize the response, and alerts notify the right people.

A production problem often appears more than once. Neatlogs separates recognizing the pattern, deciding whether it needs work, and notifying the right people.

Object	What it does	Example
Detection	Defines a condition Neatlogs can look for repeatedly	The same tool is called at least five times in one run
Detection match	Records that the condition occurred on a specific span or trace	Trace `abc123` called `lookup_order` seven times
Alert	Defines when matches or operational measurements should notify someone	Notify Slack when the condition appears in three traces within 15 minutes
Incident	Tracks one production problem through investigation and resolution	Repeated order lookups are increasing latency and cost

The distinction matters. A match is evidence, an alert is a response policy, and an incident is the problem the team follows up on. A match does not automatically become an alert or an incident.

Follow one example through the system

Suppose a support agent occasionally gets stuck calling lookup_order again and again.

You create a trace-level detection for five or more calls to the same tool in one completed trace.
The detection evaluates incoming traces. A run with seven calls becomes a detection match.
The match stays attached to that trace, where you can inspect the calls and compare it with normal runs.
Once you know the usual match rate, you create an alert for a burst of three matching traces within 15 minutes.
The alert fires when that threshold is crossed. It tells the team that the known pattern is happening often enough to need attention.
Because the pattern keeps recurring, Neatlogs's automatic incident analysis can group the matching runs into an incident on incidents.
A teammate reviews the evidence, starts an investigation, and moves the incident from Open to Investigating to Resolved as the work progresses.

Nothing in this flow changes the running agent. Detections observe completed evidence. Alerts notify. Incidents organize the response.

Completed trace
    ↓
Active detection evaluates it
    ↓
Match is attached to the trace or span
    ↓
Search and trend the matches
    ↓
Alert policy may notify the team
    ↓
Incident analysis may group the evidence into an incident

A detection is a rule, not a finding

A detection is the reusable definition. It states what evidence to inspect, where to inspect it, and what counts as a match.

A detection match is one result of applying that definition. It points back to the span or trace that satisfied the rule. Matches can be filtered, counted, reviewed, and used as evidence in later work.

This separation lets one definition answer questions over time:

Where has this happened before?
Is it becoming more common?
Which workflows, environments, models, or tools are affected?
Did it stop after a change?

A well-named detection describes the observed condition, not an assumed cause. Repeated order lookup calls is useful. Broken agent logic claims a cause that the match itself has not proved.

Choose the level that contains the evidence

Neatlogs evaluates detections at either span level or trace level.

Span-level detections

Use span level when one operation contains everything needed to decide. You can scope the rule to All spans, selected span types (Run on types), or specific span names (Run on names).

Examples include:

a tool call returned an error status;
a model output contains a known secret pattern;
a retrieval span exceeded a latency limit;
a particular tool received a malformed identifier.

The match points to the individual operation that satisfied the rule.

Trace-level detections

Use trace level when the condition depends on the completed run, several operations, or the relationship between them.

Examples include:

the same tool was called repeatedly;
a required validation step never ran;
one operation occurred before or after another;
the total number, duration, or token count of selected spans crossed a limit;
the final trace status or output matched a condition.

The match points to the run as a whole. You can then open its spans to inspect the supporting sequence.

Note

Choose the smallest level that still contains all the evidence. If one tool result proves the condition, use a span-level rule. If you need counts, absence, order, or the final outcome, use a trace-level rule.

Choose a detection type

The type controls how the condition is expressed. Start with the simplest type that represents the behavior precisely.

Type	Best for	Example	Avoid it when
Conditions	Comparisons on structured span fields	Status is `ERROR` and duration is above 5 seconds	The boundary depends on meaning rather than a field value
Regex	Stable text patterns in span input or output	Output contains a known secret prefix	Wording varies enough that a pattern becomes fragile
PII	Supported categories of sensitive information	A model response contains an email address or phone number	You need to prevent capture or redact stored data
Classifier	Semantic categories scored by a built-in classifier	A response expresses refusal or a jailbreak attempt	The condition can be stated exactly with fields or text patterns
Decision Model	A plain-language criterion that a model checks, without training	The tool failed to complete the requested action	A deterministic type can express the condition exactly
Trace Rules	Counts, repeats, absence, order, and aggregates across a run	A payment tool ran before identity verification	One span contains the complete answer
Code	Business-specific logic in a Python or JavaScript `detect()` function	An output total differs from the sum of returned line items	A built-in rule can express the same condition more clearly

Regex, Conditions, and PII run at span level. Trace Rules run on the complete trace. Decision Model and Code can target either level, and Classifier can run at trace level when a built-in classifier supports traces. Session-level detections are not available yet.

Detection types can produce different kinds of evidence. A deterministic rule may provide a direct boolean match, while a classifier can also carry a score. Treat the score as part of the signal, not as proof that the underlying conclusion is correct.

When detections run

A detection is active as soon as you create it. Neatlogs evaluates every trace in the background after the trace completes, including imported traces, so matches appear shortly after a run finishes. Creating or enabling a detection also backfills matches over the last 14 days of traces.

To stop a detection, Pause keeps its past results but stops checking new traces, and Disable hides past results as well. See Create a detection for the full lifecycle.

Review matches before assigning meaning

A match means the configured condition was satisfied. It does not tell you by itself whether the behavior is harmful, why it happened, or what should change.

Review a useful sample of:

clear matches;
normal runs that should not match;
near misses around a threshold;
different workflows and environments;
changes in match frequency after deployments or traffic shifts.

This is how you find rules that are technically correct but operationally noisy. For example, repeated tool calls may be a defect in one workflow and expected pagination in another. Narrowing the scope is usually better than teaching people to ignore the signal.

A detection match remains attached to the underlying run so the label can be checked against the evidence.

How a match becomes part of an incident

An incident is one production problem on the incidents page, with its evidence, investigations, priority, and status in one place. A match does not become an incident by itself. Incidents reach the page in two ways:

Neatlogs raises them. Automatic analysis reviews recent span errors, latency above each operation's normal baseline, and detection matches. A model judges which detection matches indicate a real problem, and related events, such as the same error from the same operation, are grouped into one incident. When the problem continues, Neatlogs updates the existing incident instead of opening a new one.
You ask for one in investigate. Describe the problem in investigate and Neatlogs can create an incident for it, checking the page for an existing one first. An investigation you start from chat produces a draft visible only to you, under Drafts, until you select Track as issue.

There is no control on the incidents page for creating an incident by hand.

Each incident brings the useful context together:

the affected behavior, in What we know so far on the Overview tab;
representative traces, spans, and detection matches on the Evidence tab;
first seen, last seen, and how many runs were affected;
priority and status;
each investigation and its report on the Investigations tab, with the latest Recommended action on Overview;
status changes, hand-offs, and comments on the Activity tab.

When an investigation turns out to describe an incident that is already open, Neatlogs links it to that incident and marks it Duplicate instead of creating a second incident.

Status tracks the work

The page groups incidents into Open, Investigated (an investigation has completed), Resolved, and Drafts tabs, and keeps archived incidents behind the Archived fixes (archive icon) button. Change an incident's status to Open, Investigating, Resolved, or Archived from its Status menu, and reopen a resolved incident the same way. Neatlogs never overwrites a status or priority that a person set. It can, however, resolve an incident it raised when nobody has changed or investigated it and later analysis no longer finds the problem.

Incidents have no assignee. Use comments to coordinate who is working on one.

Priority describes impact

Priority tells the team how urgently a problem deserves attention: Critical, High, Medium, or Low. Neatlogs gives an incident it raises an initial priority from fixed rules, for example ranking an error that ends a trace above an error in a nested step, and an investigation report carries its own. Change it with Priority on the incident when you know more about the impact.

Priority should reflect customer and business impact, not simply the number of matches or the confidence of a classifier. A high-confidence match can describe a low-impact condition. A single match can be critical if it exposes sensitive data or performs an unsafe action.

When an alert should fire

An alert is useful only when a person or system should act. Create it after the detection is stable enough that you understand its normal volume.

Alert rule	Use it when	Example
Per event	One occurrence deserves immediate attention	A high-risk data exposure detection matches
Threshold	A burst or sustained count matters more than one match	Five tool failures occur within 10 minutes
Compound	Several detection conditions together indicate the problem	Repeated tool calls and an error response occur in the same window
Metric threshold	An aggregate operational measurement crosses a boundary	Average latency rises above the accepted limit
Ingestion health	Missing telemetry is itself an operational problem	A production project stops receiving traces

Set the evaluation window, threshold, severity (Critical, Warning, or Info), delivery channel, and cooldown to match the response you expect. Cooldowns prevent the same ongoing condition from sending a stream of identical notifications.

Every alert appears in the app. A rule can also notify by email, Slack, or one webhook. Alerts do not create evidence that was not already present, and firing an alert does not prove a root cause.

Incidents do not need an alert rule. Each new incident notifies project members according to their Incident alerts preference in Settings → Account → Notifications, and a project can also post new incidents to an Incident Slack channel under Settings → Project → Project notifications. Alert rules cannot be muted or snoozed; disable a rule to stop it.

Detections and Evals answer different questions

Both can identify concerning behavior, but they serve different jobs.

Use a detection when...	Use an Eval when...
You can define a repeatable matching condition	You need judgment against a quality standard
The answer comes from fields, patterns, categories, or trace structure	The answer comes from a rubric and supplied evidence
The main result is where and how often the condition appeared	The main result is a comparable verdict for each subject
You want a stable production signal for filtering or alerting	You want human or AI reviewers, calibration, and aggregate results

For the support agent, lookup_order was called at least five times is a detection. The agent resolved the request without unnecessary steps is an Eval question. A failed Eval may lead you to discover a repeatable failure mode and create a detection for it.

Boundaries to keep clear

A detection observes completed spans or traces. It does not block a model or tool call.
PII detection labels evidence that may contain sensitive data. It does not redact what is captured, stored, or displayed.
A match proves that the configured rule fired. It does not prove a defect or root cause.
An alert decides when to notify. It does not investigate or resolve the condition.
An incident records one problem and stays linked to its evidence, investigations, and resolution. Neatlogs can raise it, but people decide when it is resolved.

Frequently asked questions

Does every detection need an alert?

No. Many detections are most useful as searchable labels and trends. Add an alert only when a defined response is expected.

Does every match become an incident?

No. Neatlogs's incident analysis decides which detection matches indicate a problem and groups related ones. A match may be expected, low impact, or useful only as a trend.

Can I test a detection before creating it?

Yes, for most types. While you configure a detection, the wizard previews recent spans or traces marked Match or No match. PII has no preview step. Review both matching and non-matching evidence before relying on the rule. Preview behavior differs by type because deterministic, scored, and code-based rules use different evaluation paths.