Monitor quality and operations

Monitoring tells you where behavior changed. Traces and investigations tell you what happened and why.

Start on Home when you want a period briefing and prioritized findings. Open Analytics when you need to explore the underlying quality, cost, latency, token, and error measurements directly.

The useful monitoring loop is:

Choose a production question
    ↓
Read the relevant quality and operational signals
    ↓
Compare equivalent populations and periods
    ↓
Open representative traces
    ↓
Investigate or alert when action is needed

Start with a question, not a dashboard

Examples:

• Did refund-answer quality decline after release 2026.08.20?
• Is the checkout agent slower because of model latency or repeated tools?
• Did the recently fixed policy failure return?
• Are costs rising because traffic increased or because each run uses more tokens?

The question determines the project, time range, workflow, comparison period, and signals you need.

For a daily check, open home. Its needs attention list surfaces new incidents, detections firing well above their usual rate, alerts that fired, and traces stopped flowing when a project that normally sends traces has been silent for at least two hours. The daily digest summarizes the day. Treat these as starting points, then scope the question in analytics or traces.

Establish a fair scope

Before interpreting a movement, confirm:

• project;
• workflow, model, agent, or tool;
• time range (time pickers use your browser's local time);
• release, prompt, or model version;
• traffic volume and customer mix;
• Eval sampling rate and schedule;
• detection scope and recent edits.

analytics covers one project. It filters by Workflow, Model, Agent, Tool, Framework, Detection, Status, Session ID, and User ID, over presets from 15 mins to 3 months or a Custom range of up to 90 days (the default is the last 14 days). It has no environment or release filter. Record environment and release as tags, as described in Keep environments and releases separate, and filter traces by Tags or Metadata to compare them.

Compare equivalent populations. A quality score can fall because a new workflow or harder customer cohort entered the sample.

Read quality signals

Eval results

Use a Recurring Eval schedule for questions that require a rubric. Compare batches with the same question, reviewers, and Sampling rate. Neatlogs does not chart Eval results across batches, so open each batch's Report tab, or export its responses, and compare them side by side. See Read results and calibrate evaluators.

Open failed and surprising verdicts. A changed average may come from one failure slice or a rubric update rather than a broad regression.

Detection frequency

Detections track known conditions. Read counts together with traffic volume and open representative matches.

The detections list shows a Trend sparkline for each detection, and a detection's Overview tab charts it over 7d, 14d, 30d, or 3m. In analytics, the Detection rate card and the Detection analysis widget show hits and hit rate per detection.

A spike can mean:

• the behavior truly became more common;
• traffic shifted toward an affected workflow;
• the detection's rule or scope was edited, which clears its earlier matches and backfills the last 14 days;
• instrumentation began capturing evidence that was previously absent.

Team and customer reports

Reports can reveal failures not covered by existing Evals or detections. Link them to traces, then decide whether the pattern deserves a new signal.

Read operational signals

analytics opens with six cards: Total traces, Avg duration, Total cost, Error rate, Avg tokens, and Detection rate. Each card compares the selected range with the previous period of the same length. Below them are the Time breakdown, Cost breakdown, Error analysis, Detection analysis, Tool performance, Trace comparison, and Arc patterns widgets.

Scope the dashboard before interpreting a change, then open the runs behind the affected signal.

Do not rely only on averages:

Trace duration is shown as an average only. Percentiles are available per tool in Tool performance, and as the P95 and P99 aggregations of a Metric threshold alert.

Move from trend to evidence

When a chart changes:

1. identify when the movement began;
2. compare release, prompt, model, workflow, and traffic changes;
3. segment the affected population;
4. open traces from before and after (chart points do not link to traces, so filter traces to the same window and filters, or open a tool's recent traces from its row in Tool performance);
5. inspect the execution path behind the signal;
6. decide whether the change is a regression, expected movement, or measurement change.

Compare models, workflows, and tools

Change one primary dimension at a time where possible. Comparing model A in production refund traffic with model B in staging checkout traffic cannot isolate model behavior.

Use stable workflow, environment, release, and model identity. Then inspect cases that explain the aggregate difference.

In analytics, filter by Model, Workflow, Agent, or Tool, and group Cost breakdown or Time breakdown by model or agent. Trace comparison lines up recent runs of one workflow and flags outliers against the median; select Pin as baseline to compare the others with a known-good run. It compares runs within a workflow, not releases.

Watch recently fixed behavior

For every important fix, monitor:

• the original failure signal;
• the Eval that represents expected behavior;
• operational guardrails affected by the change;
• representative new production traces;
• the release or prompt version containing the fix.

Define how long and how much traffic is enough before closing the incident. See Verify a fix.

Decide whether to alert or investigate

Not every regression should page someone. Alert only when an owner has a defined response.

Frequently asked questions

Why do Eval and detection trends disagree?

They may measure different questions, subjects, or populations. Compare scope, time range, rubric, and detection condition.

What is the difference between Analytics and Investigate?

Analytics summarizes configured measurements. Investigate gathers evidence to answer an open causal question.

How do I compare releases?

Neatlogs has no release filter. Attach a stable release identifier as a tag during instrumentation, filter traces by Tags for each period, and keep other population dimensions comparable.

What if cost rises while cost per run stays stable?

Traffic likely increased. Confirm trace volume and workflow mix before treating it as an efficiency regression.

Can I export Analytics or change its layout?

Analytics has no export. You can rearrange widgets, hide them, and reset the layout from its edit control.

Should a dashboard movement automatically create an incident?

No. Confirm the evidence and impact first. Track the behavior as an incident only when it represents actionable work.

Next step

Investigate a change in production →