Verify a fix

A merged change is not yet a verified fix.

Verification tests the investigation's prediction: if the reported cause and implemented change were correct, the affected behavior should improve while important controls, cost, latency, and error behavior remain acceptable.

Neatlogs supplies the evidence for that test (traces, Evals, detections, and analytics), but it does not mark a fix as verified for you. You run the checks below, record the decision on the incident, and change its status yourself.

Define the acceptance gate first

Write the decision rule before running the candidate. This prevents the result from being judged against whatever happened to improve. Start from the Verification steps in the report's Recommended action, then tighten them.

Ship when:
1. all reproducible expired-policy failures pass;
2. current-policy and ineligible-item controls do not regress;
3. the policy-validity Eval improves;
4. p95 latency stays within the agreed limit;
5. the original production signal remains at baseline after rollout.

Verify in three stages

1. Reproduce the known failure

Start with the exact traces cited by the investigation; they are on the incident's Evidence tab. When direct replay is unsafe or unavailable, create representative inputs that preserve the facts required by the failure. To compare a baseline and a candidate prompt or model on the same inputs, follow Test a change.

For every case, record:

• baseline behavior;
• candidate behavior;
• applicable Eval or deterministic result;
• important execution differences;
• whether the expected behavior was met.

One passing example proves only that one example passed.

2. Protect unchanged behavior

Add successful and edge cases that the fix could accidentally affect. For a policy-date change, include:

• a request under the current policy;
• a request at the policy boundary;
• a missing or malformed date;
• an ineligible item;
• a retrieval timeout;
• a non-refund workflow if shared code changed.

Compare outputs and execution paths, not only aggregate scores. An overly broad fix can satisfy the original failure by refusing every request or skipping the affected tool.

3. Watch production evidence

After normal code review and deployment, scope Neatlogs to the changed workflow and release. On Traces, filter by Workflow and by the Tags or Metadata your application records for the environment and release.

Watch:

• the original detection, or the production question you asked in investigate;
• the relevant recurring Eval;
• error, cost, token, latency, and tool-use changes;
• representative new traces;
• customer or team reports.

If the behavior returns, the incident's Evidence tab may show Recurrence history with the new occurrences.

Choose a monitoring window that covers meaningful traffic and relevant periodic behavior. Ten quiet minutes may be enough for a high-volume workflow and meaningless for a weekly process.

Compare baseline and candidate fairly

Use equivalent inputs and conditions where possible.

If the production population changed, segment the results before attributing movement to the fix.

Record the verification decision

Add a comment on the incident that preserves:

• implementation or release identifier;
• cases and populations tested;
• baseline and candidate results;
• Eval, detection, and operational evidence;
• reviewer and decision;
• rollout scope;
• production monitoring period;
• remaining uncertainty.

This record makes future recurrence easier to understand and keeps “verified” from becoming an undocumented opinion.

Close, continue, or reopen

To change an incident's status, open the status menu in its sidebar (it shows the current status, such as Open) and choose another. Changing status can require the Project Member role; viewers should ask a project member or admin.

Reopen the original incident when the recurrence represents the same problem. When a similar symptom has a different cause or owner, keep it separate: investigate a representative trace and use Track as issue to track the new finding.

Frequently asked questions

Does Neatlogs deploy the change?

No. Deployment remains in the team's engineering workflow.

Does Neatlogs verify the fix or reopen the incident automatically?

No incident shows an automatic verification result. Don't rely on Neatlogs to reopen a resolved incident for you: watch the original signal after release and reopen it yourself if the behavior returns.

How many cases are enough?

Use enough diverse evidence to cover the known failure, boundaries, and likely regression paths. One case is rarely sufficient.

Can an Eval verify latency or cost?

Use Evals for quality judgment. Use trace and analytics signals for latency, cost, errors, and tool behavior.

What if pre-release tests pass but production does not improve?

Keep the incident open, or reopen it. Compare the test population with real traffic and continue the investigation.

When should I resolve the incident?

After the agreed checks pass and enough new production evidence supports the expected behavior.

Next step

Monitor quality and operations →