Flag a problem and add context
Turn an observed failure into a precise, evidence-linked report that another person can investigate.
A useful problem report separates what happened from why you think it happened.
For example, “The assistant offered a refund under an expired policy” is an observation. “The vector database is stale” is a hypothesis. Recording the first without presenting the second as fact gives investigators room to find the real cause.
What a complete report contains
| Part | Question it answers |
|---|---|
| Observed behavior | What exactly did the agent say or do? |
| Expected behavior | What should have happened instead? |
| Evidence | Which trace, span, output, or report demonstrates it? |
| Impact | Who or what was affected, and how seriously? |
| Context | Which policy or business fact is missing from telemetry? |
| Unknowns | What still needs investigation? |
Start from the strongest evidence available
If you know the affected run, open it and confirm that its input, output, time, workflow, and customer context match the report. Highlight the smallest useful evidence, such as the exact answer, tool result, or retrieved document.
If the report came from Slack, support, or a customer without a trace ID, search by the most stable facts first:
- project and environment;
- approximate time and timezone;
- workflow, session, or end-user identifier;
- distinctive output or tool result.
When no single run is known, attach a few confirmed examples rather than a large unreviewed search result.
Attach the report to the evidence
Neatlogs has no separate Flag or Report button. You flag a problem by commenting on the exact evidence, so the report stays attached to the run it describes. On the trace detail page:
| To flag | Do this |
|---|---|
| A passage in an input, output, or tool result | Select the text, then choose Comment in the menu that appears |
| One span | Hover the span in the trace tree and select Comment |
| One field in a span's details | Select Comment on field next to the field |
| The whole run | Select the Comments button in the trace header |
Anyone with access to the project can comment, including a Project Viewer. Type @ in the comment to mention the person who should look next. See Comment and mention.
Spans a detection flagged also carry Upvote and Downvote buttons, on the detection's Flagged spans tab. A vote is a quick, visible reaction that shows who voted. It does not record what went wrong and does not create an incident, so add a comment when the problem needs explaining.
Anchor the report to the evidence that demonstrates the failure, not to a screenshot that loses execution context.
Write the report
Use language another person can test.
Observed: The final answer offered a refund under policy v2.
Expected: Use the policy that was active on the order date.
Impact: The customer received an option support cannot honor.
Evidence: Final answer and retrieve_policy span in the linked trace.
Unknown: Why policy v2 was selected.Describe the mismatch
State what the agent received and what it produced or did. Avoid adjectives such as “bad” or “broken” when a concrete mismatch is available.
Add the expected behavior
Define the desired outcome without prescribing an unverified implementation. Link the governing policy, acceptance rule, or prior decision when one exists.
Add impact without inflating it
Describe affected customers, operational risk, financial or safety consequences, recurrence, and any safe workaround. Severity should reflect impact and urgency, not the confidence of the reporter.
Record unknowns
Naming what is unknown helps the investigation. It is acceptable not to know which component failed.
Decide how the problem gets tracked
Neatlogs creates incidents itself, from detections and investigations. There is no control for creating an incident by hand, so choose the path that matches the report:
| The report is | Do this |
|---|---|
| A question, expected behavior, or a low-impact observation | Leave it as a comment and mention whoever can answer |
| Actionable work that needs a cause | Select the evidence and choose Ask neat AI, or start from investigate. When the investigation proposes a fix, select Track as issue on the result card to add it to Incidents |
| Likely to recur | Create a detection so Neatlogs flags future matches and raises incidents for them |
| Already covered by an open incident | Add your evidence as a comment on that incident instead of starting a parallel thread |
Once the problem is an incident:
- add representative traces and highlighted spans as comments on it;
- include the expected behavior and impact;
- link relevant detections or failed Evals;
- set Priority from impact (Critical, High, Medium, or Low);
- mention the person who should follow up. Incidents have no assignee field, so the mention is the handoff.
Check the report before handing it off
A reader should be able to answer:
- Which exact behavior was wrong?
- What should have happened?
- Where can I inspect the evidence?
- Why does this deserve attention?
- Which statements are facts and which remain hypotheses?
If the report cannot answer those questions, add context before starting an investigation.
Frequently asked questions
Do I need to know the cause?
No. A precise symptom and expected behavior are enough to begin. Investigation determines the supported cause.
Can I create an incident by hand?
No. Incidents come from detections and investigations. To get a problem tracked, investigate it and select Track as issue, or create a detection that catches it.
Does a downvote create an incident?
No. Upvote and Downvote record a reaction on the span. Write a comment to explain the problem.
Can I flag imported evidence?
Yes, provided the import preserved the fields needed to demonstrate the problem.
What if the report contains customer data?
Include only the context needed to understand impact. Do not copy secrets or unnecessary personal data into comments or external tickets.
Who can see the report?
People with access to the project. A comment on a trace or span also appears as an in-app notification for the other members of your organization, so keep sensitive detail out of the comment text. External destinations apply their own permissions.