Test a change

A useful change test begins with a decision rule, not a candidate that already exists.

Define the behavior you want to improve, the behavior that must not regress, and the operational limits the candidate must respect.

Ship when policy-validity improves on known failures,
successful controls remain stable,
and p95 latency increases by less than 10 percent.

Build the test around the failure

Suppose an investigation finds that refund retrieval ignores the request date.

The test should include:

• traces that reproduced the expired-policy failure;
• current-policy cases that already worked;
• requests at the policy boundary;
• missing or malformed dates;
• non-refund cases if shared retrieval code changes;
• ordinary production traffic.

This set tests both improvement and overreach.

Keep baseline and candidate identifiable

The candidate may be a managed prompt version, model, Playground configuration, or application branch. Preserve the baseline and label both clearly.

Give each variant its own tag, such as prompt-v2 for the baseline and prompt-v3 for the candidate, so an Eval condition on Tags can select each variant's traces separately.

Record:

• prompt, model, workflow, and release identity;
• input set;
• environment and external dependencies;
• evaluator and rubric versions;
• relevant runtime settings.

Without stable identity, an apparent difference may come from several changes at once.

Try prompt and model variants in Playground

experiments (marked beta in the sidebar) has two pages: prompts, which stores managed prompt versions and labels, and playground, which runs prompt and model variants side by side.

Use Playground for a quick first comparison before you change the application:

1. Open experiments → playground, or open a saved prompt and select Open in Playground.
2. Add one side-by-side panel per variant, give each the same messages and variable values, then run them.
3. Select Diff to highlight how each output differs from the first panel's.
4. Select Save as prompt on the variant you want to keep. It becomes a version under prompts, which your application can fetch through the managed prompts SDK.

Playground calls models through your own provider credentials. If the model picker offers no models, add a provider under Settings → Project → AI Providers or ask a teammate with Project Member or Project Admin access to add one (see Model providers and keys). Running Playground and saving or labeling prompts also require Project Member or Project Admin access. A Project Viewer can read prompts and comment on them.

Playground runs a handful of typed inputs. It does not replay a candidate over a set of production traces, and its runs are not recorded as traces, so Evals cannot score them. For the comparison that decides the ship, run the baseline and the candidate through your application.

Choose representative evidence

Do not test only the examples that motivated the change.

Use the same inputs for baseline and candidate where possible. When non-determinism matters, run enough repetitions to distinguish a stable movement from random variation.

Apply the same judgment

Use the same calibrated questions and evaluators for both versions. Add another Eval only when the existing rubric cannot measure the intended behavior.

Evals score traces your application has sent to Neatlogs. Create one Eval per variant with identical settings, and use a Tags condition to select each variant's traces. Select Save as template on the Form step of the first Eval, then pick that template from Templates in the second, so both variants answer the same questions.

Inspect individual cases before aggregate results. Averages can hide one severe regression or a candidate that improves easy cases only.

Compare quality and operations together

Open traces where execution changed. A quality improvement caused by skipping a required tool or refusing more requests is not necessarily acceptable.

Investigate regressions

For every important worsened case, determine whether it is:

• a systematic failure;
• limited to one workflow, cohort, model, or language;
• evaluator noise;
• missing evidence;
• a tradeoff accepted by the decision rule.

Update the candidate or narrow rollout. Do not average away a high-impact regression.

Make the ship decision

Record:

1. whether each acceptance gate passed;
2. evidence for the decision;
3. unresolved risks;
4. rollout scope;
5. production signals to watch;
6. rollback condition.

Promotion still follows an approval step. For a managed prompt, choosing the production label in prompts opens Request approval for production promotion, and the label moves to the new version only after the approver you pick approves it. Code deployments follow your normal release process.

Verify after release

Pre-release comparison shows how the candidate behaved on the selected evidence. Production verification tests whether that result holds under real traffic.

Monitor the relevant recurring Eval, detection, cost, latency, errors, and representative traces. If the original behavior returns, set the related incident back to Open.

Frequently asked questions

Is Playground a complete experiment system?

No. Experiments is in beta, and Playground is a prompt and model test surface for inputs you type. It does not run a candidate over stored traces, split traffic, or record its runs as traces. Run broader comparisons through your application and score them with an Eval.

Are Datasets or Golden Sets required?

No. Datasets, experiment runs, and A/B tests are not available yet and do not appear in navigation. Select the comparison set with Eval conditions instead.

Can imported and native traces be compared?

Yes, when the required fields and subject semantics are comparable.

What if quality improves but cost doubles?

Apply the decision rule. If cost is a constraint, the candidate does not pass without an accepted tradeoff or another iteration.

Should a test use only failed traces?

No. Include successful controls and ordinary traffic to expose overfitting and regressions.

Next step

Verify the change in production →