Model providers and keys

Model-provider settings control which models supported Neatlogs AI workflows can use. They do not control which models your traced application uses.

Choose the source of model access

Model access comes from two places:

• a Neatlogs-managed model, which needs no configuration;
• your own provider key, configured for the project or inherited from an existing organization key.

How each AI workflow uses them:

A model name appearing in a picker does not by itself guarantee that a provider connection is configured. When no key is set up for the project, each picker links to AI provider settings.

If an AI evaluator is set to a provider whose key is later removed or deactivated, the evaluator runs on the Neatlogs managed model instead of failing. When evaluation must stay with your own provider, check each evaluator's model after you change or remove a key. An LLM judge's Decision model setting is a separate data flow: see Data and security.

Understand the credential types

Do not substitute one for another or store provider credentials in trace attributes.

Configure a provider

Open Settings → Project → AI Providers with the intended project selected. Provider keys are project-scoped: keys configured on this page apply only to the selected project.

The page has two sections:

• Project keys: the provider catalog for this project. Use the All, Configured, and Available tabs or search to find a provider, then select Set up (or Configure for an existing key).
• Organization keys: existing keys shared across the organization, which this project inherits. New keys are added per project. Use Test connection to check one. Removing one may interrupt AI features in other projects.

Project members and admins can set up, test, and remove provider keys. Project viewers see the page as view-only; ask a project member or admin to configure a key.

The catalog includes OpenAI, Anthropic, Google Gemini, Azure OpenAI, xAI, Groq, Mistral, DeepSeek, Together AI, Fireworks AI, Cerebras, a Self-hosted / Custom OpenAI-compatible endpoint, and many other OpenAI-compatible providers. Azure OpenAI asks for a Base URL, API key, optional API version, and the Deployment names you created in Azure.

In the configuration dialog, choose a Default model, then select Test key before Save.

After saving:

1. confirm the key tests successfully;
2. run one safe Playground or evaluator example;
3. confirm the expected model appears in the intended picker;
4. inspect any authorization, region, deployment-name, or model-access error.

Choose a model for the task

Consider:

• judgment or investigation quality;
• evidence size and context requirements;
• structured-output reliability;
• tool-use support;
• latency;
• token and model cost;
• data-processing requirements.

Test on representative traces before broad assignment. A model that succeeds on one short example may fail on long traces, ambiguous rubrics, or tool-heavy investigations.

Different Neatlogs workflows can support different provider and model combinations. Verify the picker in the intended evaluator, Playground, or Investigate surface.

Keep credentials out of evidence

Store provider credentials only through the intended settings and secret paths.

Never place keys in:

• prompts;
• trace inputs, outputs, or attributes;
• comments;
• evaluator examples;
• tickets;
• screenshots shared with support.

If a key appears in captured evidence, revoke it at the provider and follow the organization's incident process.

Rotate a provider key

Use a staged rotation:

1. create the replacement key at the provider;
2. update the credential in Neatlogs (Configure on the provider row);
3. select Test key;
4. run a safe dependent workflow;
5. confirm evaluators and other consumers succeed;
6. revoke the old key at the provider;
7. record the rotation according to your security process.

Avoid revoking the active key before confirming the replacement, unless immediate revocation is required for security. Removing a project key with Remove provider key can make models unavailable or fall back to the managed model.

Troubleshoot provider access

Frequently asked questions

Does one configured provider work for every AI feature?

Not necessarily. Model and provider support can differ between Playground, evaluators, and Investigate.

Can project members see the raw key?

No. Saved keys are returned masked. Keep your own copy in your secret manager: you will need the original key to reconnect after removing it.

Why is a model unavailable after the key tests successfully?

The account may not have access to that model, or the intended Neatlogs workflow may not support it.

Is the provider key the same as the Neatlogs API key?

No. Provider keys authorize model use. Neatlogs project keys authorize telemetry or supported project API access.

Next step

Review data and security controls →