Model deployments

Model deployments are an in-product setting. They live under Admin, apply at runtime, and never require a chart change or a redeploy — on Helm or on Embedded Cluster alike. Only platform admins see these screens.

Note

Adding the very first deployment is a post-install step — see First model deployment. This page covers ongoing management and the model underneath, so you can configure your own providers rather than copy a screenshot.

How a model request travels

Dreadnode does not call model providers directly. Every request goes through LiteLLM, an OpenAI-compatible proxy that owns provider routing, credentials, rate limits, and usage accounting. A deployment is a row in LiteLLM’s routing table that you edit through the Dreadnode admin UI.

flowchart LR
  C["Chat, agent,<br/>or evaluation"] -->|"asks for dn/my-model"| L["LiteLLM proxy"]
  L -->|"provider_model<br/>at api_base<br/>with credential"| P["Your model endpoint"]
  P -->|"response + token usage"| L
  L --> C

Two things follow from this, and both surprise people:

• The name the platform uses and the name the provider uses are different strings. dn/my-model is the alias your users pick from a dropdown. Provider Model is what LiteLLM sends upstream. A deployment is the mapping between them.
• Agents and sandboxes talk to LiteLLM directly, not through the API. When a session starts, the platform mints a short-lived LiteLLM virtual key and hands the sandbox that key plus a base URL. The inference traffic itself never transits the Dreadnode API.

Which topology you are running

Topology	What it means	How you get it
Bundled	LiteLLM runs in-cluster as part of the chart, served at /llm on your platform domain	Enable the bundled proxy in chart configuration
External	You already run a LiteLLM proxy and point the platform at it	Set both LiteLLM URLs and provide the proxy’s master and salt keys

Either way the admin screens work the same. What changes is which URL has to be reachable from where.

The bundled proxy stores its model catalog in a litellm schema on the platform’s PostgreSQL. If you run an external database, the proxy uses that one, and the account you configured needs CREATE SCHEMA on it. AWS IAM database authentication is not compatible with the bundled proxy.

Bundled proxy

Helm

dreadnode-litellm:
  enabled: true

dreadnode-api:
  config:
    litellm:
      enabled: true

Admin Console

Open Config → Inference Proxy and enable Enable bundled LiteLLM proxy.

KOTS ConfigValues

apiVersion: kots.io/v1beta1
kind: ConfigValues
spec:
  values:
    litellm_enabled:
      value: "1"

External proxy

For an external proxy, create a Secret whose values match the proxy’s LITELLM_MASTER_KEY and LITELLM_SALT_KEY, then reference it from the API chart.

Helm

Set $NAMESPACE to the release namespace, then create the Secret:

kubectl -n "$NAMESPACE" create secret generic external-litellm \
  --from-literal=masterKey='<external-proxy-master-key>' \
  --from-literal=saltKey='<external-proxy-salt-key>'

dreadnode-api:
  config:
    litellm:
      enabled: true
      internalUrl: https://litellm.internal.example
      publicUrl: https://litellm.example.com/v1
      secretName: external-litellm

Admin Console

Enter the cluster shell and create the Secret in kotsadm:

sudo ./dreadnode shell
kubectl -n kotsadm create secret generic external-litellm \
  --from-literal=masterKey='<external-proxy-master-key>' \
  --from-literal=saltKey='<external-proxy-salt-key>'

Then open Config → Advanced Helm Values → Helm Values, paste the Helm block from the Helm tab, and deploy.

KOTS ConfigValues

A ConfigValues file cannot create the referenced Kubernetes Secret. Install Embedded Cluster, create external-litellm in kotsadm, then apply the Helm block through the Admin Console. The dedicated ConfigValues surface supports the bundled proxy only.

The master key must be accepted by the external proxy. The API also needs the matching salt key to create and update stored provider credentials.

Who has to reach the proxy

Two separate URLs, and confusing them is a common install failure:

Setting	Who uses it	Bundled-proxy default
Internal URL	The Dreadnode API, from inside the cluster	In-cluster LiteLLM service on port 4000
Public URL	Sandboxes and agent runtimes, from wherever they run	<scheme>://<domain>/llm/v1

The default public URL is correct only when your platform domain is reachable from wherever sandboxes run. On-cluster OpenSandbox is fine. E2B sandboxes run in E2B’s cloud, so an internal-only platform must expose a tunnel or bastion ingress and set the public URL to it — that is the LiteLLM Public URL field under Inference Proxy, and the matching dreadnode-api.config.litellm.publicUrl value on Helm.

Caution

A public URL that sandboxes cannot reach fails at run time, not at configuration time. Model pickers look healthy; agent sessions fail to call any model.

Credentials and deployments are separate

A credential is a named, reusable secret — an API key, and optionally a base URL and API version. A deployment is a routable model that references one. The split exists so you can rotate a key once and have every model that uses it pick up the change, and so exports can carry model configuration between environments without carrying secrets.

Reuse a credential when several deployments hit the same provider account. That is the normal case: one openai-prod credential behind five dn/* models. The credentials list shows which deployments reference each one, so you can see what a rotation will touch.

Note

Saving a credential under an existing name overwrites its values. That is the supported way to rotate a key — no redeploy, no deployment edits.

Two details that cost people time:

• A credential’s provider label is display metadata only. It never affects routing. Routing comes from the provider/model prefix in Provider Model.
• A credential is not required. For a one-off endpoint you can put the key directly on the deployment, but nothing else can reuse it.

The dn/ rule

Model Name must start with dn/. A deployment whose name does not is invisible to every model surface in the platform — chat, agents, and evaluations all filter on that prefix. It will sit in LiteLLM working perfectly and never appear in a dropdown.

The admin API rejects the mistake rather than letting it disappear quietly:

model_name must start with 'dn/' (e.g. 'dn/claude-sonnet-bedrock');
put the provider identifier in provider_model instead

That error is the whole rule: dn/… is your alias, and the provider’s own identifier belongs in Provider Model.

Where api_base comes from

A base URL can be set in three places, so “which one is actually used?” has a real answer. On every model call, LiteLLM resolves them in this order:

Precedence	Source	Where you set it
1 (wins)	The deployment’s API Base field	Add / Edit deployment
2	The deployment’s raw litellm_params	Advanced Parameters JSON
3	The credential’s API Base	Admin → Credentials

A credential only fills in what the deployment left unset. So a shared credential can carry the default endpoint for a provider account while one deployment overrides it — but if you set API Base on the deployment, the credential’s value is ignored for that model. The same rule applies to the API key and API version.

When you need it, and when it must be omitted

Situation	API Base
A commercial provider’s own API (OpenAI, Anthropic)	Leave empty. LiteLLM knows the endpoint; setting it wrong breaks routing
A self-hosted OpenAI-compatible server (vLLM, TGI, llama.cpp, LM Studio)	Required — this is the whole point
Azure OpenAI	Required, plus an API version
An internal gateway or proxy in front of a provider	Required

The suffix convention is what trips people up: OpenAI-compatible servers expose their API under /v1, and LiteLLM appends the route (/chat/completions) to whatever you give it. Enter http://vllm.internal:8000/v1, not the bare host and not the full completions path. If test connection returns a 404, a wrong /v1 is the first thing to check.

Get to a working model

The shape most on-prem deployments need is an OpenAI-compatible endpoint you run yourself.

1. Admin → Credentials → Add credential. Name it after the account (gpu-cluster) and paste the API key. If every model on that endpoint shares a base URL, set API Base here and leave it off the deployments.

2. Admin → Model Deployments → Add deployment.

   Model Name:     dn/qwen3-coder
   Provider Model: openai/Qwen3-Coder-30B
   Credential:     gpu-cluster
   API Base:       <leave empty; the credential supplies it>

   The openai/ prefix tells LiteLLM to speak the OpenAI protocol — it does not mean OpenAI the company. Use it for any OpenAI-compatible server. The part after the slash is whatever model name your server answers to.

3. Test connection, and only then Create deployment.

4. If the model is not in LiteLLM’s pricing catalog, optionally set Behaves Like to a comparable known model or enter explicit input and output prices. The model works without either on self-hosted Dreadnode, but its cost and capability metadata remain unknown. See Pricing and capabilities.

Configure any provider

LiteLLM supports well over a hundred providers, and its own documentation is the authoritative reference for each one. Rather than restate it, here is how to read a LiteLLM provider page and land it in these fields.

Every LiteLLM provider page shows a Python snippet like this:

response = completion(
    model="azure/<your-deployment-name>",
    api_base="https://<resource>.openai.azure.com",
    api_version="2024-10-21",
)

Map it across:

In LiteLLM’s docs	In Dreadnode
model="<prefix>/<model-id>"	Provider Model
api_base=	API Base, or the credential’s API Base
api_key=, or the provider’s *_API_KEY env var	The credential’s API Key
Any other keyword argument (api_version, aws_region_name, vertex_project)	Advanced Parameters as JSON
Nothing — this is ours	Model Name, which must start with dn/

That last row is the only Dreadnode-specific part. Anything LiteLLM accepts as a completion() keyword argument is accepted in Advanced Parameters under the same name.

Worked provider shapes

For Azure OpenAI, store the key, endpoint, and API version on one reusable credential:

Credential name: azure-prod
API Key:         <azure-api-key>
API Base:        https://<resource>.openai.azure.com
API Version:     2024-10-21

Model Name:      dn/gpt-4o-azure
Provider Model:  azure/<azure-deployment-name>
Credential:      azure-prod

For AWS Bedrock, use the Bedrock model prefix and put AWS-specific connection values in Advanced Parameters:

Model Name:      dn/claude-bedrock
Provider Model:  bedrock/<bedrock-model-id>

{
  "aws_access_key_id": "...",
  "aws_secret_access_key": "...",
  "aws_region_name": "us-east-1"
}

If the LiteLLM pod already receives an AWS workload identity, omit the static access keys and set only the region and any provider-specific values it needs.

For a direct commercial provider, save its API key on a credential, leave API Base empty, and use the provider’s prefix:

Credential name: openai-prod
API Key:         <openai-api-key>

Model Name:      dn/gpt-4.1
Provider Model:  openai/gpt-4.1
Credential:      openai-prod
API Base:        <leave empty>

Provider pages worth bookmarking

Provider	Prefix	LiteLLM docs
Any OpenAI-compatible server	openai/	openai_compatible
vLLM	openai/ or vllm/	vllm
Ollama	ollama/	ollama
Azure OpenAI	azure/	azure
AWS Bedrock	bedrock/	bedrock
Google Vertex AI	vertex_ai/	vertex
Databricks	databricks/	databricks
OpenAI	openai/	openai
Anthropic	anthropic/	anthropic
DeepSeek	deepseek/	deepseek
Together AI	together_ai/	togetherai
Fireworks AI	fireworks_ai/	fireworks_ai

For anything not listed, start from the full provider index — the mapping above works the same for all of them.

Caution

When you put an OpenAI-compatible gateway in front of a provider, the prefix follows the gateway, not the provider behind it. Fronting Vertex AI with such a gateway means openai/<whatever path the gateway exposes> — not vertex_ai/… — because the prefix selects the wire protocol and everything after it is passed through as the model name. This is why a working deployment can legitimately read openai/vertexai/gemini-2.5-pro.

Providers needing more than a key

Some providers authenticate with something other than a single API key. The credential form has fields for API key, base, and version only, so the rest goes in Advanced Parameters:

{
  "aws_access_key_id": "...",
  "aws_secret_access_key": "...",
  "aws_region_name": "us-east-1"
}

LiteLLM credentials can also carry vertex_project, vertex_location, vertex_credentials, region_name, and watsonx_region_name. Set them per-deployment as above, or share them across deployments through the admin API’s credential values bag.

Verify, and read the failures

Test connection issues a real completion against the deployment. Admin → Diagnostics shows model-service health. LiteLLM routing and credential changes apply immediately. A model picker served by another API replica can take up to about a minute to refresh its model-list cache.

Symptom	Cause
Model missing from every picker	Model Name does not start with dn/
Model in the picker, sessions fail to call it	Sandboxes cannot reach the LiteLLM public URL
Test connection returns 404	Wrong API Base — usually a missing or doubled /v1
Test connection returns 401 or 403	Credential is wrong, or an API Base on the deployment points somewhere the credential’s key is not valid for
Adding a deployment returns 503	LiteLLM is not enabled or not reachable — see troubleshooting
Costs read as zero or obviously wrong	Model is outside the pricing catalog and has no Behaves Like or explicit prices

Move deployments between environments

Actions → Export deployments downloads the portable subset of the current configuration. Exports never carry API keys—create the credentials in the destination first, using the same names.

Actions → Import deployments previews every row before applying it, classifying each as create, update, skip, or error. Review that preview: a row fails outright when it names a credential the destination does not have. Leave Create all as new off for normal imports — it forces duplicates instead of reconciling in place. Import accepts only the version 2 export envelope; unversioned files, bare arrays, and version 1 exports are rejected.

Actions → Refresh model cost map pulls current pricing and context windows for newly released models.

Exports carry model_name, provider_model, hidden, credential_name, api_base, base_model, per-million prices, rpm, tpm, and order—never API keys or LiteLLM’s internal model IDs. They do not carry raw litellm_params, raw model_info, or provider-specific advanced values. Every DB-backed deployment remains a separate row, including deployments that share the same (model_name, provider_model) pair. Treat the file as a portable baseline, not a lossless backup; recreate provider-specific advanced settings in the destination. Import reconciles each shared pair one-to-one, preferring exact row matches before updating another member or creating a new one:

Verdict	Meaning
create	No unused editable route remains for that shared pair
update	Reconciles with one distinct editable deployment, exact matches first
skip	Matches a config-backed deployment, which the API cannot edit
error	A credential is missing or the model name is not dn/-prefixed

Field reference

Credential

Field	Required	Notes
Name	Yes	Unique. Deployments reference this string. Saving an existing name overwrites its values
API Key	—	The provider secret. Returned masked once saved
API Base	—	Default endpoint for every deployment using this credential
API Version	—	Required by Azure OpenAI; ignored by most other providers

A credential needs at least one of API key, API base, or an additional value. The admin API also accepts a provider label (display metadata, never routing) and an open values bag.

Deployment

Field	Required	Notes
Model Name	Yes	Must start with dn/. Several deployments may share one name — see Rate limits and routing
Provider Model	Yes	prefix/model-id sent upstream
Hidden deployment	—	Hides the model ID from pickers and member allowlists only when every deployment sharing its Model Name is hidden
Credential	—	Named credential to resolve connection values from
API Base	—	Endpoint for this deployment. Wins over the credential’s API Base
Behaves Like	—	A known model to inherit pricing, context window, capabilities, and supported parameters from
Input Price / Output Price	—	USD per 1M tokens. Overrides catalog and inherited rates
Advanced Parameters	—	Raw litellm_params JSON
Model Info	—	Raw model_info JSON

api_key (inline, instead of a credential), rpm, tpm, and order are accepted by the admin API but have no field on the form — set them through Advanced Parameters.

Caution

Both JSON bags merge on save, and each is capped at 8,000 serialized characters. Removing a key from the editor does not unset it—set it to a new value instead. Masked **** secrets are kept as stored unless you replace them.

Pricing and capabilities

Models outside LiteLLM’s pricing catalog have no cost data, context window, or capability metadata until you supply one of the following.

Behaves Like (base_model) inherits all of it from a known model, including which parameters are allowed through (for example reasoning controls). Use it for a self-hosted model comparable to a catalog one. It does not change which endpoint is called. An unrecognized value is a warning, not an error — the deployment saves and silently inherits nothing:

base_model 'gpt-5-turbo-xl' is not recognized by LiteLLM — it will not inherit
pricing, context window, capabilities, or supported parameters (e.g.
reasoning_effort). Check the spelling or use a known model name.

Input Price / Output Price override cost only, and take precedence over anything Behaves Like supplies. Set them for a negotiated or self-hosted rate. Self-hosted deployments may omit prices; the model still works, but usage has no meaningful cost value.

Rate limits and routing

Several deployments may share one Model Name. That forms a routing group: users see a single dn/* entry and LiteLLM distributes requests across the members. Use it for a primary plus a fallback endpoint, or to spread load across regions.

Hidden deployment applies to one member of the routing group. The shared model ID disappears from model pickers and member allowlists only when every deployment in the group is hidden. If any member remains visible, the model ID remains visible and member access rules still apply. A task can declare a fully hidden model ID explicitly; deployment availability still applies.

Field	Behavior
rpm	Requests-per-minute capacity hint. Under the bundled default router, this weights random selection; it is not a hard quota
tpm	Tokens-per-minute capacity hint, used as a routing weight under the bundled default
order	Priority tier within the group. LiteLLM tries the lowest numbered tier first and falls back to higher tiers after eligible retryable failures

If any member has an order, give every member one: unordered members are excluded while an ordered member is available. Drag-to-reorder writes the same display order to every deployment in a model-name group, so it cannot create primary/fallback tiers and overwrites distinct tier values. Set routing tiers through the admin API after arranging the display order.

Related

LiteLLM providers The authoritative per-provider reference for model strings and required parameters.

Chat models How the dn/* namespace looks from the user's side.