Model fallbacks

Beta: The LLM Gateway is in beta.

Model fallbacks retry a request against one or more backup models when the primary model returns a configured error, such as a rate limit or provider outage. Define the fallback order once in LangSmith, then continue using the standard LLM Gateway endpoint and model ID in your application.

How it works

A fallback chain has:

A primary model: the provider and model that trigger the chain when a request fails.
One to five fallbacks: an ordered list of direct provider models or saved model configurations.
Triggers: the upstream HTTP status codes that move the request to the next model. For example, use 429 for rate limits, or 500, 502, 503, and 504 for provider errors.

For each request, the gateway:

Calls the primary model selected by the request’s provider-prefixed model ID.
If the request fails with a configured trigger status or a transport error, loads the matching fallback chain.
Calls each fallback in order until one succeeds, returns a status that does not trigger another fallback, or the chain is exhausted.
Returns the final response in the API format used by the client.

Fallbacks can use a different provider and API format than the primary model. The gateway translates requests and responses between supported API formats, so an Anthropic primary can fall back to an OpenAI model without client-side changes. Each attempt is traced and counted against spend policies separately. A request that uses two fallbacks records three model calls: the primary attempt and two fallback attempts.

Create a fallback chain

Creating and managing fallback chains requires organization:manage permission. For the full permissions breakdown, see Access control.

To create a fallback chain:

Go to Settings → Gateway → LLM Gateway and select the Model Fallbacks tab.
Click Create fallback chain.
Select the Workspace where the chain applies.
Select the primary provider and model. Requests to this provider-prefixed model ID use the chain when the primary attempt fails.
Under Fallbacks, add one to five backup models in the order the gateway should try them. Choose a provider and model directly, select an existing model configuration, or create a custom model configuration.
Under Configure fallback triggers (advanced), review the HTTP status codes that should trigger the next fallback. Add or remove status codes as needed.
Click Create chain.

A provider and model can have one fallback chain in each workspace. To change its behavior, edit the existing chain.

Make a call

Call the standard LLM Gateway endpoint with the primary provider-prefixed model ID. You do not need a route-specific URL or additional request fields:

curl https://gateway.smith.langchain.com/v1/chat/completions \
    -H "Authorization: Bearer $LANGSMITH_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"anthropic/claude-sonnet-4-6","messages":[{"role":"user","content":"Hello!"}]}'

curl https://<data_plane_host>/gateway/v1/chat/completions \
    -H "Authorization: Bearer $LANGSMITH_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"anthropic/claude-sonnet-4-6","messages":[{"role":"user","content":"Hello!"}]}'

The gateway applies the fallback chain configured for anthropic/claude-sonnet-4-6 in the API key’s workspace. If no chain matches, the gateway returns the primary model’s response without attempting a fallback.

Choose fallback candidates

You can add two types of fallback candidates:

Direct provider model: select a supported gateway provider and model. This option uses the workspace’s secret for that provider, or Gateway Credits for eligible hosted models.
Model configuration: select a saved workspace model configuration. Use this option for a custom OpenAI-compatible or Anthropic endpoint, a custom model name, or configuration-specific parameters.

Model configurations are workspace-scoped. A fallback chain can only use configurations from its selected workspace. For example, configure anthropic/claude-sonnet-4-6 as the primary model, openai/gpt-5.4-mini as the first fallback, and a saved OpenAI-compatible model configuration as the second fallback. The application continues to request anthropic/claude-sonnet-4-6; the gateway selects and translates fallback calls when needed.

Core capabilities

Administration and governance

Advanced

How it works

Create a fallback chain

Make a call

Choose fallback candidates

See also

​How it works

​Create a fallback chain

​Make a call

​Choose fallback candidates

​See also

How it works

Create a fallback chain

Make a call

Choose fallback candidates

See also