> ## Documentation Index
> Fetch the complete documentation index at: https://langchain-5e9cc07a-preview-opensw-1787674483-6bd21af.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Model fallbacks

> Automatically retry a request against backup models when the primary model rate-limits, errors, or returns another configured status code.

<Note>
  **Beta:** The LLM Gateway is in [beta](/langsmith/release-stages).
</Note>

Model fallbacks retry a request against one or more backup models when the primary model returns a configured error, such as a rate limit or provider outage. Define the fallback order once in LangSmith, then continue using the standard LLM Gateway endpoint and model ID in your application.

## How it works

A fallback chain has:

* **A primary model**: the provider and model that trigger the chain when a request fails.
* **One to five fallbacks**: an ordered list of direct provider models or saved [model configurations](/langsmith/model-configurations).
* **Triggers**: the upstream HTTP status codes that move the request to the next model. For example, use `429` for rate limits, or `500`, `502`, `503`, and `504` for provider errors.

For each request, the gateway:

1. Calls the primary model selected by the request's provider-prefixed model ID.
2. If the request fails with a configured trigger status or a transport error, loads the matching fallback chain.
3. Calls each fallback in order until one succeeds, returns a status that does not trigger another fallback, or the chain is exhausted.
4. Returns the final response in the API format used by the client.

Fallbacks can use a different provider and API format than the primary model. The gateway translates requests and responses between [supported API formats](/langsmith/llm-gateway-api-formats), so an Anthropic primary can fall back to an OpenAI model without client-side changes.

Each attempt is traced and counted against [spend policies](/langsmith/llm-gateway-spend-policies) separately. A request that uses two fallbacks records three model calls: the primary attempt and two fallback attempts.

## Create a fallback chain

<Warning>
  Creating and managing fallback chains requires `organization:manage` permission. For the full permissions breakdown, see [Access control](/langsmith/llm-gateway-access).
</Warning>

To create a fallback chain:

1. Go to **Settings → Gateway → LLM Gateway** and select the **Model Fallbacks** tab.
2. Click **Create fallback chain**.
3. Select the **Workspace** where the chain applies.
4. Select the primary provider and model. Requests to this provider-prefixed model ID use the chain when the primary attempt fails.
5. Under **Fallbacks**, add one to five backup models in the order the gateway should try them. Choose a provider and model directly, select an existing model configuration, or create a custom model configuration.
6. Under **Configure fallback triggers (advanced)**, review the HTTP status codes that should trigger the next fallback. Add or remove status codes as needed.
7. Click **Create chain**.

A provider and model can have one fallback chain in each workspace. To change its behavior, edit the existing chain.

## Make a call

Call the standard LLM Gateway endpoint with the primary provider-prefixed model ID. You do not need a route-specific URL or additional request fields:

<CodeGroup>
  ```bash Cloud theme={null}
  curl https://gateway.smith.langchain.com/v1/chat/completions \
      -H "Authorization: Bearer $LANGSMITH_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"model":"anthropic/claude-sonnet-4-6","messages":[{"role":"user","content":"Hello!"}]}'
  ```

  ```bash BYOC theme={null}
  curl https://<data_plane_host>/gateway/v1/chat/completions \
      -H "Authorization: Bearer $LANGSMITH_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"model":"anthropic/claude-sonnet-4-6","messages":[{"role":"user","content":"Hello!"}]}'
  ```
</CodeGroup>

The gateway applies the fallback chain configured for `anthropic/claude-sonnet-4-6` in the API key's workspace. If no chain matches, the gateway returns the primary model's response without attempting a fallback.

## Choose fallback candidates

You can add two types of fallback candidates:

* **Direct provider model**: select a supported gateway provider and model. This option uses the workspace's secret for that provider, or Gateway Credits for eligible hosted models.
* **Model configuration**: select a saved workspace [model configuration](/langsmith/model-configurations). Use this option for a custom OpenAI-compatible or Anthropic endpoint, a custom model name, or configuration-specific parameters.

Model configurations are workspace-scoped. A fallback chain can only use configurations from its selected workspace.

For example, configure `anthropic/claude-sonnet-4-6` as the primary model, `openai/gpt-5.4-mini` as the first fallback, and a saved OpenAI-compatible model configuration as the second fallback. The application continues to request `anthropic/claude-sonnet-4-6`; the gateway selects and translates fallback calls when needed.

## See also

* [API formats](/langsmith/llm-gateway-api-formats): review supported request formats and translation behavior.
* [Custom model providers](/langsmith/llm-gateway-custom-providers): create model configurations for custom OpenAI- or Anthropic-compatible endpoints.
* [Spend policies](/langsmith/llm-gateway-spend-policies): apply cost limits alongside fallback routing.

***

<div className="source-links">
  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>

  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/langsmith/llm-gateway-fallbacks.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>
</div>