Skip to main content

​
Routing and reliability

Every chat, messages, and embeddings request is routed to a provider from the model catalog. When that provider fails, Prism can retry it and move to the next configured provider before returning an error. This page explains what happens, what you control, and how to read the result.

​
Timeouts

Each upstream attempt has a timeout to first byte. The default is 60 seconds. Send x-prism-timeout-ms to change it for one request:
Values are clamped to the server’s bounds (at least 1000 ms). Anything that is not a whole number of milliseconds is ignored and the default applies. A timeout lower than the server default disables the timeout retry for that request, so a short timeout on a long non-streaming answer cannot cost two generations.

​
Retries and fallback

When enabled by the operator, Prism retries the same provider on transient failures (rate limits, 5xx, connection errors, one timeout) with exponential backoff, honouring the provider’s Retry-After. If the provider still fails and the model has fallback providers configured, the request moves to the next one. The whole sequence shares one time budget, so a request never waits on retries indefinitely. Prism never retries or falls back on errors caused by the request itself: a prompt that exceeds the model’s context window, an invalid parameter, or a provider rejecting the request as malformed comes back to you unchanged. You are billed once, for the attempt that produced the response. Failed attempts are never charged.

​
Conditional routing

An operator can attach rules to a model that send a request to a different provider, upstream model or credential when its context matches. The context is the organization’s tags (set by the operator), the key’s scopes, the model id, and request metadata: the body’s metadata object merged over the x-prism-metadata header (a JSON object, up to 4 KB, at most 32 primitive values). Metadata is only ever compared against; it never names a provider.
The first matching rule wins and replaces the model’s whole chain, including its fallbacks, so a rule that pins a region never leaks out of it on a transient failure. x-prism-provider still reports the provider that served the request. Metadata is sent by the caller, so any key can send any value: operators use it for preferences (a region, an experiment), and match on organization tags or key scopes for anything with a cost or compliance consequence. A missing metadata key never equals null; it does satisfy $ne.

​
Reading the response headers

These headers are also set on error responses, so you can see what Prism tried before giving up.

​
Error codes

When every attempt fails, the error code names what happened last: Embeddings use the same codes with the runtime_embeddings_failed: prefix. See Errors for retry guidance. อ่านภาษาไทย