Retries and idempotency¶

When interacting with APIs in distributed environments, network latency, connection drops, and timeouts can occur. In these scenarios, the client often cannot know whether the service received and processed the request or whether a response was generated.

To protect against unintentional duplicate charges and ensure safe request retries, Nexus provides an idempotency mechanism via the Idempotency-Key header.

Key concepts: Idempotency-Key vs x-request-id¶

Nexus distinguishes between two identification headers:

Header	Purpose	Retry behavior
`Idempotency-Key` (or `X-Idempotency-Key`)	Semantic idempotency for generation requests	Retrying with the same key and identical body returns the saved result without additional balance charges.
`x-request-id`	End-to-end tracing of an individual network attempt	Must be unique for every new distinct request. Reusing a past `x-request-id` on a new request is rejected (`duplicate_request_id`).

When a request is executed with an Idempotency-Key, a replayed response retains the original x-request-id of the first attempt and includes an additional header:

Idempotency-Replayed: true

On the initial successful execution, the Idempotency-Replayed header is omitted.

Key format and scoping¶

The Idempotency-Key header must satisfy the following requirements:

Length: 1 to 256 printable ASCII characters (1..=256).
Characters: alphanumeric characters, hyphens, underscores, and standard delimiters (e.g. UUID v4: 9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d).
Invalid format: an empty key, length exceeding 256 characters, or non-ASCII characters returns 400 Bad Request with code invalid_idempotency_key.

Scoping¶

An idempotency key is scoped to:

Your Nexus profile;
The specific API key;
The endpoint family (e.g. Chat, Messages, Responses).

Identical keys sent under different API keys or for different endpoint families will not collide.

Idempotency lifecycle¶

Each idempotency key transitions through several states:

[ New Request ]
       │
       ▼
[ Check Key ] ─── Key already exists? ───► No ──► [ Execute Request (Winner) ]
       │                                                      │
      Yes                                                     ▼
       │                                         [ Save response for 24 hours ]
       ├─► Request body mismatch? ──► [ 409 idempotency_conflict ]
       │
       ├─► Initial request still running? ──► [ Wait for completion / 409 idempotency_in_progress ]
       │
       ├─► Over 24 hours ago (but under 7 days)? ──► [ 410 result_expired ]
       │
       └─► Exact replay within 24 hours ──► [ Return cached response (Replay) ]
                                             (Idempotency-Replayed: true, debit: 0)

1. Initial execution (Winner)¶

The first request with a given key executes the model invocation normally. Upon receiving a response, its status, headers, and body are saved in the durable idempotency cache for 24 hours. The profile balance is debited for the actual execution usage. The response is returned to the client without the Idempotency-Replayed header.

2. Exact replay (Replay)¶

When a client sends a request with the same Idempotency-Key and an identical request body within 24 hours, Nexus returns the cached response:

HTTP status code and body match the original response;
The x-request-id header matches the original request ID;
The header Idempotency-Replayed: true is included;
No balance debit occurs (exactly 0 additional units).

3. Concurrent requests (In Progress)¶

If two identical requests arrive concurrently with the same idempotency key:

The second request joins the broadcast waiting channel of the active request;
If the first request completes within the waiting window, the second request receives the result with Idempotency-Replayed: true;
If the wait times out or coordination fails, a 409 Conflict response is returned with code idempotency_in_progress and header Retry-After: 1. The client should retry after the specified interval.

4. Body mismatch (Conflict)¶

If a request arrives with an existing key but a different request body (for instance, a different model, prompt, or temperature), Nexus does not execute the request and returns 409 Conflict with code idempotency_conflict.

Never reuse an Idempotency-Key for semantically different requests. Always generate a fresh unique key (e.g. UUID) for each distinct operation.

5. Expiration (Expired)¶

Cached results remain available for replay for 24 hours.
After 24 hours, the response payload is cleared, leaving a tombstone record for 7 days.
If a client retries with the key while the tombstone is active, Nexus returns 410 Gone with code result_expired. The client must generate a new key for a new execution.

Streaming lifecycle (SSE and WebSocket Realtime)¶

For streaming operations, the following rules apply:

Arbitrary chunk resumption is not supported: if an SSE connection drops on the client side, resumed playback of remaining chunks is not supported.
Client disconnect before response headers: if the client disconnects before response headers are generated, the request is cancelled and remains unbilled.
Client disconnect during an active stream: if the client disconnects after data transmission has started, processing continues in the background to capture final token usage.
If final token usage is captured, the actual execution is debited exactly once.
If generation halts or usage cannot be determined, the request remains unbilled.
Intentionally dropping the client connection does not make completed generation free.

Decision matrix: when and how to retry¶

Use this table when designing client retry policies:

Scenario / Error	HTTP	Error code	Safe to retry?	Identity to use	Action
Network drop / Connection timeout	0 / `timeout`	—	Yes	Same `Idempotency-Key`	Retry with exponential backoff and jitter
Client rate or concurrency limit	429	`customer_rate_limit_exceeded`, `customer_concurrency_exceeded`, `key_*`	Yes	Same or new key	Wait `Retry-After` seconds, reduce concurrency
Service capacity / Queue full	503	`provider_capacity_exceeded`, `global_capacity_exceeded`, `queue_full`, `queue_timeout`, `start_capacity_exceeded`	Yes	Same `Idempotency-Key`	Wait `Retry-After` seconds and retry
Temporary protection / Server reload	503	`protection_mode_engaged`, `billing_protection_engaged`, `server_shutting_down`	Yes	Same `Idempotency-Key`	Wait `Retry-After` seconds and retry
Request with key still in progress	409	`idempotency_in_progress`	Yes	Same `Idempotency-Key`	Wait `Retry-After` (typically 1s) and retry
Body mismatch for existing key	409	`idempotency_conflict`	No (with same key)	New key	Fix body or generate a new key
Cached result expired	410	`result_expired`	No (with same key)	New key	Generate a new key for a new execution
Invalid idempotency key format	400	`invalid_idempotency_key`	No	Fix key	Use 1 to 256 printable ASCII characters
Duplicate request ID	409	`duplicate_request_id`	No (with same ID)	New `x-request-id`	Generate a new `x-request-id` or use `Idempotency-Key`
Insufficient profile balance	402	`insufficient_balance`	After top-up	Same or new key	Top up balance via the Telegram bot
Invalid API key	401	`invalid_key`	No	—	Check API key header
Model or endpoint not allowed	403	`model_not_allowed`, `endpoint_not_allowed`	No	—	Check available models in `GET /v1/models`
Invalid JSON syntax or parameters	400	`invalid_request_json`, `invalid_request_error`	No	—	Correct request syntax or parameters
Model timeout	504 / 502	`upstream_timeout`, `upstream_unavailable`	Yes	Same `Idempotency-Key`	Retry with backoff

Handling timeouts and network drops¶

If your client sends a request and does not receive an HTTP response (read timeout or TCP reset):

If you sent an Idempotency-Key:
Resend the request with the exact same Idempotency-Key and the exact same request body.
If the server completed the original execution, you will receive the response with Idempotency-Replayed: true and no extra charge.
If the server is still processing the request, you will receive the completed response or a 409 status with Retry-After.
If you did not send an Idempotency-Key:
Retrying will create a completely new request, which may result in a duplicate charge.
We recommend sending an Idempotency-Key on all stateful generation calls.
Do not generate a new Idempotency-Key for an uncertain outcome:
Creating a new key prevents the service from recognizing that the request is a retry.

Code examples¶

cURL¶

export NEXUS_API_KEY='your-key'
export IDEMPOTENCY_KEY="req-$(uuidgen)"

# Initial request
curl -i https://api.nexus-hub.tech/v1/chat/completions \
  -H "Authorization: Bearer $NEXUS_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $IDEMPOTENCY_KEY" \
  -d '{
    "model": "MODEL_FROM_V1_MODELS",
    "messages": [{"role": "user", "content": "Reply exactly OK."}],
    "max_tokens": 16
  }'

# Safe retry following a network timeout (same IDEMPOTENCY_KEY and identical body):
curl -i https://api.nexus-hub.tech/v1/chat/completions \
  -H "Authorization: Bearer $NEXUS_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $IDEMPOTENCY_KEY" \
  -d '{
    "model": "MODEL_FROM_V1_MODELS",
    "messages": [{"role": "user", "content": "Reply exactly OK."}],
    "max_tokens": 16
  }'

The second response includes the header:

Idempotency-Replayed: true

Python OpenAI SDK¶

The official OpenAI SDK accepts arbitrary custom headers via extra_headers:

import os
import uuid
import httpx
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["NEXUS_API_KEY"],
    base_url="https://api.nexus-hub.tech/v1",
)

idempotency_key = str(uuid.uuid4())

try:
    response = client.chat.completions.create(
        model=os.environ["NEXUS_MODEL"],
        messages=[{"role": "user", "content": "Reply exactly OK."}],
        max_tokens=16,
        extra_headers={"Idempotency-Key": idempotency_key},
    )
    print("Response received:", response.choices[0].message.content)
except (httpx.TimeoutException, httpx.NetworkError) as err:
    print(f"Network error: {err}. Performing safe retry...")
    # Retrying with the same idempotency key guarantees no duplicate charge
    response = client.chat.completions.create(
        model=os.environ["NEXUS_MODEL"],
        messages=[{"role": "user", "content": "Reply exactly OK."}],
        max_tokens=16,
        extra_headers={"Idempotency-Key": idempotency_key},
    )
    print("Response after retry:", response.choices[0].message.content)

Python Anthropic SDK¶

For Anthropic Messages, headers are also supplied via extra_headers:

import os
import uuid
import httpx
from anthropic import Anthropic

client = Anthropic(
    api_key=os.environ["NEXUS_API_KEY"],
    base_url="https://api.nexus-hub.tech",
)

idempotency_key = str(uuid.uuid4())

try:
    message = client.messages.create(
        model=os.environ["NEXUS_MODEL"],
        max_tokens=16,
        messages=[{"role": "user", "content": "Reply exactly OK."}],
        extra_headers={"Idempotency-Key": idempotency_key},
    )
    print("Response:", message.content[0].text)
except (httpx.TimeoutException, httpx.NetworkError) as err:
    print(f"Network error: {err}. Retrying with the same key...")
    message = client.messages.create(
        model=os.environ["NEXUS_MODEL"],
        max_tokens=16,
        messages=[{"role": "user", "content": "Reply exactly OK."}],
        extra_headers={"Idempotency-Key": idempotency_key},
    )
    print("Response after retry:", message.content[0].text)

Compatibility¶

Sending Idempotency-Key is completely optional. Standard clients without this header continue operating normally.
This mechanism does not require switching clients to an asynchronous pattern (202 Accepted) with polling: all responses are returned synchronously on both the initial attempt and subsequent replays.