Idempotency and retries

The PDF from the queue guide is now searchable. The user asks, “What are the key decisions in this document?” and presses Send once. We create reply task A7: produce one response to that message. The browser may disconnect and a worker may crash, but those failures do not mean the user asked another question.

For this constructed example, the work runs in the background and the completed answer is saved before the chat displays it. Background execution lets accepted work outlive the connection; it is not a requirement for every chat product. We are protecting the stored reply, rather than making model output deterministic or proving its answer correct.

Name the promise precisely: at most one saved response per reply task. That leaves room for no response if every allowed attempt fails, and for repeated model computation during recovery. A user who intentionally asks for another answer creates new logical work. Those distinctions tell us which identities and safeguards the design needs.

The API saves the user message, creates A7 and records a recoverable queue handoff. As in the upload guide, accepted work cannot depend only on an API process remembering to publish. The dispatcher can publish again after a failure, but every delivery still identifies A7.

A worker receives the task and calls the LLM provider. The returned text is a candidate answer. It becomes the saved response only when our database accepts it. Saving the response and marking A7 complete happen together, so a reconnecting browser cannot observe “complete” without the corresponding reply.

The worker acknowledges queue progress after that durable save. The browser can fetch the saved response later; queue acknowledgment does not wait for the screen to display it. Keep user message, task, delivery, attempt, candidate answer and saved response distinct. We can now move a crash between these steps and identify the missing evidence without changing what the user requested.

Suppose the provider returns text, then the worker crashes before saving it. A7 is still unfinished, so recovery tries again. The provider may do expensive work twice even though only one task exists. A queue can preserve unfinished work without preventing that repeated computation.

With SQS, receiving a message normally hides it for a visibility period. Successful processing is followed by deletion. If it is not deleted before the period expires, another worker can receive it. A worker can extend the period, but Standard queues can still deliver duplicates during visibility. Treat that timer as delivery coordination, not an exclusive correctness lock.

With a traditional Kafka consumer group, reading does not remove a record. The group saves a restart position. If its committed position is 41, it can read record 41 again after a crash. Once that record and all earlier work are durably handled, committing 42 records where to resume. Never advance progress past unfinished work merely because a later attempt finished first.

Two workers can both return candidate text. If each independently reads “no saved response” and later appends its answer, both may succeed. Move the decision into the authoritative database operation: accept a candidate only when the task still has no response and the worker satisfies the completion rule developed below.

Once a candidate is accepted, its response and completed state commit together. Repeating completion leaves that saved answer unchanged. The chat reads this accepted result rather than displaying every attempt that returns. Different model text is compatible with this behavior because idempotency is about the saved effect at this boundary.

This does not guarantee an answer exists or refund duplicate model work. It also does not undo text already streamed to the user. A streaming product needs an accepted stream, attempt-aware events and reconnect rules. Our simpler save-before-display contract avoids exposing unaccepted candidates while we establish the local completion mechanism.

The browser creates a submission ID for one press of Send and reuses it when that submission must be retried. The API scopes the key to the authenticated caller or workspace and operation. The first accepted submission creates one message and A7; a repeated submission resolves to those existing records and their status.

Tie the key to the original conversation, document version and other relevant inputs. Reusing it with changed inputs is an error, rather than permission to mutate the old task. Retain the record for the supported retry and replay window. Deleting it too soon can make a delayed duplicate look like new work.

Regenerate is different: the user deliberately requests another answer, so it receives a new task even if the words are identical. Prompt hashing cannot reliably express that intention. Give executions separate attempt IDs for tracing, while keeping A7 stable across its retries. Submission deduplication prevents new tasks; it does not prevent repeated deliveries of the existing one.

A worker first checks whether A7 is already complete. If so, it skips the provider call. Otherwise it asks the shared database to claim the available task in one atomic operation. Two workers cannot both win the same current claim; the loser avoids an unnecessary call.

A permanent running flag would strand the task after a crash. Instead, the claim has a renewable time limit, called a lease. A live worker renews it while making progress. When it expires, a replacement can take over under the retry policy. Keep database operations short instead of holding a transaction open throughout the provider call.

The lease is separate from SQS visibility, and neither can cancel a remote model call already in progress. A duplicate that sees an active claim should defer or rely on a durable scheduler that will revisit unfinished tasks. It must not remove the only recovery path simply because another worker appears active. Completed work can safely be acknowledged because its result is already durable.

A request leaves the worker, then its connection fails. Perhaps the provider never received it. Perhaps generation is still running. Perhaps it finished and the returned answer was lost. All can look like “no result” in the local task record, and a new lease does not settle which occurred.

First check for a documented recovery contract at that endpoint: suitable idempotency support or durable lookup using an operation identity already retained. A request ID useful for support logs does not automatically permit result recovery. Payload rules, key scope, retention and concurrent-call behavior determine what a provider contract actually protects.

If recovery is unavailable, choose a policy. In this example, a bounded retry is worth the possible duplicate computation because the user still needs a reply. Another product may stop and reconcile instead. Neither choice follows automatically from local idempotency; one controls the risk of a missing result and the other the risk of a repeated remote effect.

Worker A claims A7, then loses contact without dying. Its lease expires and worker B takes over. A later returns with text while the response slot is still empty. Checking only “no answer yet” would let the old assignment win, even though the recovery protocol selected B.

BEGIN;

SELECT task_id
FROM reply_tasks
WHERE task_id = :reply_task_id
FOR UPDATE;

UPDATE reply_tasks
SET saved_response = :candidate_answer,
    status = 'completed'
WHERE task_id = :reply_task_id
  AND assignment_version = :my_assignment_version
  AND status = 'running'
  AND lease_until > clock_timestamp()
  AND saved_response IS NULL
RETURNING task_id;

COMMIT;

Each new assignment receives an atomically increasing version. The worker remembers its version. The database accepts completion only if that version is still current, the assignment remains valid, the task is running and no response has been saved. Saving the answer and completing the task still commit together. The old version cannot authorize a new write after takeover; this is fencing.

The completion transaction first locks the task row. On the same connection, its guarded update checks the lease against the database’s changing clock_timestamp(), then saves and completes while that lock is held. This defines lease validity at completion authorization; commit may follow later, while a takeover waits for the lock. CURRENT_TIMESTAMP records transaction-start time and would not provide a fresh expiry check after lock waiting. Keep this transaction short and give claims and renewals equivalent ownership and time checks.

If the completion operation reports no accepted row, read durable state rather than generate again blindly. Another worker may have completed, or ownership may have expired. If a save succeeded but its acknowledgment was lost, the saved task resolves that uncertainty. If it succeeded before a queue-acknowledgment crash, the replacement observes completion and skips another model call.

Separate failures by what is known. If the provider was definitely not called, an eligible worker may try the same task. A documented retryable response follows the endpoint’s retry rules. An unknown outcome needs recovery or the explicit duplicate-cost policy. Invalid input or another terminal failure should become a clear failure instead of unchanged resubmission.

For allowed retries, backoff increases the wait between attempts and jitter varies it so workers do not all retry together. Limit total attempts or elapsed time, and respect cancellation and the task deadline. Backoff without a stopping rule can keep failing work alive indefinitely; waiting does not make invalid input valid.

Count retries from the browser, API, worker and provider SDK together. Repeated layers can multiply remote calls. Queue redelivery must not reset A7’s budget. If repeatedly failing work is moved to a dead-letter queue, record a visible task outcome and preserve identity for deliberate replay. Moving a message aside does not tell the waiting user what happened.

A submission key prevents a repeated HTTP request from creating another task. A renewable claim reduces simultaneous work on an existing task. An atomic completion gate protects the accepted stored response. We use them together because none covers the others’ whole responsibility.

If a consumed record’s local database effect and restart position are stored in one database, one transaction can sometimes cover both. That does not bring a separate LLM service into the transaction. Downstream idempotency or durable operation lookup can help only where the owner of that effect provides a suitable contract.

Sometimes repeated computation is acceptable while repeated publication is not. That is our choice here: attempts may produce different candidates and cost money, but A7 has at most one saved reply. A payment, notification or other external action would need its own protection. Always name the effect before calling processing exactly once.

For ingestion, logical work might mean “index this document version at this stage.” A retry keeps that identity; uploading a new version creates new work. The harmful repeat could be duplicate index entries, while a late older version becoming current is another effect to prevent. The details differ from a saved chat response, but the boundary questions remain useful.

Follow the crash around the durable effect. Before publication, unfinished work must remain discoverable. After publication but before queue progress, redelivery must recognize completion or repeat safely. A replacement must not mistake its empty local cache for evidence that the task never ran, and a stale worker must not overwrite a newer accepted result.

These are expected behaviors to verify in the actual implementation. A diagram demonstrates the intended protocol; it is not proof of a tested service. Explain the authoritative state behind each outcome and name the policy for an unknown external result. That produces a transferable design instead of a list of queue and database features.