When to add a queue

A user uploads a PDF so that search and chat can reference it later. The product allows two visible states: the upload is accepted, then the document becomes searchable. That permission is the reason we can consider background processing. A slow operation alone would not establish it. If the user needed the complete searchable result in this response, we would have a different request contract.

The product needs fast acknowledgement of a durable upload. Making the PDF searchable can finish afterward.

Follow document doc-456 and ingestion job ing-123 throughout the guide. The document is the stored input. The job is our responsibility to process that input. The request is only the HTTP interaction used to submit it. Those three lifetimes need not end together: the connection may close while the document and unfinished job remain.

We choose to track the ingestion until it reaches READY or FAILED. We have not promised that every PDF will succeed: a corrupt document may be impossible to process. We have promised that acknowledging responsibility will not silently discard its work. That is the shared correctness requirement, including in this first explanation.

The simplest path validates and stores the PDF, performs OCR, runs required checks, creates embeddings, updates the search index, and then responds. It can be a sensible first implementation when volume is low and those steps reliably fit the user’s wait. Avoid treating the absence of a broker as an automatic design flaw.

The response returns only after OCR, compliance, embeddings, and search indexing have finished.

Concurrent clients keep requests open while slow OCR and throttled embeddings constrain the processing path.

Synchronous does not mean that the server handles only one request at a time. It may process many uploads concurrently. Each request still waits for its own slowest stage. If the embedding dependency becomes slow, those requests retain connections and other resources while waiting. New requests can arrive faster than this path finishes them.

A client timeout makes that coupling visible. The client no longer knows whether the upload failed, is still processing, or completed after the connection closed. A retry can add another attempt. Before changing the architecture, explain the actual problem: the request’s lifetime is tied to processing that the product already permits to happen later.

Move OCR, checks, embeddings and indexing after the response. Keep request validation, durable input storage and the recoverable ingestion record before it. Otherwise, the API could say “accepted,” crash, and leave no process able to discover the work it owes the user.

Validation and the ingestion record belong before the response. OCR, compliance, embeddings, and indexing can finish later.

{
  "ingestion_id": "ing-123",
  "status": "ACCEPTED",
  "status_url": "/ingestions/ing-123"
}

One concrete implementation stores doc-456 in object storage, then commits ing-123 and a pending publication record together in a database transaction. A dispatcher reads committed pending records and publishes messages. If the accepting API crashes, another dispatcher can still find the pending record. The dispatcher may publish twice if it loses an acknowledgment, so consumers must tolerate repetition.

The response can be 202 Accepted with ing-123 and a status URL. HTTP 202 says processing is not complete; it does not guarantee eventual success. Our application adds the stronger promise that accepted input and recoverable work are durable. The status URL lets the client distinguish ACCEPTED, QUEUED, PROCESSING, READY and FAILED without keeping the original connection open.

The dispatcher publishes a small message containing ing-123, doc-456, the tenant identity and an object-storage reference. It does not put the PDF bytes in the message. Object storage owns the input, the job record owns status, and the queue tells a worker which job is available. These roles let a replacement worker find the same work without depending on the API process’s memory.

The API returns 202 Accepted after durable acceptance, then the queue holds small pending jobs until workers can run OCR, checks, embeddings, and indexing.

{
  "ingestion_id": "ing-123",
  "document_id": "doc-456",
  "tenant_id": "tenant-7",
  "object_key": "uploads/doc-456.pdf"
}

A worker fetches the PDF, marks the job as processing, performs the pipeline, marks it ready, and acknowledges the queue only after success. At-least-once delivery makes idempotent processing necessary.

The worker fetches the durable input, records that processing has started, and performs the pipeline. It records the successful result durably before acknowledging its queue delivery. READY must mean the required result is actually available to the product; merely starting indexing is insufficient if search still cannot use the document.

Suppose completion succeeds but the worker crashes before acknowledging. The queue can deliver the job again. The replacement must recognize completed work or repeat safe operations, rather than create duplicate index entries. The queue has preserved delivery, but the application still owns the effect of processing twice. The separate idempotency guide develops that mechanism.

Suppose, only for illustration, ten comparable PDFs arrive each second while workers finish six each second for twenty seconds. Forty more arrive than finish in each ten-second interval, so the backlog grows by eighty PDFs over the burst. The numbers describe a simplified workload with constant rates and no retries, rather than measured performance.

After the burst, arrivals fall to two per second while completions remain six. The net drain rate is four per second, so those eighty waiting PDFs need twenty seconds to clear. Dividing by six would ignore the continuing arrivals. If ten keep arriving indefinitely, the backlog grows indefinitely until some storage, waiting or admission limit is reached.

Adding workers can increase useful throughput only if the shared dependencies can sustain their calls. Twenty workers do not help if a single embedding-service limit still allows six documents per second. The queue changes where work waits and allows controlled consumption. Capacity, reduced work per document or reduced accepted demand must resolve sustained overload.

For this example, jobs are independent, one logical worker pool performs them, global ordering is unnecessary, and the product does not need retained upload-event history for replay. Amazon SQS Standard is a reasonable AWS implementation of that work-queue need. This is a choice from the stated behavior, not a claim that every background task needs SQS.

Independent jobs with no replay or global-ordering requirement point toward SQS Standard. Retained history, replay, partition ordering, or multiple consumer groups make Kafka more compelling.

SQS Standard can deliver a message more than once and uses best-effort ordering. A worker therefore must tolerate repeats and must not rely on uploads being processed in a global sequence. A broker’s delivery behavior is part of the design contract; adding its name to a diagram does not remove application recovery work.

If several independent systems need to read the same retained stream, or rebuilding a projection by replay becomes central, Kafka becomes more compelling. Kafka retains events according to topic configuration and supports independent consumer groups. Ordering is within a partition. Those changed requirements justify evaluating another abstraction; they do not imply Kafka is automatically required for high traffic.

Start with request lifecycle: can the user receive acceptance before completion? Then ask whether the work should survive a process failure, whether finite bursts exceed processing rate, and whether consumers need bounded concurrency to protect a dependency. These questions explain the asynchronous boundary, recoverable waiting and controlled consumption we actually used.

Next identify the cost. We added a dispatcher, messaging infrastructure, worker recovery and a status path. The user now waits for a later outcome. We must observe pending handoff age and unfinished work as well as API latency. A short HTTP response is not proof that documents become searchable within an acceptable time.

The decision can reverse. A tiny internal tool with short predictable processing may keep the synchronous path. A product that requires immediate search results may need a smaller synchronous operation or a changed user experience. Our final explanation is specific: this PDF product permits later searchability, so durable jobs and controlled workers solve a real lifecycle and burst problem, while processing capacity remains a separate obligation.