Backpressure and capacity

A user uploads a PDF. The API durably accepts its processing job, returns a receipt, and a worker later extracts, embeds, and indexes the document. Acceptance means that our service has taken responsibility for this job; searchable completion is a later event. The three-minute target runs from acceptance to that completion.

Suppose uploads keep succeeding immediately while documents wait longer in the queue. The upload API can look healthy even as users wait too long to search their documents. We therefore measure accepted-to-searchable time as well as response time. Waiting is part of the user experience even when it happens outside the HTTP request.

The queue lets already accepted work survive a worker interruption and wait for another attempt. That is useful, but it cannot make workers finish faster. Our next decision is how much waiting the product can absorb, when additional workers help, and what to do before accepting work we cannot safely finish.

Normally four documents arrive each second. Tests on this workload suggest that the current worker fleet can finish about five documents per second when it has enough work. Five per second is its useful capacity under those test conditions. With only four arriving and no backlog, actual throughput is about four, because there is no fifth document to finish.

Once enough work is waiting, completions can approach the tested five-per-second rate. If fifteen are accepted each second while five finish, unfinished work increases by about ten each second. Count successful document completions, rather than worker starts or attempts, because a failed attempt can consume resources without producing a searchable document.

Our estimates assume similarly sized jobs, arrival-order waiting, roughly even completions, and a stable useful service rate. We count waiting documents separately from documents already being processed. Those assumptions make the arithmetic understandable; they do not guarantee an exact deadline for every PDF. A different document mix needs another capacity measurement.

Start with almost no waiting work. Uploads jump from four to fifteen per second for thirty seconds. Workers finish five per second, so the waiting backlog grows at approximately fifteen minus five, or ten documents per second. Thirty seconds of that gap leaves about three hundred waiting documents.

Then arrivals return to four per second. Five can finish while four new documents join, leaving one completion per second to reduce the backlog. Clearing three hundred at that net rate takes about three hundred seconds. Dividing by all five completions would ignore the continuing arrivals.

Five minutes to empty the queue does not mean every document waits five minutes. A document already in line is ahead of later arrivals. We will estimate its own wait separately. Buffering alone may be enough if that wait fits the product target; a short spike is not automatically a reason to start more workers. If the waiting budget is threatened, additional useful capacity can shorten recovery.

Now start the example again, but keep arrivals at fifteen per second. Five still finish. After thirty seconds approximately three hundred documents wait; after ninety seconds approximately nine hundred wait. There is no quiet period in which the existing workers catch up.

A larger queue delays running out of storage, but it does not reduce this ten-per-second growth. Letting users wait indefinitely also changes the three-minute promise. To support the higher demand, we need more useful capacity, a lower rate of accepted work, or a combination while capacity is being prepared.

We prefer to serve users when we can do so safely. That means first finding the bottleneck and checking whether workers can increase actual document completions. Refusing new work can protect the service during a capacity gap, but it does not erase the obligation to finish documents already accepted.

A worker is a running copy of the processing program. Several copies can process different PDFs concurrently; they need not each occupy a separate physical machine. For this example, assume workers are the bottleneck and their storage, embedding, and indexing dependencies have spare capacity.

Use an explicit new snapshot: three hundred documents are waiting when added workers become useful, and fifteen still arrive each second. Capacity of fifteen per second keeps up with new arrivals but leaves the backlog around three hundred. Capacity of twenty leaves five completions per second for recovery. Three hundred divided by twenty minus fifteen gives about sixty seconds to clear.

That minute begins when useful capacity is available. If workers take time to start, more work can accumulate first, and the recovery estimate must use that larger backlog. More running instances are only a means to the goal: verify that successful searchable completions actually rise before relying on their proposed capacity.

A scaling controller periodically reads the workload, compares it with a configured target, and requests more or fewer workers. A useful signal is backlog per ready worker: waiting work divided by workers currently able to process it. Three hundred waiting documents is a different burden for five workers than for twenty.

For illustration, choose a target of fifteen waiting documents per ready worker. Three hundred waiting divided by five ready gives sixty per worker, above the target. Three hundred divided by the target of fifteen suggests twenty total workers. This is an example controller decision, not a measured production policy or a deadline guarantee.

The controller must remember workers already starting, allow time for their initialization, and respect a maximum consistent with cost and downstream capacity. When scaling down, let work finish or make interrupted jobs safely recoverable. AWS documents backlog-based scaling and worker scale-in protection; the application still has to measure whether the added fleet improves completions and waiting.

Return to five-per-second capacity, four-per-second arrivals, and three hundred waiting documents. A new document joins behind those three hundred. At approximately five completions per second, work ahead takes about sixty seconds to clear. Later arrivals join behind it. They extend whole-backlog recovery, but do not jump ahead of this document in our arrival-order model.

The three-minute target must also cover this document’s processing. Suppose sixty seconds is reserved for processing, movement between services, and variation. That leaves one hundred twenty seconds for waiting. At five per second, six hundred waiting documents would consume that entire allowance. This is a rough planning estimate, not proof that every job meets its deadline.

Act before that point because new workers need startup time and processing varies. Count all accepted waiting work: four hundred visible in the main queue plus two hundred prefetched but not started is six hundred waiting. Moving work into a worker buffer or retry queue does not restart its clock. Already executing work is a separate part of the workload.

Change the assumption: workers could process twenty documents per second, but every document requires an embedding dependency that supports only five of these documents per second. End-to-end completion capacity is at most five. Starting more workers can produce more calls and waiting without more searchable documents.

One response is more dependency capacity. Another is less required work per document while preserving the promised result, or another processing path. Until the bottleneck changes, admission must remain within what the whole path can sustain. For a mixed workload, a two-page PDF and a five-hundred-page PDF should not automatically count as equal demand.

There is also a latency limit distinct from throughput. If one required call takes three minutes, a document cannot finish in under three minutes even with an empty queue. Running several calls concurrently can increase total throughput, but it cannot shorten that document’s required step. We must change its critical path or change the promise.

Start a separate protection snapshot with three hundred waiting, five-per-second capacity, and callers sending fifteen per second. A cooperative batch uploader can respond to feedback by sending four per second. We accept all four, and the backlog shrinks by one per second. Work not yet sent remains the uploader’s responsibility.

HTTP/1.1 503 Service Unavailable
Retry-After: 30
Content-Type: application/json

{
  "message": "We cannot accept more documents right now. Please try again later."
}

That upstream adjustment is backpressure. If callers ignore it, the service must enforce admission: fifteen sent equals four accepted plus eleven refused before acceptance in this example. Refusing excess new work is load shedding. A feedback message alone cannot enforce the limit, and accepting five would hold this backlog steady rather than reduce it.

Temporary overload can use 503 Service Unavailable, optionally with Retry-After; caller rate-limit policies can use 429 Too Many Requests. Neither response grants permission to forget accepted jobs. If even the next document’s estimated wait is already too long, pause new acceptance until it fits again with margin. Reducing arrival rate does not instantly remove the work ahead.

With three hundred waiting, fifteen arrivals, and five completions per second, request more useful capacity if workers are the bottleneck. Twenty-per-second capacity can catch up; fifteen only keeps up. Meanwhile, observe backlog, oldest waiting age, successful completions, and which requested workers have become useful.

Our rough six-hundred-document waiting estimate leaves only three hundred documents of headroom. Continued ten-per-second backlog growth consumes that space in thirty seconds. We cannot assume startup will finish inside that interval. If measured startup and variation threaten the waiting allowance, signal backpressure and enforce safe new admission while protecting accepted work.

If the embedding service is actually capped at five, the worker plan must change. Address that dependency and keep protection in place until useful capacity returns. A clear interview explanation connects each action to the same promise: buffer a finite excess, increase real completion capacity where possible, and stop accepting more than the remaining waiting budget can safely hold.