Processing slots, threads, and concurrency

Now the list contains images A, B and C. Starting another image means keeping track of another unfinished download. The worker needs memory for its progress information and any data received but not yet saved. It also has an unfinished output file to manage for that image.

If the worker starts every image in a long list, it can end up with a large amount of unfinished work and too many resources tied up at once. We therefore give it an explicit limit. For our small example, configure the worker to keep up to two downloads in progress.

Picture two available places. Before starting A, the worker takes one place for A. Starting B takes the other. We call each place a processing slot: one slot allows the worker to keep one job active.

The worker now has two occupied slots. A owns one and B owns the other. C is still in the list of images that have not started. It cannot start until the worker has a free slot.

Suppose A and B are both waiting for data. Their requests and saved progress still exist. They are unfinished jobs, so each keeps its slot during the wait.

If the worker returned A’s slot merely because A was waiting, it could start C as well. It would then have A, B and C all unfinished. The supposed two-slot limit would no longer limit unfinished downloads. This is why we count the whole job, including its waits.

When B has received its complete image, finished saving it and finished using its download resources, its slot becomes available again. C can then take that slot and start. A can remain in progress in its original slot.

Two is an illustrative limit, chosen so we can follow every job. The useful rule is that each unfinished download holds one slot. A real worker’s configured count depends on the resources and workload it must support.

There is still a separate question: how does the worker advance B while A is waiting? The slots allow two active jobs, but the program needs an execution arrangement that lets it make use of those waits.

First consider a program that starts A and waits for A’s entire download to finish before looking at the next image. B remains unstarted, even while A is waiting for data. Configuring a second slot does not change this sequence by itself.

We can instead use a network interface that starts A’s request, then lets the program continue to its next step before the whole image arrives. The request remains pending, and the program keeps A’s progress record so it can continue the right job later.

The worker can now take its next permitted action: start B in the second slot. A is still unfinished. The program has simply stopped making the rest of its work wait for A’s complete response.

This is asynchronous I/O: an input or output operation can remain pending while the program continues other work. In our example, the input is image data arriving over the network. The worker can start both permitted downloads without waiting for the first to finish.

How does it know when to return to A or B? The networking library and operating system track the pending network activity. They can report when data is available, allowing the program to resume the matching download. The program uses its saved job state to handle that data for the right image.

Suppose some data for B becomes available before A’s data. The worker handles B’s ready step: it reads the available data and records its progress. B still occupies its slot because the rest of the image and its required saving may remain unfinished.

This distinction matters: ready to make progress is not the same as complete. B may receive one piece, wait again, then receive another piece. C cannot take B’s slot just because the worker has handled one piece.

When the whole of B has arrived, its file has been saved and its download resources are no longer in use, B releases its slot. The worker can use that slot to start C, even though A is still waiting. Downloads do not have to finish in the order they started.

If A and B are both waiting for more network data and neither has another ready download step, the worker has no download step to run at that moment. Both slots remain occupied, so C still waits to start. An event-driven implementation can wait for a network notification instead of repeatedly asking whether each image is ready.

A common coordinator for this pattern is an event loop: it runs ready steps, waits for notifications when none are ready, and resumes the relevant job. This is one way to organize asynchronous I/O. The network wait continues without requiring the program to keep executing instructions for that download.

Step through the example below. After “B data ready,” predict whether C may start. Then compare that state with “B finishes”: what has changed about the work the worker is still responsible for?

Asynchronous I/O gives the worker a way to advance other ready work during network waits. The two processing slots keep a separate promise: at most two image downloads remain in progress. We need both ideas to explain this worker’s behavior.