Putting it together: a worker pool

The shape of the program

A worker pool is the everyday concurrency pattern: one channel of jobs, a handful of goroutines taking jobs from it, one channel of results, and a collector that waits for the answers. The producer closes the job channel when it is done; the workers close the results channel when the last of them finishes; the collector ranges until it ends.

It is worth building once by hand, because the same shape reappears as soon as a program has more work than it can do in one pass: parsing files, calling a service, resizing images. The names change; the channels do not.

Where the course's pieces land

The job is a struct with a method that can fail, returning an error rather than panicking — the errors chapter. The workers are goroutines coordinated by a WaitGroup, and their results travel on a channel instead of into a shared slice, which is the ownership rule from the concurrency chapter.

The pool size is a decision, not a constant to copy: three workers for a handful of jobs, more when each job is slow. And the collector sorts the results at the end, because channel order reflects which worker finished first, not what you asked for.

What a real one adds

Two things turn this pattern into production code. The first is context.Context — one value threaded through every call that says "stop", so a shutdown reaches workers waiting on a channel. The second is a limit on what a failure does: some errors stop the batch, some are collected and reported at the end, and the program has to decide which.

Both are additions to the shape you are about to write, not replacements for it. Once the channels are in place, cancellation is a case in a select, and error collection is a second channel or a struct with a mutex.

The pattern in miniature

package main

import "fmt"

type Job struct{ N int }

func (j Job) Run() (int, error) {
	if j.N < 0 {
		return 0, fmt.Errorf("job %d: negative", j.N)
	}
	return j.N * j.N, nil
}

func main() {
	jobs := make(chan Job)
	results := make(chan string)
	done := make(chan struct{})

	go func() {  // the producer
		for _, n := range []int{3, -1, 5} {
			jobs <- Job{N: n}
		}
		close(jobs)
	}()

	go func() {  // the collector
		for line := range results {
			fmt.Println(line)
		}
		close(done)
	}()

	for job := range jobs {  // one worker, for two values
		value, err := job.Run()
		if err != nil {
			results <- fmt.Sprintf("failed: %v", err)
			continue
		}
		results <- fmt.Sprintf("job %d -> %d", job.N, value)
	}
	close(results)
	<-done
}

Backpressure, ordering and cancellation

An unbuffered jobs channel makes the producer wait for a free worker; a buffered one lets it run ahead by the buffer's size and no further. That waiting is backpressure, and it is a feature: a pipeline with no limit will read a whole file into memory to hand it to three workers.

Results arrive in completion order, not input order, so anything that must stay stable — a report, a test, a generated file — needs sorting or an index carried with the result. Work fans out to the workers and back in to one collector, and the collector is the only place that appends to the shared slice.

Cancellation is the missing half. context.Context is passed to every stage; when it is cancelled, a select on ctx.Done() lets a worker abandon a job and return instead of blocking on a send nobody will read. The closing order stays the same; only the reason for stopping changes.

A worked run, in order

Trace the first example in time. main starts three workers, each blocked on range jobs. The producer sends 4; one worker takes it, squares it, and blocks sending to results. The producer sends -2; a second worker takes it and sends a failure line. The collector is already ranging, so both sends complete. When the producer closes jobs, each worker’s range ends, Done runs three times, and the goroutine waiting on wg closes results. The collector’s range ends after the last line, and main sorts and prints. Every step is a handoff, and no step needs a sleep.

A pool bounds concurrency; it does not make the work safe. If two jobs write the same file or the same map, the pool only makes the race happen sooner. The pattern decides how many run at once, not whether they can run together.

Try these next

• Add a second stage that turns each square into a label, keeping the collector the only reader.
• Give the jobs channel a buffer of two and explain what changes when the producer runs ahead.
• Add a context.WithCancel and stop the producer after ten jobs, with every send selecting on ctx.Done().