One Goroutine per Directory: Rereading the Concurrent Search in My Go File Explorer

Rewritten in October 2026. The original 2024 version of this post described the project's goals. This version reviews the code that actually shipped.

In October 2024 I built a small file explorer in Go to practice concurrency. Rereading the code two years later, the first surprise was where the concurrency ended up.

The part that touches the disk, walking /home with os.ReadDir, is a plain sequential recursion. The part that uses goroutines is the search, and it runs over a tree that is already in memory. The concurrent piece also has a data race, and its cancellation path returns the same answer as a search that found nothing.

This post walks through that search line by line, because each problem in it is a common Go mistake. It is a code review of a learning project, not a performance report. I have no benchmark comparing a sequential and a concurrent version, and nothing below claims a speedup.

What the explorer does

The repository is a Go server rendering templ and HTMX pages. Listing files calls filesystem.ListAll("/home"). That function builds a Node tree by recursing through os.ReadDir, then stores the tree in a process-wide singleton named fakeStorage:

type Node struct {
	Name     string
	IsDir    bool
	Children []*Node
}

The walk runs on the request goroutine, one directory after another. Permission errors are logged and skipped. Search then works only on the stored tree, without touching the disk.

The search, as written

func (f *fakeStorage) SearchFile(ctx context.Context, file string) ([]string, error) {
	var wg sync.WaitGroup
	var foundFiles []string
	var goFuncCounter int
	var searchNode func(nodes []*Node, path string)
	searchNode = func(nodes []*Node, path string) {
		defer wg.Done()
		for _, node := range nodes {
			select {
			case <-ctx.Done():
				return
			default:
				currentPath := path + "/" + node.Name
				if node.Name == file {
					foundFiles = append(foundFiles, currentPath)
				}
				if node.IsDir {
					wg.Add(1)
					go searchNode(node.Children, currentPath)
				}
				goFuncCounter++
			}
		}
	}
	wg.Add(1)
	go searchNode(f.tree.Children, "")
	wg.Wait()
	if len(foundFiles) == 0 {
		return nil, fmt.Errorf("file does not exist")
	}
	log.Printf("Go func was called %d", goFuncCounter)
	return foundFiles, nil
}

The handler passes r.Context(), so the search is cancelled when the client disconnects.

The WaitGroup is correct: Add always runs before the matching go statement, so Wait cannot return early. The other three properties of the function hold up less well.

Two goroutines writing the same slice

Every goroutine closes over the same foundFiles and goFuncCounter variables. Two directories that each contain a match can call append at the same moment. append reads the slice header, may allocate a new backing array, and writes the header back. Two unsynchronized writers can overwrite each other's result. Under the Go memory model, that is a data race, and a race on a multiword value such as a slice header can lose results or corrupt the slice. goFuncCounter++ races too, on every tree with more than one directory. That counter also counts visited nodes, not goroutines, so its log line is wrong even without the race.

The lost-match race is easy to miss in manual testing. Matches are usually rare and spread out, so most runs return the correct list. It shows up when two matches land in sibling directories processed in parallel, which is exactly the case a file search exists for. I found it by reading the code. I have not reproduced lost results.

The race detector is built for this. Run under go test -race, almost any search over a tree with several directories should report the counter race, and a tree with matches in sibling directories also exercises the append.

The smallest fix keeps the structure and puts every shared write under one lock:

var mu sync.Mutex

mu.Lock()
if node.Name == file {
	foundFiles = append(foundFiles, currentPath)
}
goFuncCounter++
mu.Unlock()

One more shared read sits outside the search: SearchFile reads f.tree without taking f.mu, while every list request calls SaveTree. If no list request has run yet, f.tree is nil and the search panics.

A cancelled search looks like "not found"

When the context is cancelled, each goroutine returns at its next loop iteration, and Wait returns. The function then cannot tell a finished search from an interrupted one:

• if nothing had matched yet, it returns file does not exist;
• if some matches had been found, it returns them as a complete, successful answer.

Neither result says the search was incomplete. The caller here is an HTTP handler whose client has already left, so the practical damage is small. The same function inside a CLI or a background job would report a partial list as complete. After wg.Wait(), it should check ctx.Err() and return that error before looking at foundFiles.

Goroutines too small to pay for themselves

The search spawns one goroutine per directory, with no upper bound. Goroutines are cheap, but they are not free. Each directory's work here is a short loop over an in-memory slice, comparing strings, with no system call or network request to wait on. CPU-bound work can still gain from running on several cores, but only if each unit of work is large enough to outweigh the cost of creating and scheduling it. A goroutine per directory, however small the directory, is fine-grained enough that my inference is the overhead is comparable to the work. I have not measured it, so that remains a hypothesis to test.

The part that waits on the operating system, os.ReadDir across a real directory tree, is the sequential part. Overlapping those calls might help, depending on the storage, the cache state and how the tree is shaped.

A version that addresses all three problems could look like this. It is a sketch, not code from the repository:

g, ctx := errgroup.WithContext(ctx)
g.SetLimit(runtime.GOMAXPROCS(0))
// each directory submits g.Go(...) for its subdirectories,
// matches are collected under a mutex,
// and each worker returns ctx.Err() when it sees cancellation,
// so g.Wait() reports it.

errgroup.SetLimit bounds how many goroutines run at once. Wait returns the first error a worker returns, so cancellation becomes an error only if the workers return ctx.Err(). One caveat: a search that blocks while trying to submit child directories under a limit can deadlock if every running goroutine waits for a free slot. Submitting from inside a worker needs TryGo or a queue. Bounded recursive concurrency is harder than it looks.

For a tree that already sits in memory, a sequential depth-first search is the baseline I would write first. Any concurrent version has to beat it under the benchmark.

The next experiment

The questions this code raises can be checked:

1. Add a test with matches in sibling directories and run it under -race. It should fail on the current code and pass once both shared writes are under the mutex.
2. Add a test that calls the search with an already-cancelled context and asserts that it returns context.Canceled. A pre-cancelled context makes the test deterministic.
3. Write a testing.B benchmark comparing a sequential search with the concurrent one on generated trees of different widths and depths.
4. Only if the walk itself is slow, measure a bounded concurrent os.ReadDir on a cold and a warm cache, since a warm cache alone can make a second run look faster.

What I take from this rereading: decide the size of each concurrent unit of work and the limit on how many run at once before writing the first go statement. Treat shared state and cancellation as part of that same design, and let a benchmark against the sequential version decide whether the concurrency stays.