Fan-out reads: query count matters more than query latency

A page loads thirty tiles. Each tile is a query. The page is slow, so you do the obvious thing: profile it, find the slowest query, and make it twice as fast.

The page barely moves.

This is the standard shape of dashboards, activity feeds, GraphQL resolvers, aggregation endpoints, and anything that assembles one response from many independent reads. And in that shape, the intuition that per-unit optimization adds up to system optimization is simply wrong.

The arithmetic that doesn't add up

Start with the observation that should stop you. Suppose the endpoint takes an order of magnitude longer than its slowest individual query. Every query has been measured. None of them is slow.

So where is the time?

Not the network — these are same-datacenter reads. Not serialization, which is microseconds against hundreds of milliseconds. The time is spent waiting for a turn.

Somewhere in your stack there's a bounded pool: a connection pool, a semaphore around your database client, a thread pool, a max_connections on the server itself. You almost certainly put it there on purpose, to stop one page load from exhausting the database. It's doing its job. It's also the thing setting your latency.

A bounded pool turns count into latency

With N queries and C slots, the queries don't run concurrently. They run in roughly ceil(N / C) waves, and the endpoint takes about that many multiples of the mean query time.

That single expression explains why your optimization did nothing:

Halving one query out of thirty removes a fraction of one wave. Halving the count removes half the waves. The two levers are not comparable, and the one everybody reaches for first is the weak one.

The general form: page latency ≈ (fan-out ÷ capacity) × mean unit cost. Fan-out and capacity are multiplicative terms. Unit cost is a single factor, and you only have real leverage on it once the other two are sane.

The instrumentation trap

Here's the part that makes this genuinely hard to diagnose, rather than just counter-intuitive.

The natural way to instrument fan-out is to time each unit:

# the measurement that lies to you
async def run_tile(tile, pool):
    started = time.perf_counter()
    async with pool:                      # may block here for a long time
        rows = await db.query(tile.sql)
    log.info("tile done", tile=tile.id,
             duration=time.perf_counter() - started)
    return rows

That timer starts before the pool is acquired, so duration is queue wait plus execution. A query that waited three waves and then ran quickly reports a large duration — indistinguishable, in your logs, from a genuinely slow query.

So your profile ranks tiles by how late they were submitted, and you go optimize whichever query happened to be last in line. It's not a slow query. It's a query that queued. Meanwhile the actual problem — thirty of them — doesn't appear in a per-tile view at all, because no individual tile looks disproportionate.

Measure the two phases separately and the shape becomes obvious:

async def run_tile(tile, pool):
    queued_at = time.perf_counter()
    async with pool:
        started_at = time.perf_counter()
        rows = await db.query(tile.sql)
        done_at = time.perf_counter()

    log.info("tile done", tile=tile.id,
             wait_ms=(started_at - queued_at) * 1000,   # capacity problem
             exec_ms=(done_at - started_at) * 1000)     # query problem
    return rows

Two fields, and they point at different fixes. High exec_ms is a query to optimize. High wait_ms across many tiles is a count-or-capacity problem, and no amount of query tuning will touch it. If you only ever add one thing to a fan-out system, add this split — it's the difference between knowing which lever to pull and guessing.

Fixes, in order of leverage

Reduce the count. This is the term that multiplies, so it's where the wins are. Coalesce identical in-flight queries so N tiles asking the same question issue one read. Merge tiles that share a shape into a single query with a grouping key and split the result in application code. Delete tiles nobody looks at. On a page where many tiles differ only by a filter value, this can collapse fan-out by an order of magnitude — far more than any query rewrite.

Raise the cap — carefully. Doubling the pool halves the waves, which looks like a free win, and sometimes is. But the pool exists to protect something. If the database can't absorb the extra concurrency, you haven't removed the queue, you've relocated it from your application (where it's visible, bounded, and yours) to the database (where it's shared with every other caller). Raise it only with a measurement showing downstream headroom.

Then optimize the queries. Genuinely worth doing, genuinely last. At a sensible fan-out and a sensible cap, unit cost is what's left, and now a 2× improvement actually shows up in page time.

Note the ordering is the reverse of the instinct. Query optimization is the most visible, most satisfying work — there's a plan to read and a number that moves — which is exactly why it absorbs effort that belongs elsewhere.

The general principle

Any shared bounded resource converts count into latency. Connection pools, worker pools, rate limiters, third-party API quotas, a semaphore someone added during an incident two years ago. Wherever one exists, the request that fans out widest pays for its own width, and the per-unit numbers will all look fine.

So the useful question about a slow composite endpoint isn't which unit is slow. It's:

• how many units does one request issue?

• how many can run at once?

• and is my instrumentation telling me which of those two I'm short on?

Answer those and the slow query, if there even is one, will be waiting for you afterwards.