Scalable Is Not Fast: Latency, Throughput and the Queue in Between

In staging, the endpoint answers in 40 milliseconds. On launch day, the same endpoint takes four seconds. Nobody touched the code. What changed was the number of people asking at the same moment. The service did not get slower at its work; it started making people wait to be served.

That distinction is the whole subject of this note. “Fast” and “scalable” are routinely used as if they meant the same thing, and they are measured by different numbers. Mixing them up is how a service that looked excellent in every test ends up failing in front of customers.

Three Numbers That All Get Called Speed

01 · Definitions

Latency, throughput and capacity are different questions

• Latency is how long one request takes, from the moment it is sent to the moment the answer is back.
• Throughput is how many requests the system finishes each second.
• Capacity is the highest rate it can sustain. It is set by whichever resource runs out first: worker threads, database connections, processor time, or a service you depend on.

Think of a supermarket. The time one shopper spends at the till is latency. The number of shoppers the store clears per hour is throughput. The ceiling is the number of tills that are open. A shopper at an empty store is served quickly whatever the ceiling is; the ceiling only shows itself when the queue arrives. And latency starts rising well before the ceiling is reached, not at it.

So the first diagnostic question is simple: is it slow for one request, or slow for many? If a single request on an idle system is slow, the work itself is expensive and the fix is to make it cheaper. If it is only slow when the system is busy, you are close to capacity and the fix is a different one entirely.

Fast Is About One Request; Scalable Is About Many

02 · A worked comparison

The fast endpoint is the one that fails first

Take two endpoints. Endpoint A answers in 10 ms, but can only work on 2 requests at once, which puts its ceiling at 200 requests per second. Endpoint B is five times slower per request at 50 ms, but can work on 20 at once, so its ceiling is 400 requests per second.

Send both 150 requests per second. A is 75% busy; B is under 40% busy. Raise it to 190 per second. A is now 95% busy and its average time climbs from 10 ms to roughly 100 ms. B is still under half busy and stays near 50 ms. Push past 200 per second and A’s queue grows without limit, because work arrives faster than it can ever be cleared.

Endpoint A is ten times faster per request and it is the one that falls over first. Speed told you how long a request takes. It told you nothing about how many the service can take at once.

Where Requests Actually Wait

03 · The queue between

Time in a system is waiting plus working

Every request spends its life in one of two states: being worked on, or waiting for its turn. The total is the sum of the two.

When a system is quiet, waiting time is close to zero and the total is just the service time, which is the number your tests measured. When it is busy, the waiting term grows and the service term does not move. That is why launch day can feel so unlike staging while the code stays identical.

Lines form in more places than people expect: the queue of requests in front of the worker threads, the pool of database connections, any limit on calls out to another service, and any lock that serialises access to shared data. Wherever a limited resource sits on the path, a line can form in front of it.

Speed belongs to one request. Scalability belongs to many.Kintoro Labs

Why Nearly Full Is the Danger Zone

04 · Queueing theory in one formula

The last stretch of utilisation costs the most

Utilisation is simply how much of capacity you are using.

The classic model for a single line is called M/M/1: arrivals come at random, service times vary at random, there is one server and one queue. The two Ms stand for “Markovian”, meaning memoryless, so what happened a moment ago tells you nothing about the next arrival. For that model, the time a request spends in the system is:

The denominator is the whole story. As utilisation approaches 100%, it approaches zero, and the time heads for infinity. With a 10 ms service time, the numbers look like this.

Going from 50% to 80% adds 30 ms. Going from 80% to 95% adds 150 ms. The final fifteen points of utilisation quadruple the time, from 50 ms to 200 ms. The curve is gentle for most of its length and then turns sharply upward.

Real services usually have several workers rather than one. The more general model, M/M/k, lowers the waiting time at any given utilisation, because a free worker is more likely to exist. But the steep climb is still there. It is where Endpoint A’s roughly 100 ms and Endpoint B’s roughly 50 ms came from in the comparison above.

One caveat belongs next to the formula. Real traffic arrives in bursts, not in neat random streams, so the exact figures will not match what you see. The shape carries over; the numbers do not. The practical lesson is to run your busiest resource well below the steep part of the curve.

Little’s Law

05 · Sizing a pool · Little · 1961

How many workers does a given load keep busy?

There is a second formula, and it is remarkable for how little it assumes. It was proved by John Little in 1961. For any stable system, whatever the arrival pattern and whatever order requests are served in, and for any boundary you choose to draw around the system:

Suppose requests arrive at 200 per second and each spends 0.25 seconds in the system. Then L = λW = 200 × 0.25 = 50, so on average 50 workers are busy at once.

Be careful what that 50 means. A pool of exactly 50 would run at 100% utilisation, which is the far end of the curve in the last section. Fifty is a floor, not a size. Size for peaks, then add headroom on top.

Two cautions keep the formula honest. First, it works on averages, and averages hide peaks. Second, use the right W. For a connection pool, W is the time a connection is held. For a request, W is the full response time including time spent waiting, and that grows under overload, which in turn demands more workers, which spirals. In a Java service, these are concrete settings you can look up today: the maximum number of worker threads, and the maximum size of the database connection pool.

Averages Hide the Slow Requests

06 · The tail · Dean and Barroso · 2013

The mean can describe nobody at all

Imagine 99 requests that each take 20 ms and one that takes 2 seconds. The average comes out near 40 ms. No actual request took 40 ms. Ninety-nine people had a fast experience and one had a terrible one, and the average describes neither.

This is why serious teams report percentiles. The p50 is the time that half of requests beat. The p95 is the time that 95% beat. The p99 is the time that 99% beat, so it describes the slowest one in a hundred.

The tail matters more as systems grow. In “The Tail at Scale” (2013), Jeffrey Dean and Luiz André Barroso point out that if each server is slow just 1% of the time, a request that has to touch 100 servers is slow about 63% of the time.

So watch the tail as load rises. The p99 degrades before the average looks alarming, which makes it the earlier warning.

The average tells you how the system feels to nobody. The tail tells you how it feels to someone.

What to Do on Monday

07 · Four steps

Measure, find the queue, keep headroom, then choose the fix

1. Measure the three numbers. Record latency at p50, p95 and p99, record throughput, and find the load at which latency begins to climb. That load is your practical capacity, whatever the configuration says.
2. Find the queue. Work out which pool or limit fills first: worker threads, database connections, calls to another service, or a lock. That is where your time is being spent waiting.
3. Keep headroom. Run that resource below the steep part of the curve, and use Little’s Law to size it for peaks rather than averages.
4. Then choose the fix. Make each request cheaper, add capacity, or limit and shed load. Which one is right depends on which of the three numbers is actually the problem.

Each fix answers a different question, and picking one before measuring is how teams spend a quarter making fast things faster.

Waiting, Not Work

The 40 ms endpoint was never broken. It was fast, and it was not scalable: its capacity ran out, a queue formed, and waiting rather than work filled the time. The code was the same on launch day. What differed was the number of people standing in line.

That is the habit worth keeping: before asking how to make something faster, ask how many it can serve at once and where the line forms.

Fast is one request. Scalable is how many you can serve.