An Inconvenient Latency
An Inconvenient Latency
Join the DZone community and get the full member experience.Join For Free
Container Monitoring and Management eBook: Read about the new realities of containerization.
Why don't we use GPUs for everything?
Why web servers favour horizontal scalability
Throughput, latency and concurrency
A given problem has a "natural concurrency" the problem can be easily broken into. If you have 100 concurrent users, you may have a natural latency of about 100. There is also a "minimum concurrency" implied by a benchmark.
To achieve the throughput a benchmark suggests your natural concurrency should greater than the minimum concurrency in the benchmark. If you have less natural concurrency, you might expect you get pro-rata throughput as well.
Consider these benchmarks for three key-value stores.
|Product 1||1,000,000/s||46 ms or 0.046s|
|Product 2||600,000/s||10 ms or 0.01s|
|Product 3||2,000,000/s||0.002 ms or 0.000002s|
You might look at this table and say, they all look good, they all have high throughputs and low enough latencies. The problem is; there is an implied natural concurrency requirement to achieve the throughput stated for the latency measured. Lets see what the minimum concurrency needed to achieve the latency
|Product 1||1,000,000/s||46 ms or 0.046s||46,000|
|Product 2||600,000/s||10 ms or 0.01s||6,000|
|Product 3||2,000,000/s||0.002 ms or 0.000002s||4|
Many problems have around 4 independent tasks, but 46K is pretty high. So what? What if you only have 10 concurrent tasks/users.
|Product 1||10||46 ms or 0.046s||220/s|
|Product 2||10||10 ms or 0.01s||1,000/s|
|Product 3||10||0.002 ms or 0.000002s||2,000,000/s|
Note: having more concurrency than you need, doesn't help throughput much, but a lack of natural concurrency in your problem will hurt you throughput (and horizontal scalability)
ConclusionNext time you read a benchmark which includes throughput and average latency, multiply them together to see what level of concurrency would be required to achieve that throughput and compare this with the natural concurrency of the problem you are trying to solve to see if the solution fits your problem.
If you have more natural concurrency, you have more solutions you can consider, if you have a low natural concurrency, you need a solution with a low latency.
Published at DZone with permission of Peter Lawrey , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.