I see that aiohttp has 5, uwsgi has 16, and gunicorn has 12, 14, and 16 depending on a web-framework, is this your definition of the same?
The author says:
> The rule I used for deciding on what the optimal number of worker processes was is simple: for each framework I started at a single worker and increased the worker count successively until performance got worse.
That's not how benchmark is supposed to be conducted, one doesn't fit workers number to the result that one finds "optimal", one should use the same amount of workers and find bottlenecks and either eliminate them or explain why they cannot be eliminated without affecting benchmark invariants.
> this wouldn't affect the variance between sync and async results very much because both frameworks were run on the same machine.
it will affect the variance. Firstly, because the db will spawn processes on the same machine, pgbouncer will spawn processes on the same machine, they all will compete for the same CPU and the order of preemptive context switches affects individual benchmark runs differently. On top of that, there are periodic and expensive WAL checkpoints, and fsync that competes with a benchmark for the kernel system call interruptions and context switches, and the multi-process workers setup may be affected dramatically. If you don't believe that external processes affect the numbers to the extent they become incomparable, try to serf the Internet with your web-browser randomly while running a benchmark.
> Real world connection pools have an upper bound limit. I don't see why setting an upper bound limit to be closer to reality is not a good test.
Because benchmarks are not real-world workloads, they are designed to show unbound performance of the implementation detail that is selected for a test, where external resources are non-exhaustable for the purpose of avoiding side-effects external to the functionality that is being tested.
> Also you're completely wrong about the connection pool blocking when it is exhausted. See source code:
> If all connections are exhausted then the system still yields to compute and incoming requests.
I didn't say that it wouldn't yield, I said that the coroutine will be blocked at the point where it tries to acquire a non-existing connection from the pool, which affects the benchmark. Now, instead of one blocking context switch at a network socket call that queries Postgres, the coroutine will yield AND WAIT twice - at the exhausted connection pool, and at the network socket call after the connection is acquired. This is exactly the reason why resources should be unbound, and why DB should be on a separate machine (unbound spawning of connection processes upon request), and why the number of OS workers should be the same in all benchmarks, because the sync version will also block twice, and the consequence of blocking there will be much more dramatic and different than in the case of async, WHICH IS THE POINT of a proper benchmark - https://github.com/calpaterson/python-web-perf/blob/master/s...
I see that aiohttp has 5, uwsgi has 16, and gunicorn has 12, 14, and 16 depending on a web-framework, is this your definition of the same?
The author says:
> The rule I used for deciding on what the optimal number of worker processes was is simple: for each framework I started at a single worker and increased the worker count successively until performance got worse.
That's not how benchmark is supposed to be conducted, one doesn't fit workers number to the result that one finds "optimal", one should use the same amount of workers and find bottlenecks and either eliminate them or explain why they cannot be eliminated without affecting benchmark invariants.
> this wouldn't affect the variance between sync and async results very much because both frameworks were run on the same machine.
it will affect the variance. Firstly, because the db will spawn processes on the same machine, pgbouncer will spawn processes on the same machine, they all will compete for the same CPU and the order of preemptive context switches affects individual benchmark runs differently. On top of that, there are periodic and expensive WAL checkpoints, and fsync that competes with a benchmark for the kernel system call interruptions and context switches, and the multi-process workers setup may be affected dramatically. If you don't believe that external processes affect the numbers to the extent they become incomparable, try to serf the Internet with your web-browser randomly while running a benchmark.
> Real world connection pools have an upper bound limit. I don't see why setting an upper bound limit to be closer to reality is not a good test.
Because benchmarks are not real-world workloads, they are designed to show unbound performance of the implementation detail that is selected for a test, where external resources are non-exhaustable for the purpose of avoiding side-effects external to the functionality that is being tested.
> Also you're completely wrong about the connection pool blocking when it is exhausted. See source code: > If all connections are exhausted then the system still yields to compute and incoming requests.
I didn't say that it wouldn't yield, I said that the coroutine will be blocked at the point where it tries to acquire a non-existing connection from the pool, which affects the benchmark. Now, instead of one blocking context switch at a network socket call that queries Postgres, the coroutine will yield AND WAIT twice - at the exhausted connection pool, and at the network socket call after the connection is acquired. This is exactly the reason why resources should be unbound, and why DB should be on a separate machine (unbound spawning of connection processes upon request), and why the number of OS workers should be the same in all benchmarks, because the sync version will also block twice, and the consequence of blocking there will be much more dramatic and different than in the case of async, WHICH IS THE POINT of a proper benchmark - https://github.com/calpaterson/python-web-perf/blob/master/s...