Maybe that’s because their own Web Application server, G-WAN, is smoking everybody else – whether this is in the kernel or in user-mode.

Things can be simple, after all, when one does not fear comparisons.

What benchmarks would you like to see that are currently being missed?

Where do you get that 1% from? What value that random percentage has? If a request goes into an infinite loop most likely the request will timeout one one end or the other.

> But we all know these things happen.

If it’s a bug, it should be unit tested, not performance tested.

> This represents exactly the kind of situation you can’t test in context (..)

If it can be shown under a certain load only, then there is a context.

> we need some common understanding of performance

True but this should not be translated into code. Merely asking people who have a good experience in those areas what should be taken into account when discussing about server performances. Why the compulsory requirement for some code?

> Server developers aren’t going to tweak their servers for your one particular application

I don’t expect them to but they can’t expect their servers to be used without an application context either (apart from static serving). So they must be aware of the context in which their server will be used.

What Ian expects is much more of course but I’d say that if by 2010 no one has come with a generic load/performance/etc testing of a web server, maybe it’s because it’s not actually doable in any meaningful way. Testing always depends on the context it takes place.

Do you know what your web server does when 1% of requests go into an infinite loop? That’s not hard to test, and it’s both a reasonable and easy to test. But how many people know what the result is? This represents exactly the kind of situation you can’t test in context, because developers don’t develop expecting to have that problem (it is after all a bug). But we all know these things happen.

Right now I see primarily microbenchmarks (that benchmark one small isolated aspect of a system) and people who claim no general benchmarking is valid. There is a middle ground. It does not seem that difficult to me to just start to benchmark somewhat more interesting things. Server developers aren’t going to tweak their servers for your one particular application; we need some common understanding of performance because server and application are two separate lines of development. Outside of the process model decision (async/greenlets, threaded, multiprocess), I don’t see anything interesting happening in terms of performance, nor do I expect any server to accomplish anything by using the results of this kind of microbenchmark except perhaps outliers that want to get in line with the rest of the pack.

As for combinations, there certainly are many; adding Nginx to the mix doubles the combinations, for example. And uWSGI can be embedded in a bunch of environments. And there are at least several distinct ways to configure mod_wsgi, and I don’t particularly know which one is best. Because of this I think getting a testing rig that other people can use and extend is more useful than one presentation of benchmark results. Actually putting together the results would be handy every so often, it also gives people a baseline to compare against (so they can test their new server setup against some other setup that seems good, instead of testing it against all other setups).

When you see people say “wow, awesome, I’ll really need to try CoolNewWSGIServerX!” in a reddit comment against a set of benchmarks, you might remember:

• They aren’t going to try it. They would have just tried it instead of leaving a comment saying they’re going to try it.

• Even if they do try it, they probably don’t have a production application anyway. If they had a production application, they’d be more concerned about stability and maintainability than raw speed, because except for a very few cases, the server isn’t their bottleneck; their app is always their bottleneck.

As a result, although it’s good to stay involved and try to correct misstatements and benchmarking misconfigurations, I wouldn’t worry about sheep or flocking or whatever. At the end of the day, people are lazy, and they’re likely to choose the system that has the best production feature-set and the best documentation. Marketing also helps, of course, and positive benchmark results can be seen as a form of marketing. But if a server doesn’t have good docs and a good production feature set, it’s not going to see a lot of use, even if it goes at plaid speed.

> That benchmark looked specifically at the raw socket performance of various frameworks.

What Ian expects is much more of course but I’d say that if by 2010 no one has come with a generic load/performance/etc testing of a web server, maybe it’s because it’s not actually doable in any meaningful way. Testing always depends on the context it takes place.

Probably the only realistic test that can be performed across various servers is to test for static serving performances. Once you hit the application server, it becomes way harder. Personally I’m not finding what Ian proposes to test to be actually helpful. This is just a fancier benchmark, not a real test.

For instance, when testing an intranet application, I would be interested in seeing how an operation that I consider sensitive perform under certain conditions. For example, I would like to know how login in performs on a Monday morning by simulating 500 virtual users. How many of those can log at once? How long does it take? Do I have failures? How long does it take for the servers to cool down? Most likely, the HTTP server performances won’t impact any of those.

This is why in part you can only really compare synchronous WSGI servers against other synchronous WSGI servers. Similarly you should only compare asynchronous servers against other asynchronous based servers. To try and compare them against each other just doesn’t make a great deal of sense because the programming model is different and one application isn’t necessarily appropriate for the other model or going to be directly portable between them such that a fair comparison can be drawn.

This is why the original Tornado results were arguably flawed, although that didn’t stop a lot of people flocking to it as the latest and greatest thing. The Tornado folks made it even more misleading by comparing a heavy weight Django application against what was for its asynchronous server not much more than a hello world. If one compares Tornado to a hello world application running on a WSGI server like you did, there isn’t much difference and not the huge difference Tornado folks claimed.

One has to realise that the synchronous and asynchronous models are suited for different tasks. To turn Tornado into a server suitable for properly supporting WSGI applications with concurrent request processing, you need to stick a thread pool on top of it and bridge between the asynchronous and synchronous worlds. This is the only real way to avoid the problem of time.sleep(n) blocking the whole server with a true asynchronous server. Once you stick such a thread pool on top you loose various of the features that make asynchronous systems useful for certain tasks, such as being able to handle high number of concurrent requests with low resource usage. If you are going to loose features like that, what is the point, you may as well stick with a normal synchronous server.

We get back to what I originally said to Ian, if one is going to develop a rang of benchmarks, there is no point just saying, here is a bunch of benchmarks and some pretty graphs, if one cant also explain how that is meaningful and actually applicable to what people are doing or need for there specific application so they can make a reasoned judgement as to what is best for them. All these graphs tend to do at the moment is cause the sheep who are easily impressed to move to whatever system the results give the impression of being better than anything else for that specific test.

I agree with your points, that benchmarking is hard and that performance is not the most important factor and also that spawning, upgrading and reaping of workers is probably one of the most interesting aspect of the different servers.

Concerning the benchmark, with an increasing number of servers things get exponentially complex. In my benchmark I have been looking at 14 servers, if I excluded some of them I would have surely gotten comments that it is not complete and that I really ought to check out this or that server as well.

At first I tried to present a more complex web application which more lies in the line of my current project and plans. However, this would quickly rule out all threaded web servers. The following might indicate how things can easily get difficult. I have been thinking about time.sleep(n), but would it be fair to compare an async server such as Gevent or Eventlet with a threaded web server? These async server have implemented such a sleeping function in Greentlet style while for example ModWSGI blocks a complete thread. This would effectively limit the RPS rate on a threaded server to threadpool/n or you would corrupt the async server by using a blocking call.

I can imagine that when looking at different settings, such as for example the size of the response there might be lots of buffer tuning options in uWSGI to specifically tune for that setting (not sure if thats the case but uWSGI sure has lots of things to tune). The problem you quickly run into is that you will be tuning your server around a specific problem, how far will you allow this to go and still call it a comparative benchmark?

Another thing to keep in mind is that when publishing your benchmark results it might give some people the idea that you are judging their lifework, which really isn’t the case. The benchmark if anything is an objective comparison on a very limited domain which might benefit the performance of some servers. I tried to keep the application simple in the hope to keep it as unbiassed as possible but even that can already stir up some commotion.

And yes, the hardest part is not setting up the applications but the specifics on how to tune the application server for the specific application on certain hardware with a certain proxy server. It is difficult to control all these variables and might probably be best suited by something like the computer language shootout, where some of these variables can be pinned down.