Maybe that’s because their own Web Application server, G-WAN, is smoking everybody else – whether this is in the kernel or in user-mode.

Things can be simple, after all, when one does not fear comparisons.

What benchmarks would you like to see that are currently being missed?

Where do you get that 1% from? What value that random percentage has? If a request goes into an infinite loop most likely the request will timeout one one end or the other.

> But we all know these things happen.

If it’s a bug, it should be unit tested, not performance tested.

> This represents exactly the kind of situation you can’t test in context (..)

If it can be shown under a certain load only, then there is a context.

> we need some common understanding of performance

True but this should not be translated into code. Merely asking people who have a good experience in those areas what should be taken into account when discussing about server performances. Why the compulsory requirement for some code?

> Server developers aren’t going to tweak their servers for your one particular application

I don’t expect them to but they can’t expect their servers to be used without an application context either (apart from static serving). So they must be aware of the context in which their server will be used.

What Ian expects is much more of course but I’d say that if by 2010 no one has come with a generic load/performance/etc testing of a web server, maybe it’s because it’s not actually doable in any meaningful way. Testing always depends on the context it takes place.

Do you know what your web server does when 1% of requests go into an infinite loop? That’s not hard to test, and it’s both a reasonable and easy to test. But how many people know what the result is? This represents exactly the kind of situation you can’t test in context, because developers don’t develop expecting to have that problem (it is after all a bug). But we all know these things happen.

Right now I see primarily microbenchmarks (that benchmark one small isolated aspect of a system) and people who claim no general benchmarking is valid. There is a middle ground. It does not seem that difficult to me to just start to benchmark somewhat more interesting things. Server developers aren’t going to tweak their servers for your one particular application; we need some common understanding of performance because server and application are two separate lines of development. Outside of the process model decision (async/greenlets, threaded, multiprocess), I don’t see anything interesting happening in terms of performance, nor do I expect any server to accomplish anything by using the results of this kind of microbenchmark except perhaps outliers that want to get in line with the rest of the pack.

As for combinations, there certainly are many; adding Nginx to the mix doubles the combinations, for example. And uWSGI can be embedded in a bunch of environments. And there are at least several distinct ways to configure mod_wsgi, and I don’t particularly know which one is best. Because of this I think getting a testing rig that other people can use and extend is more useful than one presentation of benchmark results. Actually putting together the results would be handy every so often, it also gives people a baseline to compare against (so they can test their new server setup against some other setup that seems good, instead of testing it against all other setups).

When you see people say “wow, awesome, I’ll really need to try CoolNewWSGIServerX!” in a reddit comment against a set of benchmarks, you might remember:

• They aren’t going to try it. They would have just tried it instead of leaving a comment saying they’re going to try it.

• Even if they do try it, they probably don’t have a production application anyway. If they had a production application, they’d be more concerned about stability and maintainability than raw speed, because except for a very few cases, the server isn’t their bottleneck; their app is always their bottleneck.

As a result, although it’s good to stay involved and try to correct misstatements and benchmarking misconfigurations, I wouldn’t worry about sheep or flocking or whatever. At the end of the day, people are lazy, and they’re likely to choose the system that has the best production feature-set and the best documentation. Marketing also helps, of course, and positive benchmark results can be seen as a form of marketing. But if a server doesn’t have good docs and a good production feature set, it’s not going to see a lot of use, even if it goes at plaid speed.

> That benchmark looked specifically at the raw socket performance of various frameworks.

What Ian expects is much more of course but I’d say that if by 2010 no one has come with a generic load/performance/etc testing of a web server, maybe it’s because it’s not actually doable in any meaningful way. Testing always depends on the context it takes place.

Probably the only realistic test that can be performed across various servers is to test for static serving performances. Once you hit the application server, it becomes way harder. Personally I’m not finding what Ian proposes to test to be actually helpful. This is just a fancier benchmark, not a real test.

For instance, when testing an intranet application, I would be interested in seeing how an operation that I consider sensitive perform under certain conditions. For example, I would like to know how login in performs on a Monday morning by simulating 500 virtual users. How many of those can log at once? How long does it take? Do I have failures? How long does it take for the servers to cool down? Most likely, the HTTP server performances won’t impact any of those.