aI wrote rc-httpd; it's a CGI-capable web server written in rc shell script. I started the project with the idea that I could probably write a whole web server from scratch, learning the shell tools ant HTTP protocol as I went, in less time and effort than learning how to configure Apache 2.2. I reckon I was just-about right. ;) I wrote it for myself, and I never thought it would need to handle the kind of traffic cat-v.org gets.
werc (the CMS for cat-v.org) is also written in rc shell. It's bigger and does more work than rc-httpd, but rc-httpd always seems to be the point of failure. I suspect it fails where I used awk to parse the HTTP headers; plan 9's awk is very slow.
What's amusingly ironic, considering the content of the site, is werc and rc-httpd perform well on Linux and only Linux. The two together have been used on Linux of course, OpenBSD, Mac OS X, and the 9front fork of Plan 9; Linux is the only one where they perform well. It seems Linux the only operating system optimized for shell script performance, and yes, I laugh every time I say "optimized for shell script performance!"
On 9front werc and rc-httpd perform a role beyond serving pages: they exercise the system, and have exposed some nasty bugs. They were first used when 9front had barely diverged from Bell Labs Plan 9. Back then the system could barely handle the load of a far less trafficked site than cat-v.org. It wasn't just rc-httpd, the whole system would go down when just a few pages were loaded at once! 9front has come far enough that it's serving cat-v.org today, but it still has some way to go. werc and rc-httpd also still have some way to go, today there was talk of making werc cache pages which would help a lot, and I plan to improve rc-httpd a bit too, but I'll only take it so far. I don't really think 9front would be improved by prioritizing shell script performance. :D I might try to write a web server in C, I think I could do it now, or there was mention of maybe porting Varnish, too. I've never looked at Varnish... might do that tonight.
Trailing thought: If I do write a web server, I don't think I'll take it as far as Tom Duff did. Tom Duff's httpd keeps all the web pages in the executable; changing a page means recompiling!
... and having written all that, can I say how much I HATE form cookie timeouts?
cat-v.org runs werc[1], a minimalist web not-framework written by Uriel in the rc shell[2], served by rc-httpd[3], a web daemon also written in the rc shell, by eekee.
The site is served by a single plan 9 virtual machine hosted by myself and maintained by stanley lieber. Further website optimization is not particularly high on our to-do list. Anyone who wants to donate CDN services or front-end varnish machines is welcome to get in touch.
I can't image how you'd write a web server that couldn't cope with that low load (actually I can but not for a mostly static site like this).
Even if we consider that it would be an additional page view every 5 seconds, this is barely believable that this would put the server under a stress load.
Either this one is wildly more popular or maybe something else is going on as well.
Cat-v is under "page view every five seconds" load basically continually. You don't have to imagine how to write the web server, because the source is available for your perusal, in friendly, easy-to-read shell scripting.
The site has trouble because we have devoted our meager resources to other tasks. If someone wants to donate beefier servers I'd gladly replace the aging, asthmatic machines that comprise 9cloud.
It's been my experience, however, that people generally don't want to spend that thousand dollars.
Remember, every page view also loads images, CSS, JavaScript, etc. So a single view can add up to a dozen hits. Depending on how powerful the machine is, how aggressively cached the content, and what else it is being used for, that can consume significant resources.
