Another fun thing about Apache

PDF version

So recently I have run into some fun to do with the apache Web server, and I thought it was worth sharing.  Well I didn't really, but my buddy Jeff did.  Just about every time there is a holiday over the weekend at my work, at some point the Web server would run out of memory so badly it would crash.  It would crash so hard in fact, that it had to be hard power cycled.  I have tried just about everything short of sacrificing a chicken to try and figure this out.  Everything from cron jobs to bad PHP scripts (not mine of course ), but never really figured out the root cause.  Of course that was until MLK Jr. Weekend.

I had come back from a great weekend with an old college roommate to find the server had crashed and no one really noticed.  This point became extremely relevant, that no one noticed... On Tuesday I began doing the usual forensics, greping logs, looking at configs, checking hosted scripts etc.  This really got me no where and I started to think that the machine just didn't have enough ram so i started benchmarking it with ab.  This is where things got really interesting.  The more I hammered the server the better it performed and the less memory it used.  This is when the light bulb clicked on.

The server was running out of memory and effectively committing suicide because it didn't have enough visitors.  For the next few hours I just had a console window on the Web server running top and watched as requests came in and the memory slowly ballooned.  This caused me to check out the apache conf and realize that the prefork conf was set to have each child respond to 1000 requests.  This seemed like a reasonable number for a machine with 1 GiB of RAM and 2 GiB of swap, but it turns out these children were not hitting that request limit for weeks and thus never died.  That plus a very slow memory leak caused them to eventually balloon up to 150 MiB each.

Trimming down the requests per child to around 100 (less than my tiny VPS server) has fixed the problem.  This is just one of the saddest things I have seen at my job that is generally full of IT atrocities.  As a large engineering department from one of the largest schools in the country we are literally not generating enough traffic for apache's built-in memory management (or lack thereof, really) to kick in and maintain the server.

In the words of Nicholas Studt....

Sigh