PHP Client Hangs Apache: Any Ideas?

[Russ Garrett]

On Wed, 2005-03-16 at 10:55 -0500, Greg Grothaus wrote:
> I tried using memcached across two servers using the PHP client.  After 
> running the servers for a few hours, the number of apache processes had 
> grown to the maximum number on the server and were requests were being 
> dropped.  The apache processes simply weren't terminating.

This sounds remarkably familiar to an elusive and ongoing bug that we've
been seeing for months. We've found no solution, changing memcached
clients doesn't help (we've done it 3 times), and the PHP development
team didn't have a clue. 

Further debugging (try loading up gdb against a hanging apache backend)
indicated that it was to do with some fairly funamental freeing of
constant strings (I seem to remember it was actually hanging in free()),
which obviously makes no sense.

It's not quite as pronounced now as it used to be - we have remedied the
situation by having a 5-minute cron job which kills backends over a
certain amount of memory usage, and a daily job which restarts apache on
all our servers.

We can't prove it's memcached because running the site with memcached
disabled is obviously untenable, and we're unable to reproduce it on a
single server under test loads.

The overall conclusion is that PHP is prone to doing annoying,
inexplicable things seemingly randomly. Several of our site updates
recently have caused random segfaults when put live for no good reason,
and the number of man-hours we spend debugging this (if you can call it
debugging - more like brute-forcing the answer) is insane.

If we were to re-code Audioscrobbler/Last.fm again from scratch we'd
undoubtedly use Java. Sorry this has turned into a rant against PHP,
heh.

-- 
Russ Garrett                            Last.fm Limited
russ at last.fm                             http://last.fm