Memcached works well, then hangs

[Alex Firmani]

I have 3 dedicated memcached servers up. At the moment, I'm only using
one at a time so that I can try and clear up this issue.

The memcached server runs well, but occasionally will hang. I have it
running in -vv verbose mode now so I can see the traffic and it
appears to be fine. However, I left it a few hours ago.. came back..
and part of the site using the server (HowardStern.com) was hung.

The memcached daemon was still up and running and still serving a few
requests from one section of the site. As soon as I brought memcached
down, the main site resumed loading again. I've done this three times
now so I'm positive that restarting memcached also gets the site
moving again. It's as if it is stuck on a memcached request.

The one section of the site that was still running and accessing
memcached uses it's own set of variables. The main part of the site
(the forums) using memcached has it's own set of variables. If one of
the forum variables is "stuck" for whatever reason, any pages trying
to access that variable, must wait for it to be cleared. Other pages
not trying to get one of those forum variables will continue to load.

My question -- how can I better diagnose this issue?  I thought
leaving verbose mode on would show me where the daemon stopped
running, but since it never fully stops running, I don't have a clear
picture of what command it is hung on. FYI, I'm using the PHP API.

One more thing -- as the daemon stays open, it looks like it is
keeping some connections open as well. The number in front of the
verbose output increases when it has a problem (which I assume
is the # of connections). Right now it is:

<7 connection closed.

When I came back and part of the site was hung, it was:

<164 connection closed.


Thanks for any assistance,

Alex Firmani