Memcached works well, then hangs

[Alex Firmani]

In response to a question I received:

I'm using the publically available PHP client. Since I have 3 servers,
I didn't think the C one would do much good. I also found the client
that used to be at Cytheriange.net by looking around, but that one
gave me trouble serializing and compressing the data.

UPDATE: I knew if I watched the verbose output long enough, I would
see it happen. Sure enough it just did. The site started to hang and
very quickly the number of connections to memcached increased. The
verbose output was like a counter, quickly going from 7 to 120 before
I shut down the daemon and restarted. The site started moving again
instantly.


Alex



Alex Firmani wrote:

AF> I have 3 dedicated memcached servers up. At the moment, I'm only using
AF> one at a time so that I can try and clear up this issue.

AF> The memcached server runs well, but occasionally will hang. I have it
AF> running in -vv verbose mode now so I can see the traffic and it
AF> appears to be fine. However, I left it a few hours ago.. came back..
AF> and part of the site using the server (HowardStern.com) was hung.

AF> The memcached daemon was still up and running and still serving a few
AF> requests from one section of the site. As soon as I brought memcached
AF> down, the main site resumed loading again. I've done this three times
AF> now so I'm positive that restarting memcached also gets the site
AF> moving again. It's as if it is stuck on a memcached request.

AF> The one section of the site that was still running and accessing
AF> memcached uses it's own set of variables. The main part of the site
AF> (the forums) using memcached has it's own set of variables. If one of
AF> the forum variables is "stuck" for whatever reason, any pages trying
AF> to access that variable, must wait for it to be cleared. Other pages
AF> not trying to get one of those forum variables will continue to load.

AF> My question -- how can I better diagnose this issue?  I thought
AF> leaving verbose mode on would show me where the daemon stopped
AF> running, but since it never fully stops running, I don't have a clear
AF> picture of what command it is hung on. FYI, I'm using the PHP API.

AF> One more thing -- as the daemon stays open, it looks like it is
AF> keeping some connections open as well. The number in front of the
AF> verbose output increases when it has a problem (which I assume
AF> is the # of connections). Right now it is:

AF> <7 connection closed.

AF> When I came back and part of the site was hung, it was:

AF> <164 connection closed.


AF> Thanks for any assistance,

AF> Alex Firmani