memcached hiccups?

[marc at corky.net]

I'm only commenting re the kernel message.  You should dig up the 
original message in your /var/log log files....   It's more likely a 
kernel config issue rather than a memcached issue.   I'm guessing it's 
some sort of out of socket memory condition.  Check it out and let us know.

Marc


>
> I've got something strange going on, and can't seem to figure it out. 
> One of our memcached boxes will periodically choke on memcached.
>
> Periodically usually means ~6-8 days or so, but there's some variation.
>
> Choke usually means it fails sock_to_host from lots of clients, so we 
> take it out of rotation and then check it later and put it back in.
>
> It's memcached-1.2.1 from the rpm in dag's repository, using libevent 
> 1.3b from that same repo.
>
> I'm running 4 instances on this box (4 Opteron cores, haven't had the 
> guts to try the multithreaded one yet), all of which fail 
> simultaneously, and one of the four instances says:  "Failed to write, 
> and not due to blocking: Connection reset by peer".  The others don't 
> say anything (only -v).
>
> Finally, the syslog fills up with lots and lots of these:
>
> "Mar 26 09:50:20 poseidon kernel: printk: 13382 messages suppressed."
>
> But there's no indication of what message it was that was suppressed.
>
> There's nothing else running on the box, there's no swap, it's not 
> running out of RAM, etc.
>
> Has anyone else seen this or anything like it?
>
> Thanks,
>
> Don
>
>