memcached not releasing connections when under heavy load

[Jon Drukman]

i'm using memcached 1.1.12 on redhat linux 7.3.  the client is php with 
ryan dean's memcache client, slightly modified by me to have a better 
hashing algorithm.

what we're seeing is there's one key that's being requested 150ish times 
per second by 16 different hosts.  each host runs the memcached server 
and apache.  since all the child processes are interested in the same 
key, they are hashing to the one box and hitting it a lot.  i'm seeing 
tons of connections used on the box and they are not releasing.  i am 
definitely not using persistent sockets.  on one apache instance i 
noticed hundreds of connections to memcache.  when i killed apache they 
didn't immediately terminate, i had to wait for the kernel to reap them.

so basically it looks like with 150ish connections per second, this 
particular memcached is not releasing connections properly.

any ideas on how to solve this?  i've bumped the max # of connections on 
this particular memcached to 2048 and that's bought me some breathing 
room but i am super paranoid that it's going to fall over at any moment. 
  this is of course our biggest traffic week of the year and we're using 
memcache to keep some important stats that are displayed on every page.

-jsd-