Memcached "squeezed" out of memory?

Jeremy Blain jeremy at belent.com
Wed Dec 21 22:20:30 UTC 2005


I experienced a similar problem today, on one of our dev servers.
v 1.1.12.  The process had been running fine since late august.

I was running using the following options:
-u nobody
-m 64
-d
-l 127.0.0.1

The system is a slackware machine running linux 2.6.11 using epoll.

The process just stopped accepting set commands for new items, and failed
with an out of server memory message. 
Existing objects could be accessed with get, or replaced with set.
I tried a flush_all, and no change.
I'm not sure when it failed exactly, our cache code hadn't been worked on
in a while, so the process may have been in this state for a few days 
without
it being noticed because it was a development machine.

If no new objects could be cached, the os may well page it out eventually as
things aren't being used, so might explain the problem.

I restarted the process, and it's fine now (didn't think to do any 
further analysis
of it).


(pasting from the archives)


>Can you run a script that logs top -p <pid> every hour or so, and post a
>log of it all, from startup to crash?
>
>We have memcached processes up for 172 days, so any time I hear about
>a problem like this, I freak out.  I hope it's something else on your box
>bloating in memory and making memcached swap, but I'm still paranoid.
>
>Have you run memcached with -k to keep its memory pinned?
>
>- Brad
>
>
>On Wed, 7 Sep 2005, Don MacAskill wrote:
>
>>/
>/>/ Hi Joel,
>/>/
>/>/ Did you ever get any replies about this?
>/>/
>/>/ I think I'm seeing similar behavior on RHEL4.  memcached stops
>/>/ functioning and I have to stop and restart the processes every few days.
>/>/
>/>/ I run 4 instances per machine, and all 4 fail at the same time.
>/>/
>/>/ I haven't let them run for long enough to see if they're getting
>/>/ 'squeezed', but all the other symptoms sound the same, so I'll bet they are.
>/>/
>/>/ Anyone else seeing this?
>/>/
>/>/ Don
>/>/
>/>/
>/>/ Joel Lee wrote:
>/>/ > Greetings..
>/>/ >
>/>/ > I'm running memcached 1.1.12 with libevent 1.1a on an RHEL4 box (uses a 2.6 linux kernel).  I've varified that this installation is using epoll.
>/>/ >
>/>/ > What I notice happening is that memcached will suddenly stop working and as it does so, it's share of resident memory steadily decreases--almost as if it's being "squeezed out" of memory.  I'm not sure if that is actually what is happening, memory wise, but that is what seems most remarkable to me.
>/>/ >
>/>/ > I can cure this by restarting memcached and apache.  It will then work fine again... typically for a couple of days or so, until another such incident will occur.
>/>/ >
>/>/ > Here is the line I use to start memcached:
>/>/ >
>/>/ > ./memcached -d -p 9008 -m 512 -l 127.0.0.1
>/>/ >
>/>/ > I have 2GB of system memory on this machine.
>/>/ >
>/>/ > Any ideas, or methods for collecting additional information I could post?
>/>/ >
>/>/ > Thanks for any help./
>




More information about the memcached mailing list