SERVER_ERROR out of memory

[Balázs Lécz]

Hi!

We've been running memcached in production for 3 consecutive months with no
problems at all. Memcached is our object cache, supporting a pool of application
servers (9 nodes) that serve 16M pages a day (1700 memcached reqs/s).
Four days ago I've seen sporadic "SERVER_ERROR out of memory" errors in the
application log files (4-10 entries a day). The next day at 08:19 this
message began
to flood the logs as 98-99% of store requests failed with "out of memory".

The memcached process runs on a dedicated node with 4Gbytes of physical RAM,
Linux kernel 2.6.13, 32bit.
No other processes consumed much memory on the node. There was no swapping
activity and the system still had plenty of "free" RAM.

The initial configuration for our memcached was:
"-m 3000 -p 11401 -u root" (we had an 82% hit ratio with this memory setting).

I tried lowering the "-m 3000" setting to 2000, then 1500 and bumbed into the
same problem after a few minutes using both memory settings.

I executed a "stats slabs" command and received the same "SERVER_ERROR
out of memory".

We were running memcached 1.1.12. I applied the "preallocate" patch to no avail.

After placing some "debugging printf"s into the memcached code and
recompilation,
I learned that the error is printed at line 528 of memcached.c right after
it = item_alloc(key, flags, expire, len+2);

Going further, I found that item_alloc() returned 0 after exhausting the default
number of 50 tries.

Any hints on what may cause this problem?


--
Balázs Lécz
System Engineer,
Hungary