SERVER_ERROR out of memory

[Balázs Lécz]

I suspect, item_alloc() can not unlink items as the refcount of the
items on the tail
are all non-zero. What my cause this?

Further stats from the production system:

stats
STAT pid 19054
STAT uptime 2091
STAT time 1139228517
STAT version 1.1.12
STAT rusage_user 41.518594
STAT rusage_system 93.001812
STAT curr_items 559732
STAT total_items 723126
STAT bytes 2505065879
STAT curr_connections 919
STAT total_connections 1402
STAT connection_structures 921
STAT cmd_get 5014580
STAT cmd_set 895330
STAT get_hits 4056152
STAT get_misses 958428
STAT bytes_read 4661717156
STAT bytes_written 20426640054
STAT limit_maxbytes 3145728000
END


stats items
STAT items:10:number 253952
STAT items:10:age 2081
STAT items:11:number 14848
STAT items:11:age 1321
STAT items:12:number 424
STAT items:12:age 2082
STAT items:13:number 234584
STAT items:13:age 2056
STAT items:14:number 55975
STAT items:14:age 1541
END


On 2/6/06, Balázs Lécz <leczbalazs at gmail.com> wrote:
> Hi Jacques!
>
> To rule out operating system level limits, I purposely lowered the memory limit
> of the memcached process by supplying -m 1000.
> The problem persisted.
> I implemented a retry count histogram in items.c and found that the
> built-in limit
> of 50 is too low. I set the retry limit to 2000 to see what happens.
> It did not solve the problem, only deferred the appearance of the
> first "out of memory" error :(
>
> Please note, that we already have 3 months of problem-free operation
> with the exact
> same settings/kernel/hw/memcached.
>
> The memcached process starts giving "out of memory" errors "well after" reaching
> the -m limit (that is: a few minutes after it started rotating out
> items off the LRU).
>
> What I learned so far is that the search function in item_alloc() can
> not unlink an item
> in 50 (the default) or 100 or 2000 tries.
> Setting a higher retry limit only defers the problem but does not solve it.
>
>
> --
> Balázs Lécz
>
>
> On 2/6/06, Jacques Caron <jc at oxado.com> wrote:
> > Hi,
> >
> > At 10:08 06/02/2006, Balázs Lécz wrote:
> > >The initial configuration for our memcached was:
> > >"-m 3000 -p 11401 -u root" (we had an 82% hit ratio with this memory setting).
> >
> > Check for hard limits (see limit -h) on your processes, and kernel
> > settings related to the max size of a process (don't know the
> > specifics on Linux, on FreeBSD that would be maxdsiz).
> >
> > Hope that helps,
> >
> > Jacques.