Memcached keys expiring to soon

[Kenneth Pedersen]

Hi,

We have a cluster with five webservers accessing a single memcached
instance. When setting keys, the keys typically have a lifetime of only
a few seconds (typically around 10s) in the cache, even if we set expiry
time to 0 (do not expire). The memory usage of memcached is not even
near its maximum quota. We have also tested explicitly setting expiry
time to some large value (e.g. 10000 seconds), but this makes no difference.

For example, this behaviour is present when memcache is using 80 MB out
of a maximum quota of 256. The machine the memcache daemon is running on
has over 500 MB free memory.

We are using version 1.2.0 of memcached, and this is the command line
used for starting the daemon:

/usr/bin/memcached -d -p 11211 -l 10.1.2.3 -m 256 -c 1024 -u nobody

I have made a small test program that exposes this behaviour, it is
attached below. Note that this only happens when the webserver daemons
are actively using the cache, when we run the script with another
memcached instance that has no other clients, the keys persist in the
cache as expected.

Does anyone have an idea about what is going on here?

Regards,
Kenneth Pedersen
Netlife Backup Solutions AS

Below:
test script (uses python memcache bindings)
typical output from "stat" command
output from "free"

-------------
import memcache
import time
import sys

mc = memcache.Client(['10.1.2.3'])

print "Setting a value"
mc.set('foo', 'bar')
t = time.time()

print "Waiting for value to expire from cache",
sys.stdout.flush()
while True:
    if mc.get('foo') is None: break
    print ".",
    sys.stdout.flush()
    time.sleep(1)
print ""
print "Item expired in %.2f seconds." % (time.time() - t)

-------------
STAT pid 15782
STAT uptime 410
STAT time 1166641383
STAT version 1.2.0
STAT pointer_size 32
STAT rusage_user 6.736421
STAT rusage_system 14.232889
STAT curr_items 66637
STAT total_items 202056
STAT bytes 30107093
STAT curr_connections 82
STAT total_connections 90607
STAT connection_structures 227
STAT cmd_get 108548
STAT cmd_set 202056
STAT get_hits 57672
STAT get_misses 50876
STAT bytes_read 115376234
STAT bytes_written 31361237
STAT limit_maxbytes 268435456
END

-------------
             total       used       free     shared    buffers     cached
Mem:        906340     852888      53452          0     237860     205828
-/+ buffers/cache:     409200     497140
Swap:      1052248        112    1052136