Strange perl Cached::Memcached behavior

Mon Apr 14 18:37:34 UTC 2008

Hi all,

While debugging a memcached issue we are having, I came across some 
strange behavior of the perl Cached::Memcached module. These are the 
versions I was using:

Memcached 1.2.5
Cached::Memcached 1.24

My test involved a simple script that just increments a key in a loop:

thrash_memd.pl:
---
   use strict;
   use Cache::Memcached;
   my $memd = new Cache::Memcached { servers => [ '127.0.0.1:11211' ] };
   my $key = 'test';
   $memd->add($key => 1, 3600) or warn 'Already added';
   while (1) {
     print $memd->get($key)."\n";
     $memd->incr($key) or warn 'FAIL!';
   }
---

The idea was that I would run this script simultaneously from two 
separate consoles with the aim of replicating an incr failure we are having.
In the first console I restart memcached immediately before executing 
the script:

# sudo /etc/init.d/memcached restart; perl thrash_memd.pl

Shortly after that script starts printing out the incrementing value I 
start the script in the second console:

# perl thrash_memd.pl

This is the output I get from the second script:
---
   Use of uninitialized value in string eq at 
/usr/local/share/perl/5.8.8/Cache/Memcached.pm line 494.
   Already added at thrash_memd.pl line 8.

   Use of uninitialized value in pattern match (m//) at 
/usr/local/share/perl/5.8.8/Cache/Memcached.pm line 525.
   FAIL! at thrash_memd.pl line 12.

   Use of uninitialized value in pattern match (m//) at 
/usr/local/share/perl/5.8.8/Cache/Memcached.pm line 525.
   FAIL! at thrash_memd.pl line 12.
---

Every increment fails in the second script for a period of time, and 
then when it does start incrementing the first script will start 
throwing warnings.

But this only happens when I restart memcached *immediately* before 
executing the first script; if I wait a second or two between the 
restart and the first script then both scripts will happily increment.

I tried this exact same test with Cache::Memcached::libmemcached, and 
did not get this behavior (maybe just because it is slower?).

As I said, the reason for this test was to try and debug a problem in 
one of our applications, and I don't think this strange behavior is 
necessarily related, but it does make me wonder if Cache::Memcached can 
be trusted.

We are trying to solve a problem where incrs randomly fail (ostensibly) 
in our application. We use the following cliche to increment a counter:

---
   unless ($cache->incr($memd_key)) {
     unless ($cache->add($memd_key, 1, 86400) {
       unless ($cache->incr($memd_key)) {
         $log->info("Unable to increment $memd_key");
       }
     }
   }
---

and we see a large number of failures (usually consecutively) in our 
logs when we are almost certain that memcached is running perfectly 
fine. This application receives a fairly large number of page impressions.

Does anyone have any thoughts on the strange test script behavior, or 
why our application might be failing? Is Cache::Memcached reliable 
enough for heavy use? Are we using it incorrectly?

Kind regards
Andrew Bryan

abryan at pangeamedia.com