Stress Test Script

[Stephen Woodbridge]

Hi all,

I wrote a simple script to stress test memcached as part of an 
experiment to see how it behaved. I'm not sure how valid this test is 
but I am interested in knowing about some strange behavior we ran into.

The test is pretty simple and straight forward:

1) generate a key using an md5 hash of a counter, ie: 0,1,2,3,...
2) create data for the key using a random size between 1 and 20k bytes
3) do a "set" followed by a "get", if the data doesn't compare, log it 
as an error
4) print on a stats report of "count, failed, MB's set"
5) loop forever

We then fired off multiple scripts and hammer the server.

The good news is that running 1-5 scripts seem to have a constant load 
of about 12-15% of the cpu an a 1GB cache. After pushing a large number 
of GB through the cache (like 64 GB+, don't remember the exact number) 
we started getting a lot for failures on the test clients, like one was 
failing 30% and another failing 60% of its sets. Restarting the server 
cleared the problem until we pushed a lot of data through the cache again.

So what is going on? Is this the cache fragmentation problem? Is there a 
fix for this problem? Other thoughts?

Thanks,
   -Steve

Here is the script if you want to try it, or comment on it.
./memcache-test server=localhost stress=0

#!/usr/bin/perl -w
use strict;

use Cache::Memcached;
use Data::Dumper;
use Digest::MD5 qw(md5_hex);
use CGI;

my $cgi = new CGI;
my $server  = $cgi->param('server');
my $service = $cgi->param('service') || '';
my $cid     = $cgi->param('cid');
my $data    = $cgi->param('data');
my $stress  = $cgi->param('stress');
my $size    = $cgi->param('size');

my $go      = 1;
my $cnt     = 0;
my $byte    = 0;
my $fail    = 0;

Usage() if ! ($server && ($cid || defined $stress));


sub catch_int {
     my $sig = shift;
     #warn "Caught SIG$sig\n";
     $go = 0;
}

sub report {
     printf("cnt=%7d, fail=%7d, data=%d MB\n", $cnt, $fail, $byte);
}

sub makekey {
     my $idx = shift;
     return md5_hex($idx);
}


sub setncheck {
     my $mc   = shift;
     my $key  = shift;
     my $size = shift;

     $size = int rand 20000 if ! $size;
     my $data = $key x ($size/length($key));
     $mc->set("$key.stress", $data);

     my $rdata = $mc->get("$key.stress");
     $fail++ if $data ne $rdata;
     $cnt++;
     $byte += length($rdata) / (1024*1024) if $rdata;
     report() if $cnt % 1000 == 0;
}

my @srv = split(/,/, $server);
my @srvb = ();
for my $x (@srv) {
     if ($x !~ m/:\d+$/) {
         $x =~ s/$/:11211/;
     }
     push @srvb, $x;
}
print "servers = " . join(', ', @srvb) . "\n";

my $mc = new Cache::Memcached {'servers' => \@srvb,
                             #   'debug' => 1,
                                };
if (! $mc) {
     print "ERROR: could not connect to Memcached on $server\n";
     exit(0);
}
else {
     if ($cid) {
         $mc->set("$cid.$service", $data)
             if $data && length($data);

         print "Response:\n";
         my $response = $mc->get("$cid.$service");
         print "key=$cid.$service\n";
         print Data::Dumper->Dump([$response], ['response']);
     }
     else {
         if ($stress > 0) {
             for my $x (1..$stress) {
                 setncheck($mc, makekey($x), $size);
             }
         }
         else {
             $SIG{INT} = \&catch_int;
             my $x = 1;
             while ($go) {
                 setncheck($mc, makekey($x), $size);
             }
             $SIG{INT} = 'DEFAULT';
         }
     }
     report();
}

exit(0);

sub Usage {
     die "Usage: memcache-test server=<host> service=<string> cid=<cid> 
data=<string>\n" .
         "  <host> is the IP or name of the memcached server (REQUIRED)\n" .
         "         it maybe a comma separated list of servers\n" .
         "  <stress> is the number of items to store in a stress test, 0 
= infinite\n" .
         "  <size> is the size of the item to store (OPTIONAL), default 
is random\n" .
         "    -------- or ---------\n" .
         "  <host> is the IP or name of the memcached server (REQUIRED)\n" .
         "         it maybe a comma separated list of servers\n" .
         "  <service> is the fcache service string (OPTIONAL)\n" .
         "  <cid> is the fcache ID to set or get (REQUIRED)\n" .
         "  <data> is a string that will be set for the <cid>.<service> 
(OPTIONAL)\n";
}