Slashdot's patch
Jamie McCarthy
jamie@mccarthy.vg
Sat, 26 Jul 2003 09:10:10 -0400
Hi all,
Slashdot's been using memcached to serve its comment text for a
couple of weeks now and it's working great. All part of our push
to get the number of SQL queries per page down to a bare minimum.
Next up is saving user information, but I have issues with
atomicity there. I'd love to put our getUser() call into
memcached, but we store the data for each user in 100 separate
columns, not as one giant structure, and we call setUser() to
update individual columns from dozens of places throughout our
code. If I put all 100 pieces of data into one giant structure
that gets stored in memcached, then to update one piece of data
for a user, I have to get, edit, and set -- and if two processes
do that at the same time, one of the updates could be lost.
One of the features of memcached is that it never blocks -- which
means, I think, that there won't ever be a way to lock data after
a get, in preparation for a set. I can think of some tricks with
incr and decr that I can probably use, but I'm still trying to
figure out the best way to do that, and how much that will cut
into the performance boost I hope to get. We'll see :)
Anyway, I figured I'd share with y'all the patch that we've been
using to install this on Slashdot (and my own sites)... nothing
fancy just a bit of cleanup IMHO. And I like having access to the
stats from the perl side. This still doesn't install the perl
client automatically with a 'make install', you need to cd into
api/perl and 'perl Makefile.PL && make install'.
diff -r -N -U3 memcached-1.1.6/api/perl/Makefile.PL memcached-1.1.6-new/api/perl/Makefile.PL
--- memcached-1.1.6/api/perl/Makefile.PL 1969-12-31 19:00:00.000000000 -0500
+++ memcached-1.1.6-new/api/perl/Makefile.PL 2003-07-12 18:22:48.000000000 -0400
@@ -0,0 +1,2 @@
+use ExtUtils::MakeMaker;
+WriteMakefile( 'NAME' => 'MemCachedClient' );
diff -r -N -U3 memcached-1.1.6/api/perl/MemCachedClient.pm memcached-1.1.6-new/api/perl/MemCachedClient.pm
--- memcached-1.1.6/api/perl/MemCachedClient.pm 2003-06-16 18:13:32.000000000 -0400
+++ memcached-1.1.6-new/api/perl/MemCachedClient.pm 2003-07-12 17:08:10.000000000 -0400
@@ -46,6 +46,7 @@
%host_dead = ();
}
+# should this be named host_to_sock? or maybe sock_for_host?
sub sock_to_host { # (host)
my $host = shift;
my $now = time();
@@ -69,6 +70,20 @@
return undef unless $self->{'active'};
my $hv = ref $key eq "ARRAY" ? int($key->[0]) : _hashfunc($key);
+ $self->_init_buckets() unless $self->{'buckets'};
+
+ my $tries = 0;
+ while ($tries++ < 20) {
+ my $host = $self->{'buckets'}->[$hv % $self->{'bucketcount'}];
+ my $sock = sock_to_host($host);
+ return $sock if $sock;
+ $hv += _hashfunc($tries); # stupid, but works
+ }
+ return undef;
+}
+
+sub _init_buckets {
+ my ($self) = @_;
unless ($self->{'buckets'}) {
my $bu = $self->{'buckets'} = [];
foreach my $v (@{$self->{'servers'}}) {
@@ -80,15 +95,6 @@
}
$self->{'bucketcount'} = scalar @{$self->{'buckets'}};
}
-
- my $tries = 0;
- while ($tries++ < 20) {
- my $host = $self->{'buckets'}->[$hv % $self->{'bucketcount'}];
- my $sock = sock_to_host($host);
- return $sock if $sock;
- $hv += _hashfunc($tries); # stupid, but works
- }
- return undef;
}
sub disconnect_all {
@@ -121,6 +127,14 @@
_set("set", @_);
}
+sub incr {
+ _set("incr", @_);
+}
+
+sub decr {
+ _set("decr", @_);
+}
+
sub _set {
my ($cmdname, $self, $key, $val, $exptime) = @_;
return 0 unless $self->{'active'};
@@ -230,6 +244,75 @@
}
}
+{
+my %stats_kv = ( "" => 1, malloc => 1, sizes => 1 ); # which stats are key-value?
+sub stats {
+ my ($self, $types) = @_;
+ return 0 unless $self->{'active'};
+ return 0 unless !ref($types) || ref($types) eq 'ARRAY';
+ if (!ref($types)) {
+ if (!$types) {
+ # not sure about other types, definitely not "reset" :)
+ # but: cachedump? slabs? items?
+ $types = [ '', qw( malloc maps sizes ) ];
+ } else {
+ $types = [ $types ];
+ }
+ }
+
+ $self->_init_buckets();
+
+ my $stats_hr = { };
+ foreach my $host (@{$self->{'buckets'}}) {
+ my $sock = sock_to_host($host);
+ foreach my $typename (@$types) {
+ my $type = $typename ? " $typename" : "";
+ my $cmd = "stats$type";
+ die "sock not valid" unless $sock->opened();
+ $sock->print($cmd);
+ $sock->flush;
+ if ($stats_kv{$typename}) {
+ while (my $line .= <$sock>) {
+ $line =~ s/^\0//; # not sure why but 'stats sizes' output begins with NUL
+ $line =~ s/[\r\n]+$//; # we can't { local $/="\r\n"; chomp } because 'stats maps' data is \n-terminated
+ last if $line eq 'END';
+ my($key, $value) = $line =~ /^(?:STAT )?(\w+)\s(.*)/;
+ next unless $key;
+ if ($typename) {
+ $stats_hr->{$host}{$typename}{$key} = $value;
+ } else {
+ $stats_hr->{$host}{$key} = $value;
+ }
+ }
+ } else {
+ while (my $line .= <$sock>) {
+ $line =~ s/[\r\n]+$//;
+ last if $line eq 'END';
+ $stats_hr->{$host}{$typename} ||= "";
+ $stats_hr->{$host}{$typename} .= "$line\n";
+ }
+ }
+ }
+ }
+
+ return $stats_hr;
+}
+} # closure
+
+sub stats_reset {
+ my ($self, $types) = @_;
+ return 0 unless $self->{'active'};
+
+ $self->_init_buckets();
+
+ foreach my $host (@{$self->{'buckets'}}) {
+ my $sock = sock_to_host($host);
+ $sock->print("stats reset");
+ $sock->flush;
+ }
+ return 1;
+}
+
sub _hashfunc {
my $hash = 0;
foreach (split //, shift) {
@@ -247,9 +330,9 @@
=head1 SYNOPSIS
- use MemCached;
+ use MemCachedClient;
- $memc = new MemCached {
+ $memc = new MemCachedClient {
'servers' => [ "10.0.0.15:11211", "10.0.0.15:11212",
"10.0.0.17:11211", [ "10.0.0.17:11211", 3 ] ],
'debug' => 0,
@@ -280,7 +363,7 @@
C<servers>, but that can also be set later with the C<set_servers>
method. The servers must be an arrayref of hosts, each of which is
either a scalar of the form C<10.0.0.10:11211> or an arrayref of the
-former and an integer weight value. (the default weight if
+former and an integer weight value. (The default weight if
unspecified is 1.) It's recommended that weight values be kept as low
as possible, as this module currently allocates memory for bucket
distribution proportional to the total host weights.
@@ -296,7 +379,7 @@
=item C<set_servers>
-Sets the server list this module distribute key gets and sets between.
+Sets the server list this module distributes key gets and sets between.
The format is an arrayref of identical form as described in the C<new>
constructor.
@@ -346,7 +429,7 @@
$mem->replace($key, $value);
Like C<set>, but only stores in memcache if the key already exists. The
-opposte if C<add>.
+opposite of C<add>.
=back
diff -r -N -U3 memcached-1.1.6/items.c memcached-1.1.6-new/items.c
--- memcached-1.1.6/items.c 2003-06-27 16:32:30.000000000 -0400
+++ memcached-1.1.6-new/items.c 2003-07-11 09:45:24.000000000 -0400
@@ -18,7 +18,6 @@
#include <time.h>
#include <event.h>
#include <malloc.h>
-#include <Judy.h>
#include <assert.h>
#include "memcached.h"