libmemcache(3) 1.1.0rc4...

John McCaskey johnm at klir.com
Wed Dec 22 09:33:39 PST 2004


On Tue, 2004-12-21 at 18:09 -0800, Sean Chittenden wrote:
> > This is awesome Sean.  I've been running very stable for days now under
> > an extremely high load of around 1million sets/gets per hour.
> 
> Wow!  That's fantastic to hear!  Be sure to check out the callback 
> interface as it may let you more efficiently make get calls if you 
> structure your app something like:
> 
Check out these stats for 5pm last night to 9am this morning:

stats
STAT pid 8269
STAT uptime 57909
STAT time 1103735953
STAT version 1.1.11
STAT rusage_user 3596:627229
STAT rusage_system 8962:163544
STAT curr_items 420162
STAT total_items 68031447
STAT bytes 91619292
STAT curr_connections 1
STAT total_connections 584
STAT connection_structures 7
STAT cmd_get 136062198
STAT cmd_set 68031447
STAT get_hits 135641850
STAT get_misses 420348
STAT bytes_read 16627870580
STAT bytes_written 38062445503
STAT limit_maxbytes 268435456
END

Not a single issue, average of roughly 2300 requests handled per second!
This is not with some specially designed unit test.  This is with our
actual application.  The speed up we have seen is phenomenal.

> 1) identify data needs
> 2) acquire data
> 3) act on data
> 4) display data
> 
> Where in 1), you setup a bunch of callbacks.  In 2), you do the actual 
> mc_get() and your callbacks are executed (which get the data from the 
> database in the event of a cache miss).  And by 3) and 4), 
> libmemcache(3) is probably out of the picture.  But, for step 1 and 2, 
> you may have removed a bunch of independent get requests and 
> consolidated them all into one get request.

I'm going to look into this a bit today, right now I just to single
gets.  But I believe it is actually very easy for me to change my code
to perform multi-gets on its own so the callbacks may not be needed in
my situation.

> 
> > One thing I'm curious about is that there doesn't seem to be any 
> > attempt
> > to detect a server coming back up, once it goes down its deactivated,
> > and then its down for good unless I create a new memcache object and
> > readd the server.
> 
> Yeah... this is a gapping hole in the API at the moment.  There are two 
> potential solutions to this.  The first being:
> 
> 	*) Move the server lists to shared memory
> 	*) Provide a tool that administrates the lists in shared memory.
> 
> This would work fine and dandy, but is a less than perfect solution as 
> it requires manual management of server lists.  The ideal/correct 
> solution is spiffy neato and makes all of this kinda foo automagic.  
> You turn on a new server and all of your clients magically get the 
> updated server lists.  In a few days once I've finished mdoc'ifying the 
> binary protocol proposal, it should be very obvious to those that 
> re-read it how this is going to happen.  As things stand, the published 
> protocol doesn't include support for this, but in a day or three (it'll 
> probably be an x-mas present to the folks on the list) the tracking 
> protocol will be completed, along with a fair amount of niftiness and 
> new features that more than a few will want.
> 
> > Any plans to add something to detect servers coming back up?
> 
> Yeah, the fix for this is a collusion of Brad's tracker idea and my own 
> dribblings.
> 
> > Would this
> > be something you would accept a patch for if I do it?  To avoid the
> > performance hit of always checking, I'd be fine with seeing it as a
> > seperate function that attempts to check each deactivated server that
> > must be explicitly called when you want to check.
> 
> Suffice it to say the server and client are going to have versions of 
> the server maps and notification of updated maps will pass 
> automatically from servers to clients.  The master servers (known as 
> trackers) will keep track of the other servers that are available and 
> will update the server list when a client memcached(8) disconnects from 
> the tracker, thus updating the server maps and the trackers pushing the 
> updated lists to their peers, then out to the clients.  What's even 
> more nifty, is this will all happen w/o the cluster invalidating its 
> cache because keys are hashed to virtual buckets, not direct server 
> assignments as they are now.  Basically memcached(8) is going to start 
> to look a lot like a router in that it will map ranges to servers 
> (kinda like a router does IPs to other routers).  -sc
> 

All of this sounds great, but what I had in mind for now is just a
simple mcm_reactivate_servers function or such that could be manually
called and would go through the list of deactivated servers and attempt
ro revive them.  My application isn't handling user requests, its a
background process that never quits.  As such I want to periodically say
once every 5 minutes or something try to restore any dead servers, as we
may sometimes need to take one down for maintenance or hardware may
fail.  If this happens we do not want to have to restart the daemon that
is utilizing libmemcache to get it to see the server after it comes
back.

I know this presents problems with the location of data in the cache,
but the way I see it the worst thing that happens is I get a miss when
really it was cached on the other server, and then I restore the cache
and in the future get hits.  Some memory is wasted, but the extra copy
will just never get accessed and as such rotate out of the LRU cache
anyway. Theres no concern about stale data due to expiration times being
properly set.  Am I missing some major issue?

I'm probably going to add this either today (or late next week, as I
have some time off for xmas).  I think this would likely be useful to
others as well until the full solution you outlined above is
implemented.

-- 
John A. McCaskey
Software Development Engineer
Klir Technologies, Inc.
johnm at klir.com
206.902.2027


More information about the memcached mailing list