MOM: Memcached Operations Monitoring

[Gordon Luk]

Upcoming is also interested in this sort of stuff. :)

-Gordon

john allspaw wrote:
> Please do share the love. Lots of interest here in flickr-land.
> 
> -j
> 
> ----- Original Message ----
> From: Randy Wigginton <krw at nobugz.com>
> To: memcached at lists.danga.com
> Sent: Thursday, October 11, 2007 2:16:01 PM
> Subject: MOM: Memcached Operations Monitoring
> 
> 
> 
> Hi All,
> 
> My company does something unusual with memcached that is extremely  
> valuable to us, and I'm wondering if others would find the code  
> useful.  I apologize if this email is long, but what I'm proposing  
> requires some background explanation, as it is radically different  
> from the typical use of memcached.  The ultimate question is whether   
> it is worthwhile to make the code available for others to use.
> 
> One problem with running a large site (hundreds of millions of hits  
> per hour) is keeping track of what is going on.  At one point I  
> worked on the Ebay swat team, and I would get calls at 2 in the  
> morning from the operations center, wondering why a set of machines  
> was acting up.  Without instrumentation, it is nearly impossible to  
> figure out.  With the proper instrumentation, it is child's play.   
> Ebay has a VERY large system to track all activity on the site;  
> however, that system is VERY large and VERY expensive.  Using  
> memcached, I've developed something that gives you 90% of the value  
> of Ebay's system for perhaps 1% of the cost.
> 
> I have modified memcached as well as the java client library; with  
> these modifications, and very few lines of code in the application, I  
> can tell precisely how many URLs and SQLs are executing on a  
> particular machine in any given minute or in any given hour.  I can  
> tell you the average execution time, the maximum execution time, as  
> well as the number of failures.  I can tell you which URLs were  
> expensive, which URLs invoked SQL statements, which urls failed most  
> often.  Coupled with a small mysql database, I can give you more  
> operational statistics on our site than many larger sites have  
> available.
> 
> Just to reassure those who are assuming this must be a very expensive  
> use of memcached, I can say from experience that with a single  
> instance running on a linux box with a mere 10M of memory assigned,  
> we aggregate information on about 20 pools and several hundred  
> machines at a rate of 10-15K operations per second, and have never  
> gotten close to capacity.
> 
> Would anyone else be interested in this?  Or is this too far off the  
> beaten path?  It is mostly helpful for very busy sites.  Thanks.
> 
> --randy
> 
> 
> 
> 
> 
> 
>        
> ____________________________________________________________________________________
> Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games.
> http://get.games.yahoo.com/proddesc?gamekey=monopolyherenow