MOM: Memcached Operations Monitoring

[Chris Goffinet]

Very much of an interest to me! I'd be willing to write a plugin to  
Hyperic HQ (extend the memcache data collection that exists for that  
plugin) to suite this as well to chart the data and setup alerts.



Chris Goffinet
goffinet at yahoo-inc.com



On Oct 11, 2007, at 2:16 PM, Randy Wigginton wrote:

>
> Hi All,
>
> My company does something unusual with memcached that is extremely  
> valuable to us, and I'm wondering if others would find the code  
> useful.  I apologize if this email is long, but what I'm proposing  
> requires some background explanation, as it is radically different  
> from the typical use of memcached.  The ultimate question is  
> whether  it is worthwhile to make the code available for others to  
> use.
>
> One problem with running a large site (hundreds of millions of hits  
> per hour) is keeping track of what is going on.  At one point I  
> worked on the Ebay swat team, and I would get calls at 2 in the  
> morning from the operations center, wondering why a set of machines  
> was acting up.  Without instrumentation, it is nearly impossible to  
> figure out.  With the proper instrumentation, it is child's play.   
> Ebay has a VERY large system to track all activity on the site;  
> however, that system is VERY large and VERY expensive.  Using  
> memcached, I've developed something that gives you 90% of the value  
> of Ebay's system for perhaps 1% of the cost.
>
> I have modified memcached as well as the java client library; with  
> these modifications, and very few lines of code in the application,  
> I can tell precisely how many URLs and SQLs are executing on a  
> particular machine in any given minute or in any given hour.  I can  
> tell you the average execution time, the maximum execution time, as  
> well as the number of failures.  I can tell you which URLs were  
> expensive, which URLs invoked SQL statements, which urls failed most  
> often.  Coupled with a small mysql database, I can give you more  
> operational statistics on our site than many larger sites have  
> available.
>
> Just to reassure those who are assuming this must be a very  
> expensive use of memcached, I can say from experience that with a  
> single instance running on a linux box with a mere 10M of memory  
> assigned, we aggregate information on about 20 pools and several  
> hundred machines at a rate of 10-15K operations per second, and have  
> never gotten close to capacity.
>
> Would anyone else be interested in this?  Or is this too far off the  
> beaten path?  It is mostly helpful for very busy sites.  Thanks.
>
> --randy
>