<HTML>
<HEAD>
<TITLE>Re: Largest production memcached install?</TITLE>
</HEAD>
<BODY>
<FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'>We rebuild from the database. We have enough memcached servers that losing one has a relatively small effect on our cache hit rate. Not to say there’s no effect -- our DB load spikes up for a little while when we lose a memcached server -- but we build out our infrastructure such that even at peak load, repopulating an empty memcached instance or two doesn’t slow things down noticeably for the users.<BR>
<BR>
-Steve<BR>
<BR>
<BR>
On 5/3/07 12:23 PM, "Murty Chittivenkata" <murty@aol.net> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'>Steve,<BR>
<BR>
are you replicating the hash data to hotspares or rebuilding in the event of failure from backend database?<BR>
<BR>
<BR>
Thanks<BR>
Murty<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'> <BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'><BR>
<BR>
We have a home-built management and monitoring system that keeps track of all our servers, both memcached and other custom backend stuff. Some of our other backend services are written memcached-style with fully interchangeable instances; for such services, the monitoring system knows how to take a hot spare and swap it into place when a live server has a failure. When one of our memcached servers dies, a replacement is always up and running in under a minute.<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'><BR>
</SPAN></FONT>
</BODY>
</HTML>