Steve,<br><br>Just curious what are the OS load averages on your database servers? Have you expanded facebook to the point where losing most of the memcache servers would cause your entire application to grind to a halt?<br>
<br>During my initial thoughts on integrating memcache into our product, I could see it eventually becoming a crutch and we wouldn't have enough database hardware to support the application anymore. I wonder if that's a good thing or a bad thing?
<br><br>Thanks!<br><br>--Cal<br><br><div><span class="gmail_quote">On 5/3/07, <b class="gmail_sendername">Steve Grimm</b> <<a href="mailto:sgrimm@facebook.com">sgrimm@facebook.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<font face="Verdana, Helvetica, Arial"><span style="font-size: 12px;">We rebuild from the database. We have enough memcached servers that losing one has a relatively small effect on our cache hit rate. Not to say there's no effect -- our DB load spikes up for a little while when we lose a memcached server -- but we build out our infrastructure such that even at peak load, repopulating an empty memcached instance or two doesn't slow things down noticeably for the users.
<br><span class="sg">
<br>
-Steve</span><div><span class="e" id="q_11253bf6dd41f591_2"><br>
<br>
<br>
On 5/3/07 12:23 PM, "Murty Chittivenkata" <<a href="mailto:murty@aol.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">murty@aol.net</a>> wrote:<br>
<br>
</span></div></span></font><div><span class="e" id="q_11253bf6dd41f591_4"><blockquote><font face="Verdana, Helvetica, Arial"><span style="font-size: 12px;">Steve,<br>
<br>
are you replicating the hash data to hotspares or rebuilding in the event of failure from backend database?<br>
<br>
<br>
Thanks<br>
Murty<br>
</span></font><blockquote><font face="Verdana, Helvetica, Arial"><span style="font-size: 12px;"> <br>
</span></font><blockquote><font face="Verdana, Helvetica, Arial"><span style="font-size: 12px;"><br>
<br>
We have a home-built management and monitoring system that keeps track of all our servers, both memcached and other custom backend stuff. Some of our other backend services are written memcached-style with fully interchangeable instances; for such services, the monitoring system knows how to take a hot spare and swap it into place when a live server has a failure. When one of our memcached servers dies, a replacement is always up and running in under a minute.
<br>
<br>
</span></font></blockquote></blockquote><font face="Verdana, Helvetica, Arial"><span style="font-size: 12px;"><br>
<br>
</span></font></blockquote><font face="Verdana, Helvetica, Arial"><span style="font-size: 12px;"><br>
</span></font>
</span></div></div>
</blockquote></div><br><br clear="all"><br>-- <br>Cal Heldenbrand<br> FBS Data Systems<br> E-mail: <a href="mailto:cal@fbsdata.com">cal@fbsdata.com</a>