Hi Jason,<br><br>We thought a bit about automatic failover when we did our .Net client, BeITMemcached, and we ended up deciding that running slightly uncached is better than the potential synchronization errors you can get with it. If a client cannot reach a memcached server, it will mark that as dead internally, and then keep an increasing retry timeout. This means that all operations that would end up on the unreachable server will immediately return false or null, except the occasional operation that gets to retry and has to wait the entire connection timeout. This means you will not get any synchronization errors, only that some parts of your app will be partly uncached, depending on the amount of memcached servers you have.<br>
<br>Also note that even in your symmetrical failure scenario, automatic failover will cause synchronization errors when the failing memcached server is brought back online. It is very likely that different parts of your application will detect this at different times, which means that some parts will have mapped back the keys to the previously failing server, and others will not, and this may or may not be fatal for your application. Failover seems like a good idea since failover itself can be done completely smooth without synchronization errors, but automatic recovery of the same can never be smooth, unless you get lucky and all parts of your application recover at the same time.<br>
<br>Since memcached is designed to explicitly guarantee non-stale data through deterministic, consistent mapping of a key, and explicitly does not guarantee persistence (It's a LRU cache after all), we designed our client to behave along those lines.<br>
<br>In the end, you have to determine what's best for your application. Are you better at handling synchronization errors or are you better at handling a high miss-rate?<br><br><br>/Henrik Schröder<br><br><div class="gmail_quote">
On Tue, Jun 24, 2008 at 12:56 AM, Jason McGuirk <<a href="mailto:jason.mcguirk@supportsoft.com">jason.mcguirk@supportsoft.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hey Folks,<br>
<br>
I'm partially through a .NET web app integration of memcached using the Enyim client bindings published on codeplex and had a few 101 questions about how the protocol behaves during a failure scenario.<br>
<br>
Since the protocol necessitates that the client bindings modulate the hash key over available servers, how does this behave during an asymmetrical failure?<br>
<br>
Consider the following example;<br>
<br>
I've got two app servers AppA and AppB, and two Memcache instances MCA and MCB. Consider the case where the server running MCB suffers a hardware failure and is universally unrouteable. The client bindings on both AppA and AppB should detect this failure and re-jigger the key space to map 100% to MCA.<br>
<br>
Consider an asymmetric failure where only one of the servers (say, AppA) is unable to route MCB. This is substantially less likely the a categorical failure, but could happen for numerous reasons- a misconfigured firewall, a housed router, what have you. In this scenario, AppA makes the determination that MCB has failed and will remap the keyspace to MCA. AppB, meanwhile, is able to route MCB successfully and will continue placing Keys into BOTH MCA and MCB, and the chance for both servers to read stale data is inevitable, since invalidations will consistently map to different nodes.<br>
<br>
My question is this, What's the best practice people have discovered here? Should I have AppA detect the failure and notify all other nodes to ignore MCB? How does that work for intermittent failure?<br>
<br>
Thanks in advance for any insight ☺<br>
<font color="#888888"><br>
Jason McGuirk<br>
SupportSoft, Inc.<br>
</font></blockquote></div><br>