<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Philip Neustrom wrote:
<blockquote
cite="midf714681d0604142247v4745d013hcaf9321599a21288@mail.gmail.com"
type="cite">
<pre wrap="">Why would a rolling restart cause cache corruption? You mentioned
that you had cache corruption to begin with (in your application?)
Maybe this is why you see it on a rolling restart?
</pre>
</blockquote>
<p>(At least with the PHP MemcacheD class we use, from PHP.net):<br>
</p>
<p>Data is stored on nodes via a hash. If a node is down, then data is
stored on a different node. So if a single server goes down, then comes
up, then goes down again, there is almost guaranteed to be not-current
data on the cluster (in a high-volume environment, this is as close to
guaranteed as to make no difference). Example:<br>
</p>
<ol>
<li> ServerA goes down,
</li>
<li> ItemA is requested, hash now points at ServerB, which doesn't
have the item cached,
</li>
<li> ItemA is retrieved from the database and cached onto ServerB,
</li>
<li> ServerA comes up again,
</li>
<li> ItemA is modified in the database,</li>
<li> ItemA is requested, hash now points at ServerA, which doesn't
have the item cached,
</li>
<li> ItemA is retrieved from the database and cached on ServerA,
</li>
<li> ServerA goes down AGAIN (pesky server),</li>
<li> ItemA is requested, hash now points at ServerB, which has the
item cached, but </li>
<li> ItemA is retrieved FROM CACHE from ServerB, which is OLDER than
ItemA in the database.</li>
</ol>
<p>At this point, there is no way for the code to know that ItemA is
older than what's in the database. Attempting a rolling restart of
every memcached daemon in the cluster while data is actively being
written to the daemons will cause flaky cache results if any server
subsequently goes back down, since it is exactly this case of a server
going down and coming back up.<br>
</p>
<p>At least... that's what I think.<br>
</p>
--<br>
timeless<br>
</body>
</html>